From nobody Wed Nov 5 07:53:13 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533730091543215.8826813997996; Wed, 8 Aug 2018 05:08:11 -0700 (PDT) Received: from localhost ([::1]:43143 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNG6-0001nw-BA for importer@patchew.org; Wed, 08 Aug 2018 08:08:10 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52351) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBk-0006eF-Ac for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:44 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBg-0008Mb-5w for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:40 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:47658 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBg-0008Lp-0e for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:36 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 874097D84D; Wed, 8 Aug 2018 12:03:35 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 449672166BA0; Wed, 8 Aug 2018 12:03:35 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 0E91A1138651; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:02:39 +0200 Message-Id: <20180808120334.10970-2-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 08 Aug 2018 12:03:35 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 08 Aug 2018 12:03:35 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 01/56] check-qjson: Cover multiple JSON objects in same string X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" qobject_from_json() & friends misbehave when the JSON text has more than one JSON value. Add test coverage to demonstrate the bugs. Signed-off-by: Markus Armbruster Reviewed-by: Eric Blake --- tests/check-qjson.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/tests/check-qjson.c b/tests/check-qjson.c index eaf5d20663..cc952c56ea 100644 --- a/tests/check-qjson.c +++ b/tests/check-qjson.c @@ -1418,6 +1418,25 @@ static void limits_nesting(void) g_assert(obj =3D=3D NULL); } =20 +static void multiple_values(void) +{ + Error *err =3D NULL; + QObject *obj; + + /* BUG this leaks the syntax tree for "false" */ + obj =3D qobject_from_json("false true", &err); + g_assert(qbool_get_bool(qobject_to(QBool, obj))); + g_assert(!err); + qobject_unref(obj); + + /* BUG simultaneously succeeds and fails */ + /* BUG calls json_parser_parse() with errp pointing to non-null */ + obj =3D qobject_from_json("} true", &err); + g_assert(qbool_get_bool(qobject_to(QBool, obj))); + error_free_or_abort(&err); + qobject_unref(obj); +} + int main(int argc, char **argv) { g_test_init(&argc, &argv, NULL); @@ -1455,6 +1474,7 @@ int main(int argc, char **argv) g_test_add_func("/errors/invalid_dict_comma", invalid_dict_comma); g_test_add_func("/errors/unterminated/literal", unterminated_literal); g_test_add_func("/errors/limits/nesting", limits_nesting); + g_test_add_func("/errors/multiple_values", multiple_values); =20 return g_test_run(); } --=20 2.17.1 From nobody Wed Nov 5 07:53:13 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533731016327335.58588809473486; Wed, 8 Aug 2018 05:23:36 -0700 (PDT) Received: from localhost ([::1]:43290 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNV1-0006z4-4D for importer@patchew.org; Wed, 08 Aug 2018 08:23:35 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52342) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBk-0006eA-78 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:44 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBg-0008MM-4g for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:40 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:51024 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBg-0008Lr-0Z for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:36 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 895BC4023461; Wed, 8 Aug 2018 12:03:35 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 45C4A20180F5; Wed, 8 Aug 2018 12:03:35 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 11AF91138656; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:02:40 +0200 Message-Id: <20180808120334.10970-3-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:35 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:35 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 02/56] check-qjson: Cover blank and lexically erroneous input X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" qobject_from_json() can return null without setting an error on lexical errors. I call that a bug. Add test coverage to demonstrate it. Signed-off-by: Markus Armbruster Reviewed-by: Eric Blake --- tests/check-qjson.c | 36 +++++++++++++++++++++++++++++++++--- 1 file changed, 33 insertions(+), 3 deletions(-) diff --git a/tests/check-qjson.c b/tests/check-qjson.c index cc952c56ea..81b92d6b0c 100644 --- a/tests/check-qjson.c +++ b/tests/check-qjson.c @@ -1307,8 +1307,36 @@ static void simple_varargs(void) =20 static void empty_input(void) { - const char *empty =3D ""; - QObject *obj =3D qobject_from_json(empty, &error_abort); + QObject *obj =3D qobject_from_json("", &error_abort); + g_assert(obj =3D=3D NULL); +} + +static void blank_input(void) +{ + QObject *obj =3D qobject_from_json("\n ", &error_abort); + g_assert(obj =3D=3D NULL); +} + +static void junk_input(void) +{ + /* Note: junk within strings is covered elsewhere */ + Error *err =3D NULL; + QObject *obj; + + obj =3D qobject_from_json("@", &err); + g_assert(!err); /* BUG */ + g_assert(obj =3D=3D NULL); + + obj =3D qobject_from_json("[0\xFF]", &err); + error_free_or_abort(&err); + g_assert(obj =3D=3D NULL); + + obj =3D qobject_from_json("00", &err); + g_assert(!err); /* BUG */ + g_assert(obj =3D=3D NULL); + + obj =3D qobject_from_json("[1e", &err); + g_assert(!err); /* BUG */ g_assert(obj =3D=3D NULL); } =20 @@ -1462,7 +1490,9 @@ int main(int argc, char **argv) =20 g_test_add_func("/varargs/simple_varargs", simple_varargs); =20 - g_test_add_func("/errors/empty_input", empty_input); + g_test_add_func("/errors/empty", empty_input); + g_test_add_func("/errors/blank", blank_input); + g_test_add_func("/errors/junk", junk_input); g_test_add_func("/errors/unterminated/string", unterminated_string); g_test_add_func("/errors/unterminated/escape", unterminated_escape); g_test_add_func("/errors/unterminated/sq_string", unterminated_sq_stri= ng); --=20 2.17.1 From nobody Wed Nov 5 07:53:13 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533730984140632.069119691813; Wed, 8 Aug 2018 05:23:04 -0700 (PDT) Received: from localhost ([::1]:43289 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNUV-0006Vd-2k for importer@patchew.org; Wed, 08 Aug 2018 08:23:03 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52350) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBk-0006eE-9d for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:47 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBg-0008MS-4o for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:40 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:45356 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBf-0008Lo-VK for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:36 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 834FC81663CF; Wed, 8 Aug 2018 12:03:35 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 4A8B12166BA8; Wed, 8 Aug 2018 12:03:35 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 152141138657; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:02:41 +0200 Message-Id: <20180808120334.10970-4-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 08 Aug 2018 12:03:35 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 08 Aug 2018 12:03:35 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 03/56] check-qjson: Cover whitespace more thoroughly X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Signed-off-by: Markus Armbruster --- tests/check-qjson.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/check-qjson.c b/tests/check-qjson.c index 81b92d6b0c..0a9a054c7b 100644 --- a/tests/check-qjson.c +++ b/tests/check-qjson.c @@ -1236,7 +1236,7 @@ static void simple_whitespace(void) })), }, { - .encoded =3D " [ 43 , { 'h' : 'b' }, [ ], 42 ]", + .encoded =3D "\t[ 43 , { 'h' : 'b' },\n\t[ ], 42 ]\n", .decoded =3D QLIT_QLIST(((QLitObject[]){ QLIT_QNUM(43), QLIT_QDICT(((QLitDictEntry[]){ --=20 2.17.1 From nobody Wed Nov 5 07:53:13 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533731365135135.21526122497494; Wed, 8 Aug 2018 05:29:25 -0700 (PDT) Received: from localhost ([::1]:43323 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNad-0003kQ-ST for importer@patchew.org; Wed, 08 Aug 2018 08:29:23 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52414) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBm-0006fp-AI for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBi-0008QV-2N for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:42 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:51046 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBh-0008PQ-TP for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:38 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 7143E4023470; Wed, 8 Aug 2018 12:03:37 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 4C6F4104083A; Wed, 8 Aug 2018 12:03:35 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 188701138659; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:02:42 +0200 Message-Id: <20180808120334.10970-5-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:37 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:37 +0000 (UTC) for IP:'10.11.54.3' DOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 04/56] qmp-cmd-test: Split off qmp-test X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" qmp-test is for QMP protocol tests. Commit e4a426e75ef added generic, basic tests of query commands to it. Move them to their own test program qmp-cmd-test, to keep qmp-test focused on the protocol. Signed-off-by: Markus Armbruster Reviewed-by: Eric Blake --- MAINTAINERS | 1 + tests/Makefile.include | 3 + tests/qmp-cmd-test.c | 213 +++++++++++++++++++++++++++++++++++++++++ tests/qmp-test.c | 191 +----------------------------------- 4 files changed, 218 insertions(+), 190 deletions(-) create mode 100644 tests/qmp-cmd-test.c diff --git a/MAINTAINERS b/MAINTAINERS index 666e936812..dc129b7034 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1695,6 +1695,7 @@ F: monitor.c F: docs/devel/*qmp-* F: scripts/qmp/ F: tests/qmp-test.c +F: tests/qmp-cmd-test.c T: git git://repo.or.cz/qemu/armbru.git qapi-next =20 Register API diff --git a/tests/Makefile.include b/tests/Makefile.include index a49282704e..48f18ad581 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -179,6 +179,8 @@ check-block-$(CONFIG_POSIX) +=3D tests/qemu-iotests-qui= ck.sh =20 check-qtest-generic-y =3D tests/qmp-test$(EXESUF) gcov-files-generic-y =3D monitor.c qapi/qmp-dispatch.c +check-qtest-generic-y +=3D tests/qmp-cmd-test$(EXESUF) + check-qtest-generic-y +=3D tests/device-introspect-test$(EXESUF) gcov-files-generic-y =3D qdev-monitor.c qmp.c check-qtest-generic-y +=3D tests/cdrom-test$(EXESUF) @@ -770,6 +772,7 @@ libqos-usb-obj-y =3D $(libqos-spapr-obj-y) $(libqos-pc-= obj-y) tests/libqos/usb.o libqos-virtio-obj-y =3D $(libqos-spapr-obj-y) $(libqos-pc-obj-y) tests/lib= qos/virtio.o tests/libqos/virtio-pci.o tests/libqos/virtio-mmio.o tests/lib= qos/malloc-generic.o =20 tests/qmp-test$(EXESUF): tests/qmp-test.o +tests/qmp-cmd-test$(EXESUF): tests/qmp-cmd-test.o tests/device-introspect-test$(EXESUF): tests/device-introspect-test.o tests/rtc-test$(EXESUF): tests/rtc-test.o tests/m48t59-test$(EXESUF): tests/m48t59-test.o diff --git a/tests/qmp-cmd-test.c b/tests/qmp-cmd-test.c new file mode 100644 index 0000000000..5e4a831adb --- /dev/null +++ b/tests/qmp-cmd-test.c @@ -0,0 +1,213 @@ +/* + * QMP command test cases + * + * Copyright (c) 2017 Red Hat Inc. + * + * Authors: + * Markus Armbruster , + * + * This work is licensed under the terms of the GNU GPL, version 2 or late= r. + * See the COPYING file in the top-level directory. + */ + +#include "qemu/osdep.h" +#include "libqtest.h" +#include "qapi/error.h" +#include "qapi/qapi-visit-introspect.h" +#include "qapi/qmp/qdict.h" +#include "qapi/qobject-input-visitor.h" + +const char common_args[] =3D "-nodefaults -machine none"; + +/* Query smoke tests */ + +static int query_error_class(const char *cmd) +{ + static struct { + const char *cmd; + int err_class; + } fails[] =3D { + /* Success depends on build configuration: */ +#ifndef CONFIG_SPICE + { "query-spice", ERROR_CLASS_COMMAND_NOT_FOUND }, +#endif +#ifndef CONFIG_VNC + { "query-vnc", ERROR_CLASS_GENERIC_ERROR }, + { "query-vnc-servers", ERROR_CLASS_GENERIC_ERROR }, +#endif +#ifndef CONFIG_REPLICATION + { "query-xen-replication-status", ERROR_CLASS_COMMAND_NOT_FOUND }, +#endif + /* Likewise, and require special QEMU command-line arguments: */ + { "query-acpi-ospm-status", ERROR_CLASS_GENERIC_ERROR }, + { "query-balloon", ERROR_CLASS_DEVICE_NOT_ACTIVE }, + { "query-hotpluggable-cpus", ERROR_CLASS_GENERIC_ERROR }, + { "query-vm-generation-id", ERROR_CLASS_GENERIC_ERROR }, + { NULL, -1 } + }; + int i; + + for (i =3D 0; fails[i].cmd; i++) { + if (!strcmp(cmd, fails[i].cmd)) { + return fails[i].err_class; + } + } + return -1; +} + +static void test_query(const void *data) +{ + const char *cmd =3D data; + int expected_error_class =3D query_error_class(cmd); + QDict *resp, *error; + const char *error_class; + + qtest_start(common_args); + + resp =3D qmp("{ 'execute': %s }", cmd); + error =3D qdict_get_qdict(resp, "error"); + error_class =3D error ? qdict_get_str(error, "class") : NULL; + + if (expected_error_class < 0) { + g_assert(qdict_haskey(resp, "return")); + } else { + g_assert(error); + g_assert_cmpint(qapi_enum_parse(&QapiErrorClass_lookup, error_clas= s, + -1, &error_abort), + =3D=3D, expected_error_class); + } + qobject_unref(resp); + + qtest_end(); +} + +static bool query_is_blacklisted(const char *cmd) +{ + const char *blacklist[] =3D { + /* Not actually queries: */ + "add-fd", + /* Success depends on target arch: */ + "query-cpu-definitions", /* arm, i386, ppc, s390x */ + "query-gic-capabilities", /* arm */ + /* Success depends on target-specific build configuration: */ + "query-pci", /* CONFIG_PCI */ + /* Success depends on launching SEV guest */ + "query-sev-launch-measure", + /* Success depends on Host or Hypervisor SEV support */ + "query-sev", + "query-sev-capabilities", + NULL + }; + int i; + + for (i =3D 0; blacklist[i]; i++) { + if (!strcmp(cmd, blacklist[i])) { + return true; + } + } + return false; +} + +typedef struct { + SchemaInfoList *list; + GHashTable *hash; +} QmpSchema; + +static void qmp_schema_init(QmpSchema *schema) +{ + QDict *resp; + Visitor *qiv; + SchemaInfoList *tail; + + qtest_start(common_args); + resp =3D qmp("{ 'execute': 'query-qmp-schema' }"); + + qiv =3D qobject_input_visitor_new(qdict_get(resp, "return")); + visit_type_SchemaInfoList(qiv, NULL, &schema->list, &error_abort); + visit_free(qiv); + + qobject_unref(resp); + qtest_end(); + + schema->hash =3D g_hash_table_new(g_str_hash, g_str_equal); + + /* Build @schema: hash table mapping entity name to SchemaInfo */ + for (tail =3D schema->list; tail; tail =3D tail->next) { + g_hash_table_insert(schema->hash, tail->value->name, tail->value); + } +} + +static SchemaInfo *qmp_schema_lookup(QmpSchema *schema, const char *name) +{ + return g_hash_table_lookup(schema->hash, name); +} + +static void qmp_schema_cleanup(QmpSchema *schema) +{ + qapi_free_SchemaInfoList(schema->list); + g_hash_table_destroy(schema->hash); +} + +static bool object_type_has_mandatory_members(SchemaInfo *type) +{ + SchemaInfoObjectMemberList *tail; + + g_assert(type->meta_type =3D=3D SCHEMA_META_TYPE_OBJECT); + + for (tail =3D type->u.object.members; tail; tail =3D tail->next) { + if (!tail->value->has_q_default) { + return true; + } + } + + return false; +} + +static void add_query_tests(QmpSchema *schema) +{ + SchemaInfoList *tail; + SchemaInfo *si, *arg_type, *ret_type; + char *test_name; + + /* Test the query-like commands */ + for (tail =3D schema->list; tail; tail =3D tail->next) { + si =3D tail->value; + if (si->meta_type !=3D SCHEMA_META_TYPE_COMMAND) { + continue; + } + + if (query_is_blacklisted(si->name)) { + continue; + } + + arg_type =3D qmp_schema_lookup(schema, si->u.command.arg_type); + if (object_type_has_mandatory_members(arg_type)) { + continue; + } + + ret_type =3D qmp_schema_lookup(schema, si->u.command.ret_type); + if (ret_type->meta_type =3D=3D SCHEMA_META_TYPE_OBJECT + && !ret_type->u.object.members) { + continue; + } + + test_name =3D g_strdup_printf("qmp/%s", si->name); + qtest_add_data_func(test_name, si->name, test_query); + g_free(test_name); + } +} + +int main(int argc, char *argv[]) +{ + QmpSchema schema; + int ret; + + g_test_init(&argc, &argv, NULL); + + qmp_schema_init(&schema); + add_query_tests(&schema); + ret =3D g_test_run(); + + qmp_schema_cleanup(&schema); + return ret; +} diff --git a/tests/qmp-test.c b/tests/qmp-test.c index 487ef946ed..b6eff4fe97 100644 --- a/tests/qmp-test.c +++ b/tests/qmp-test.c @@ -13,13 +13,10 @@ #include "qemu/osdep.h" #include "libqtest.h" #include "qapi/error.h" -#include "qapi/qapi-visit-introspect.h" #include "qapi/qapi-visit-misc.h" #include "qapi/qmp/qdict.h" #include "qapi/qmp/qlist.h" #include "qapi/qobject-input-visitor.h" -#include "qapi/util.h" -#include "qapi/visitor.h" #include "qapi/qmp/qstring.h" =20 const char common_args[] =3D "-nodefaults -machine none"; @@ -253,184 +250,6 @@ static void test_qmp_oob(void) qtest_quit(qts); } =20 -/* Query smoke tests */ - -static int query_error_class(const char *cmd) -{ - static struct { - const char *cmd; - int err_class; - } fails[] =3D { - /* Success depends on build configuration: */ -#ifndef CONFIG_SPICE - { "query-spice", ERROR_CLASS_COMMAND_NOT_FOUND }, -#endif -#ifndef CONFIG_VNC - { "query-vnc", ERROR_CLASS_GENERIC_ERROR }, - { "query-vnc-servers", ERROR_CLASS_GENERIC_ERROR }, -#endif -#ifndef CONFIG_REPLICATION - { "query-xen-replication-status", ERROR_CLASS_COMMAND_NOT_FOUND }, -#endif - /* Likewise, and require special QEMU command-line arguments: */ - { "query-acpi-ospm-status", ERROR_CLASS_GENERIC_ERROR }, - { "query-balloon", ERROR_CLASS_DEVICE_NOT_ACTIVE }, - { "query-hotpluggable-cpus", ERROR_CLASS_GENERIC_ERROR }, - { "query-vm-generation-id", ERROR_CLASS_GENERIC_ERROR }, - { NULL, -1 } - }; - int i; - - for (i =3D 0; fails[i].cmd; i++) { - if (!strcmp(cmd, fails[i].cmd)) { - return fails[i].err_class; - } - } - return -1; -} - -static void test_query(const void *data) -{ - const char *cmd =3D data; - int expected_error_class =3D query_error_class(cmd); - QDict *resp, *error; - const char *error_class; - - qtest_start(common_args); - - resp =3D qmp("{ 'execute': %s }", cmd); - error =3D qdict_get_qdict(resp, "error"); - error_class =3D error ? qdict_get_str(error, "class") : NULL; - - if (expected_error_class < 0) { - g_assert(qdict_haskey(resp, "return")); - } else { - g_assert(error); - g_assert_cmpint(qapi_enum_parse(&QapiErrorClass_lookup, error_clas= s, - -1, &error_abort), - =3D=3D, expected_error_class); - } - qobject_unref(resp); - - qtest_end(); -} - -static bool query_is_blacklisted(const char *cmd) -{ - const char *blacklist[] =3D { - /* Not actually queries: */ - "add-fd", - /* Success depends on target arch: */ - "query-cpu-definitions", /* arm, i386, ppc, s390x */ - "query-gic-capabilities", /* arm */ - /* Success depends on target-specific build configuration: */ - "query-pci", /* CONFIG_PCI */ - /* Success depends on launching SEV guest */ - "query-sev-launch-measure", - /* Success depends on Host or Hypervisor SEV support */ - "query-sev", - "query-sev-capabilities", - NULL - }; - int i; - - for (i =3D 0; blacklist[i]; i++) { - if (!strcmp(cmd, blacklist[i])) { - return true; - } - } - return false; -} - -typedef struct { - SchemaInfoList *list; - GHashTable *hash; -} QmpSchema; - -static void qmp_schema_init(QmpSchema *schema) -{ - QDict *resp; - Visitor *qiv; - SchemaInfoList *tail; - - qtest_start(common_args); - resp =3D qmp("{ 'execute': 'query-qmp-schema' }"); - - qiv =3D qobject_input_visitor_new(qdict_get(resp, "return")); - visit_type_SchemaInfoList(qiv, NULL, &schema->list, &error_abort); - visit_free(qiv); - - qobject_unref(resp); - qtest_end(); - - schema->hash =3D g_hash_table_new(g_str_hash, g_str_equal); - - /* Build @schema: hash table mapping entity name to SchemaInfo */ - for (tail =3D schema->list; tail; tail =3D tail->next) { - g_hash_table_insert(schema->hash, tail->value->name, tail->value); - } -} - -static SchemaInfo *qmp_schema_lookup(QmpSchema *schema, const char *name) -{ - return g_hash_table_lookup(schema->hash, name); -} - -static void qmp_schema_cleanup(QmpSchema *schema) -{ - qapi_free_SchemaInfoList(schema->list); - g_hash_table_destroy(schema->hash); -} - -static bool object_type_has_mandatory_members(SchemaInfo *type) -{ - SchemaInfoObjectMemberList *tail; - - g_assert(type->meta_type =3D=3D SCHEMA_META_TYPE_OBJECT); - - for (tail =3D type->u.object.members; tail; tail =3D tail->next) { - if (!tail->value->has_q_default) { - return true; - } - } - - return false; -} - -static void add_query_tests(QmpSchema *schema) -{ - SchemaInfoList *tail; - SchemaInfo *si, *arg_type, *ret_type; - char *test_name; - - /* Test the query-like commands */ - for (tail =3D schema->list; tail; tail =3D tail->next) { - si =3D tail->value; - if (si->meta_type !=3D SCHEMA_META_TYPE_COMMAND) { - continue; - } - - if (query_is_blacklisted(si->name)) { - continue; - } - - arg_type =3D qmp_schema_lookup(schema, si->u.command.arg_type); - if (object_type_has_mandatory_members(arg_type)) { - continue; - } - - ret_type =3D qmp_schema_lookup(schema, si->u.command.ret_type); - if (ret_type->meta_type =3D=3D SCHEMA_META_TYPE_OBJECT - && !ret_type->u.object.members) { - continue; - } - - test_name =3D g_strdup_printf("qmp/%s", si->name); - qtest_add_data_func(test_name, si->name, test_query); - g_free(test_name); - } -} - /* Preconfig tests */ =20 static void test_qmp_preconfig(void) @@ -474,19 +293,11 @@ static void test_qmp_preconfig(void) =20 int main(int argc, char *argv[]) { - QmpSchema schema; - int ret; - g_test_init(&argc, &argv, NULL); =20 qtest_add_func("qmp/protocol", test_qmp_protocol); qtest_add_func("qmp/oob", test_qmp_oob); - qmp_schema_init(&schema); - add_query_tests(&schema); qtest_add_func("qmp/preconfig", test_qmp_preconfig); =20 - ret =3D g_test_run(); - - qmp_schema_cleanup(&schema); - return ret; + return g_test_run(); } --=20 2.17.1 From nobody Wed Nov 5 07:53:13 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 15337308681321017.9551044844993; Wed, 8 Aug 2018 05:21:08 -0700 (PDT) Received: from localhost ([::1]:43276 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNSc-0004sF-V0 for importer@patchew.org; Wed, 08 Aug 2018 08:21:07 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52441) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBm-0006gG-PC for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:51 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBh-0008Ps-RQ for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:42 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:51038 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBh-0008Ow-LF for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:37 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 33489402345E; Wed, 8 Aug 2018 12:03:37 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 82F5F1C71D; Wed, 8 Aug 2018 12:03:36 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 1C17A113865A; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:02:43 +0200 Message-Id: <20180808120334.10970-6-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:37 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:37 +0000 (UTC) for IP:'10.11.54.5' DOMAIN:'int-mx05.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 05/56] qmp-test: Cover syntax and lexical errors X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Signed-off-by: Markus Armbruster --- tests/libqtest.c | 17 +++++++++++++++++ tests/libqtest.h | 11 +++++++++++ tests/qmp-test.c | 39 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 67 insertions(+) diff --git a/tests/libqtest.c b/tests/libqtest.c index 3706f30aa2..c02fc91b37 100644 --- a/tests/libqtest.c +++ b/tests/libqtest.c @@ -586,6 +586,23 @@ void qtest_qmp_send(QTestState *s, const char *fmt, ..= .) va_end(ap); } =20 +void qtest_qmp_send_raw(QTestState *s, const char *fmt, ...) +{ + bool log =3D getenv("QTEST_LOG") !=3D NULL; + va_list ap; + char *str; + + va_start(ap, fmt); + str =3D g_strdup_vprintf(fmt, ap); + va_end(ap); + + if (log) { + fprintf(stderr, "%s", str); + } + socket_send(s->qmp_fd, str, strlen(str)); + g_free(str); +} + QDict *qtest_qmp_eventwait_ref(QTestState *s, const char *event) { QDict *response; diff --git a/tests/libqtest.h b/tests/libqtest.h index def1edaafa..1e831973ff 100644 --- a/tests/libqtest.h +++ b/tests/libqtest.h @@ -96,6 +96,17 @@ QDict *qtest_qmp(QTestState *s, const char *fmt, ...) void qtest_qmp_send(QTestState *s, const char *fmt, ...) GCC_FMT_ATTR(2, 3); =20 +/** + * qtest_qmp_send_raw: + * @s: #QTestState instance to operate on. + * @fmt...: text to send, formatted like sprintf() + * + * Sends text to the QMP monitor verbatim. Need not be valid JSON; + * this is useful for negative tests. + */ +void qtest_qmp_send_raw(QTestState *s, const char *fmt, ...) + GCC_FMT_ATTR(2, 3); + /** * qtest_qmpv: * @s: #QTestState instance to operate on. diff --git a/tests/qmp-test.c b/tests/qmp-test.c index b6eff4fe97..5e56be105e 100644 --- a/tests/qmp-test.c +++ b/tests/qmp-test.c @@ -42,10 +42,49 @@ static void test_version(QObject *version) visit_free(v); } =20 +static bool recovered(QTestState *qts) +{ + QDict *resp; + bool ret; + + resp =3D qtest_qmp(qts, "{ 'execute': 'no-such-cmd' }"); + ret =3D !strcmp(get_error_class(resp), "CommandNotFound"); + qobject_unref(resp); + return ret; +} + static void test_malformed(QTestState *qts) { QDict *resp; =20 + /* syntax error */ + qtest_qmp_send_raw(qts, "{]\n"); + resp =3D qtest_qmp_receive(qts); + g_assert_cmpstr(get_error_class(resp), =3D=3D, "GenericError"); + qobject_unref(resp); + g_assert(recovered(qts)); + + /* lexical error: impossible byte outside string */ + qtest_qmp_send_raw(qts, "{\xFF"); + resp =3D qtest_qmp_receive(qts); + g_assert_cmpstr(get_error_class(resp), =3D=3D, "GenericError"); + qobject_unref(resp); + g_assert(recovered(qts)); + + /* lexical error: impossible byte in string */ + qtest_qmp_send_raw(qts, "{'bad \xFF"); + resp =3D qtest_qmp_receive(qts); + g_assert_cmpstr(get_error_class(resp), =3D=3D, "GenericError"); + qobject_unref(resp); + g_assert(recovered(qts)); + + /* lexical error: interpolation */ + qtest_qmp_send_raw(qts, "%%p\n"); + resp =3D qtest_qmp_receive(qts); + g_assert_cmpstr(get_error_class(resp), =3D=3D, "GenericError"); + qobject_unref(resp); + g_assert(recovered(qts)); + /* Not even a dictionary */ resp =3D qtest_qmp(qts, "null"); g_assert_cmpstr(get_error_class(resp), =3D=3D, "GenericError"); --=20 2.17.1 From nobody Wed Nov 5 07:53:13 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533730448059787.6400950410277; Wed, 8 Aug 2018 05:14:08 -0700 (PDT) Received: from localhost ([::1]:43207 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNLq-000774-Rg for importer@patchew.org; Wed, 08 Aug 2018 08:14:06 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52346) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBk-0006eC-87 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:44 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBh-0008PC-H8 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:40 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:47668 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBh-0008O9-CN for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:37 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id DF1DB7A7E8; Wed, 8 Aug 2018 12:03:36 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8487F2156897; Wed, 8 Aug 2018 12:03:36 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 1F8CB1138662; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:02:44 +0200 Message-Id: <20180808120334.10970-7-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 08 Aug 2018 12:03:36 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 08 Aug 2018 12:03:36 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 06/56] test-qga: Clean up how we test QGA synchronization X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" To permit recovering from arbitrary JSON parse errors, the JSON parser resets itself on lexical errors. We recommend sending a 0xff byte for that purpose, and test-qga covers this usage since commit 5229564b832. That commit had to add an ugly hack to qmp_fd_vsend() to make capable of sending this byte (it's designed to send only valid JSON). The previous commit added a way to send arbitrary text. Put that to use for this purpose, and drop the hack from qmp_fd_vsend(). Signed-off-by: Markus Armbruster --- tests/libqtest.c | 39 +++++++++++++++++++++------------------ tests/libqtest.h | 2 ++ tests/test-qga.c | 8 ++++---- 3 files changed, 27 insertions(+), 22 deletions(-) diff --git a/tests/libqtest.c b/tests/libqtest.c index c02fc91b37..9c844874e4 100644 --- a/tests/libqtest.c +++ b/tests/libqtest.c @@ -489,16 +489,6 @@ void qmp_fd_vsend(int fd, const char *fmt, va_list ap) { QObject *qobj; =20 - /* - * qobject_from_vjsonf_nofail() chokes on leading 0xff as invalid - * JSON, but tests/test-qga.c needs to send that to test QGA - * synchronization - */ - if (*fmt =3D=3D '\377') { - socket_send(fd, fmt, 1); - fmt++; - } - /* Going through qobject ensures we escape strings properly */ qobj =3D qobject_from_vjsonf_nofail(fmt, ap); =20 @@ -586,23 +576,36 @@ void qtest_qmp_send(QTestState *s, const char *fmt, .= ..) va_end(ap); } =20 -void qtest_qmp_send_raw(QTestState *s, const char *fmt, ...) +void qmp_fd_vsend_raw(int fd, const char *fmt, va_list ap) { bool log =3D getenv("QTEST_LOG") !=3D NULL; - va_list ap; - char *str; - - va_start(ap, fmt); - str =3D g_strdup_vprintf(fmt, ap); - va_end(ap); + char *str =3D g_strdup_vprintf(fmt, ap); =20 if (log) { fprintf(stderr, "%s", str); } - socket_send(s->qmp_fd, str, strlen(str)); + socket_send(fd, str, strlen(str)); g_free(str); } =20 +void qmp_fd_send_raw(int fd, const char *fmt, ...) +{ + va_list ap; + + va_start(ap, fmt); + qmp_fd_vsend_raw(fd, fmt, ap); + va_end(ap); +} + +void qtest_qmp_send_raw(QTestState *s, const char *fmt, ...) +{ + va_list ap; + + va_start(ap, fmt); + qmp_fd_vsend_raw(s->qmp_fd, fmt, ap); + va_end(ap); +} + QDict *qtest_qmp_eventwait_ref(QTestState *s, const char *event) { QDict *response; diff --git a/tests/libqtest.h b/tests/libqtest.h index 1e831973ff..2d1eb4b282 100644 --- a/tests/libqtest.h +++ b/tests/libqtest.h @@ -959,6 +959,8 @@ static inline int64_t clock_set(int64_t val) QDict *qmp_fd_receive(int fd); void qmp_fd_vsend(int fd, const char *fmt, va_list ap) GCC_FMT_ATTR(2, 0); void qmp_fd_send(int fd, const char *fmt, ...) GCC_FMT_ATTR(2, 3); +void qmp_fd_send_raw(int fd, const char *fmt, ...) GCC_FMT_ATTR(2, 3); +void qmp_fd_vsend_raw(int fd, const char *fmt, va_list ap) GCC_FMT_ATTR(2,= 0); QDict *qmp_fdv(int fd, const char *fmt, va_list ap) GCC_FMT_ATTR(2, 0); QDict *qmp_fd(int fd, const char *fmt, ...) GCC_FMT_ATTR(2, 3); =20 diff --git a/tests/test-qga.c b/tests/test-qga.c index c552cc0125..4e51898d23 100644 --- a/tests/test-qga.c +++ b/tests/test-qga.c @@ -147,10 +147,10 @@ static void test_qga_sync_delimited(gconstpointer fix) unsigned char c; QDict *ret; =20 - qmp_fd_send(fixture->fd, - "\xff{'execute': 'guest-sync-delimited'," - " 'arguments': {'id': %u } }", - r); + qmp_fd_send_raw(fixture->fd, + "\xff{'execute': 'guest-sync-delimited'," + " 'arguments': {'id': %u } }", + r); =20 /* * Read and ignore garbage until resynchronized. --=20 2.17.1 From nobody Wed Nov 5 07:53:13 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533730264970555.3951948968954; Wed, 8 Aug 2018 05:11:04 -0700 (PDT) Received: from localhost ([::1]:43163 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNIp-0004Fg-4Q for importer@patchew.org; Wed, 08 Aug 2018 08:10:59 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52349) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBk-0006eD-9A for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:47 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBi-0008Qi-4S for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:40 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:45366 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBh-0008PM-UN for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:38 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 6A8C981663D8; Wed, 8 Aug 2018 12:03:37 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 84C2621568A1; Wed, 8 Aug 2018 12:03:36 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 2303B1138663; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:02:45 +0200 Message-Id: <20180808120334.10970-8-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 08 Aug 2018 12:03:37 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 08 Aug 2018 12:03:37 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 07/56] check-qjson: Cover escaped characters more thoroughly, part 1 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" escaped_string() first tests double quoted strings, then repeats a few tests with single quotes. Repeat all of them: store the strings to test without quotes, and wrap them in either kind of quote for testing. Signed-off-by: Markus Armbruster --- tests/check-qjson.c | 94 ++++++++++++++++++++++++++------------------- 1 file changed, 55 insertions(+), 39 deletions(-) diff --git a/tests/check-qjson.c b/tests/check-qjson.c index 0a9a054c7b..1c7f24bc4d 100644 --- a/tests/check-qjson.c +++ b/tests/check-qjson.c @@ -22,55 +22,71 @@ #include "qapi/qmp/qstring.h" #include "qemu-common.h" =20 +static QString *from_json_str(const char *jstr, Error **errp, bool single) +{ + char quote =3D single ? '\'' : '"'; + char *qjstr =3D g_strdup_printf("%c%s%c", quote, jstr, quote); + + return qobject_to(QString, qobject_from_json(qjstr, errp)); +} + +static char *to_json_str(QString *str) +{ + QString *json =3D qobject_to_json(QOBJECT(str)); + char *jstr; + + if (!json) { + return NULL; + } + /* peel off double quotes */ + jstr =3D g_strndup(qstring_get_str(json) + 1, + qstring_get_length(json) - 2); + qobject_unref(json); + return jstr; +} + static void escaped_string(void) { - int i; struct { - const char *encoded; - const char *decoded; + /* Content of JSON string to parse with qobject_from_json() */ + const char *json_in; + /* Expected parse output; to unparse with qobject_to_json() */ + const char *utf8_out; int skip; } test_cases[] =3D { - { "\"\\b\"", "\b" }, - { "\"\\f\"", "\f" }, - { "\"\\n\"", "\n" }, - { "\"\\r\"", "\r" }, - { "\"\\t\"", "\t" }, - { "\"/\"", "/" }, - { "\"\\/\"", "/", .skip =3D 1 }, - { "\"\\\\\"", "\\" }, - { "\"\\\"\"", "\"" }, - { "\"hello world \\\"embedded string\\\"\"", + { "\\b", "\b" }, + { "\\f", "\f" }, + { "\\n", "\n" }, + { "\\r", "\r" }, + { "\\t", "\t" }, + { "/", "/" }, + { "\\/", "/", .skip =3D 1 }, + { "\\\\", "\\" }, + { "\\\"", "\"" }, + { "hello world \\\"embedded string\\\"", "hello world \"embedded string\"" }, - { "\"hello world\\nwith new line\"", "hello world\nwith new line" = }, - { "\"single byte utf-8 \\u0020\"", "single byte utf-8 ", .skip = =3D 1 }, - { "\"double byte utf-8 \\u00A2\"", "double byte utf-8 \xc2\xa2" }, - { "\"triple byte utf-8 \\u20AC\"", "triple byte utf-8 \xe2\x82\xac= " }, - { "'\\b'", "\b", .skip =3D 1 }, - { "'\\f'", "\f", .skip =3D 1 }, - { "'\\n'", "\n", .skip =3D 1 }, - { "'\\r'", "\r", .skip =3D 1 }, - { "'\\t'", "\t", .skip =3D 1 }, - { "'\\/'", "/", .skip =3D 1 }, - { "'\\\\'", "\\", .skip =3D 1 }, + { "hello world\\nwith new line", "hello world\nwith new line" }, + { "single byte utf-8 \\u0020", "single byte utf-8 ", .skip =3D 1 = }, + { "double byte utf-8 \\u00A2", "double byte utf-8 \xc2\xa2" }, + { "triple byte utf-8 \\u20AC", "triple byte utf-8 \xe2\x82\xac" }, {} }; + int i, j; + QString *cstr; + char *jstr; =20 - for (i =3D 0; test_cases[i].encoded; i++) { - QObject *obj; - QString *str; - - obj =3D qobject_from_json(test_cases[i].encoded, &error_abort); - str =3D qobject_to(QString, obj); - g_assert(str); - g_assert_cmpstr(qstring_get_str(str), =3D=3D, test_cases[i].decode= d); - - if (test_cases[i].skip =3D=3D 0) { - str =3D qobject_to_json(obj); - g_assert_cmpstr(qstring_get_str(str), =3D=3D, test_cases[i].en= coded); - qobject_unref(obj); + for (i =3D 0; test_cases[i].json_in; i++) { + for (j =3D 0; j < 2; j++) { + cstr =3D from_json_str(test_cases[i].json_in, &error_abort, j); + g_assert_cmpstr(qstring_get_try_str(cstr), + =3D=3D, test_cases[i].utf8_out); + if (test_cases[i].skip =3D=3D 0) { + jstr =3D to_json_str(cstr); + g_assert_cmpstr(jstr, =3D=3D, test_cases[i].json_in); + g_free(jstr); + } + qobject_unref(cstr); } - - qobject_unref(str); } } =20 --=20 2.17.1 From nobody Wed Nov 5 07:53:13 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533731522193867.5719911859399; Wed, 8 Aug 2018 05:32:02 -0700 (PDT) Received: from localhost ([::1]:43339 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNdB-00067Q-2Y for importer@patchew.org; Wed, 08 Aug 2018 08:32:01 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52396) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBl-0006fD-MW for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:51 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBh-0008Ob-AW for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:41 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:47662 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBh-0008O4-6K for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:37 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id B84C37A7E7; Wed, 8 Aug 2018 12:03:36 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 864AF20180F5; Wed, 8 Aug 2018 12:03:36 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 268501138664; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:02:46 +0200 Message-Id: <20180808120334.10970-9-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 08 Aug 2018 12:03:36 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 08 Aug 2018 12:03:36 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 08/56] check-qjson: Streamline escaped_string()'s test strings X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Merge a few closely related test strings, and drop a few redundant ones. Signed-off-by: Markus Armbruster Reviewed-by: Eric Blake --- tests/check-qjson.c | 14 ++------------ 1 file changed, 2 insertions(+), 12 deletions(-) diff --git a/tests/check-qjson.c b/tests/check-qjson.c index 1c7f24bc4d..8f51f57af9 100644 --- a/tests/check-qjson.c +++ b/tests/check-qjson.c @@ -54,18 +54,8 @@ static void escaped_string(void) const char *utf8_out; int skip; } test_cases[] =3D { - { "\\b", "\b" }, - { "\\f", "\f" }, - { "\\n", "\n" }, - { "\\r", "\r" }, - { "\\t", "\t" }, - { "/", "/" }, - { "\\/", "/", .skip =3D 1 }, - { "\\\\", "\\" }, - { "\\\"", "\"" }, - { "hello world \\\"embedded string\\\"", - "hello world \"embedded string\"" }, - { "hello world\\nwith new line", "hello world\nwith new line" }, + { "\\b\\f\\n\\r\\t\\\\\\\"", "\b\f\n\r\t\\\"" }, + { "\\/\\'", "/'", .skip =3D 1 }, { "single byte utf-8 \\u0020", "single byte utf-8 ", .skip =3D 1 = }, { "double byte utf-8 \\u00A2", "double byte utf-8 \xc2\xa2" }, { "triple byte utf-8 \\u20AC", "triple byte utf-8 \xe2\x82\xac" }, --=20 2.17.1 From nobody Wed Nov 5 07:53:13 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533730851600558.599032709348; Wed, 8 Aug 2018 05:20:51 -0700 (PDT) Received: from localhost ([::1]:43275 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNSM-0004cJ-9u for importer@patchew.org; Wed, 08 Aug 2018 08:20:50 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52340) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBk-0006e9-6c for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBh-0008PI-J7 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:40 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:51034 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBh-0008OB-Dm for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:37 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D8B92402346B; Wed, 8 Aug 2018 12:03:36 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 86A5920180F6; Wed, 8 Aug 2018 12:03:36 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 29F031138666; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:02:47 +0200 Message-Id: <20180808120334.10970-10-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:36 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:36 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 09/56] check-qjson: Cover escaped characters more thoroughly, part 2 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Cover surrogates, invalid escapes, and noncharacters. This demonstrates that valid surrogate pairs are misinterpreted, and invalid surrogates and noncharacters aren't rejected. Signed-off-by: Markus Armbruster Reviewed-by: Eric Blake --- tests/check-qjson.c | 53 ++++++++++++++++++++++++++++++++++++++------- 1 file changed, 45 insertions(+), 8 deletions(-) diff --git a/tests/check-qjson.c b/tests/check-qjson.c index 8f51f57af9..e899e2d361 100644 --- a/tests/check-qjson.c +++ b/tests/check-qjson.c @@ -59,6 +59,38 @@ static void escaped_string(void) { "single byte utf-8 \\u0020", "single byte utf-8 ", .skip =3D 1 = }, { "double byte utf-8 \\u00A2", "double byte utf-8 \xc2\xa2" }, { "triple byte utf-8 \\u20AC", "triple byte utf-8 \xe2\x82\xac" }, + { "quadruple byte utf-8 \\uD834\\uDD1E", /* U+1D11E */ + /* bug: want \xF0\x9D\x84\x9E */ + "quadruple byte utf-8 \xED\xA0\xB4\xED\xB4\x9E", .skip =3D 1 }, + { "\\z", NULL }, + { "\\ux", NULL }, + { "\\u1x", NULL }, + { "\\u12x", NULL }, + { "\\u123x", NULL }, + { "\\u12345", "\341\210\2645" }, + { "\\u12345", "\341\210\2645" }, + { "\\u0000x", "x", .skip =3D 1}, /* bug: want \xC0\x80x */ + { "unpaired leading surrogate \\uD800\\uD800", + /* bug: not rejected */ + "unpaired leading surrogate \355\240\200\355\240\200", .skip =3D= 1 }, + { "unpaired trailing surrogate \\uDC00\\uDC00", + /* bug: not rejected */ + "unpaired trailing surrogate \355\260\200\355\260\200", .skip = =3D 1}, + { "backward surrogate pair \\uDC00\\uD800", + /* bug: not rejected */ + "backward surrogate pair \355\260\200\355\240\200", .skip =3D 1}, + { "noncharacter U+FDD0 \\uFDD0", + /* bug: not rejected */ + "noncharacter U+FDD0 \xEF\xB7\x90", .skip =3D 1}, + { "noncharacter U+FDEF \\uFDEF", + /* bug: not rejected */ + "noncharacter U+FDEF \xEF\xB7\xAF", .skip =3D 1}, + { "noncharacter U+1FFFE \\uD87F\\uDFFE", + /* bug: not rejected */ + "noncharacter U+1FFFE \xED\xA1\xBF\xED\xBF\xBE", .skip =3D 1}, + { "noncharacter U+10FFFF \\uDC3F\\uDFFF", + /* bug: not rejected */ + "noncharacter U+10FFFF \xED\xB0\xBF\xED\xBF\xBF", .skip =3D 1}, {} }; int i, j; @@ -67,15 +99,20 @@ static void escaped_string(void) =20 for (i =3D 0; test_cases[i].json_in; i++) { for (j =3D 0; j < 2; j++) { - cstr =3D from_json_str(test_cases[i].json_in, &error_abort, j); - g_assert_cmpstr(qstring_get_try_str(cstr), - =3D=3D, test_cases[i].utf8_out); - if (test_cases[i].skip =3D=3D 0) { - jstr =3D to_json_str(cstr); - g_assert_cmpstr(jstr, =3D=3D, test_cases[i].json_in); - g_free(jstr); + if (test_cases[i].utf8_out) { + cstr =3D from_json_str(test_cases[i].json_in, &error_abort= , j); + g_assert_cmpstr(qstring_get_try_str(cstr), + =3D=3D, test_cases[i].utf8_out); + if (!test_cases[i].skip) { + jstr =3D to_json_str(cstr); + g_assert_cmpstr(jstr, =3D=3D, test_cases[i].json_in); + g_free(jstr); + } + qobject_unref(cstr); + } else { + cstr =3D from_json_str(test_cases[i].json_in, NULL, j); + g_assert(!cstr); } - qobject_unref(cstr); } } } --=20 2.17.1 From nobody Wed Nov 5 07:53:13 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533731349126250.58439862144758; Wed, 8 Aug 2018 05:29:09 -0700 (PDT) Received: from localhost ([::1]:43322 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNaJ-0003RX-TT for importer@patchew.org; Wed, 08 Aug 2018 08:29:04 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52352) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBk-0006eG-AR for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:45 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBh-0008Pm-Qu for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:40 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:47674 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBh-0008Os-Km for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:37 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 349707A7EB; Wed, 8 Aug 2018 12:03:37 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8AAFE1040840; Wed, 8 Aug 2018 12:03:36 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 2D6FE1138667; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:02:48 +0200 Message-Id: <20180808120334.10970-11-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 08 Aug 2018 12:03:37 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 08 Aug 2018 12:03:37 +0000 (UTC) for IP:'10.11.54.3' DOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 10/56] check-qjson: Drop redundant string tests X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" simple_string() and single_quote_string() add nothing to escaped_string() anymore. Drop them. Signed-off-by: Markus Armbruster Reviewed-by: Eric Blake --- tests/check-qjson.c | 59 --------------------------------------------- 1 file changed, 59 deletions(-) diff --git a/tests/check-qjson.c b/tests/check-qjson.c index e899e2d361..f0e8967a53 100644 --- a/tests/check-qjson.c +++ b/tests/check-qjson.c @@ -117,63 +117,6 @@ static void escaped_string(void) } } =20 -static void simple_string(void) -{ - int i; - struct { - const char *encoded; - const char *decoded; - } test_cases[] =3D { - { "\"hello world\"", "hello world" }, - { "\"the quick brown fox jumped over the fence\"", - "the quick brown fox jumped over the fence" }, - {} - }; - - for (i =3D 0; test_cases[i].encoded; i++) { - QObject *obj; - QString *str; - - obj =3D qobject_from_json(test_cases[i].encoded, &error_abort); - str =3D qobject_to(QString, obj); - g_assert(str); - g_assert(strcmp(qstring_get_str(str), test_cases[i].decoded) =3D= =3D 0); - - str =3D qobject_to_json(obj); - g_assert(strcmp(qstring_get_str(str), test_cases[i].encoded) =3D= =3D 0); - - qobject_unref(obj); - =20 - qobject_unref(str); - } -} - -static void single_quote_string(void) -{ - int i; - struct { - const char *encoded; - const char *decoded; - } test_cases[] =3D { - { "'hello world'", "hello world" }, - { "'the quick brown fox \\' jumped over the fence'", - "the quick brown fox ' jumped over the fence" }, - {} - }; - - for (i =3D 0; test_cases[i].encoded; i++) { - QObject *obj; - QString *str; - - obj =3D qobject_from_json(test_cases[i].encoded, &error_abort); - str =3D qobject_to(QString, obj); - g_assert(str); - g_assert(strcmp(qstring_get_str(str), test_cases[i].decoded) =3D= =3D 0); - - qobject_unref(str); - } -} - static void utf8_string(void) { /* @@ -1512,10 +1455,8 @@ int main(int argc, char **argv) { g_test_init(&argc, &argv, NULL); =20 - g_test_add_func("/literals/string/simple", simple_string); g_test_add_func("/literals/string/escaped", escaped_string); g_test_add_func("/literals/string/utf8", utf8_string); - g_test_add_func("/literals/string/single_quote", single_quote_string); g_test_add_func("/literals/string/vararg", vararg_string); =20 g_test_add_func("/literals/number/simple", simple_number); --=20 2.17.1 From nobody Wed Nov 5 07:53:13 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533732346432842.7635359352402; Wed, 8 Aug 2018 05:45:46 -0700 (PDT) Received: from localhost ([::1]:43411 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNqJ-00012K-Vi for importer@patchew.org; Wed, 08 Aug 2018 08:45:36 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52704) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBu-0006ow-RP for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:04:01 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBj-0008TK-7u for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:50 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:51060 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBi-0008RU-Ph for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:39 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 486744023475; Wed, 8 Aug 2018 12:03:38 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id B64411C716; Wed, 8 Aug 2018 12:03:37 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 30F391138668; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:02:49 +0200 Message-Id: <20180808120334.10970-12-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) for IP:'10.11.54.5' DOMAIN:'int-mx05.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 11/56] check-qjson: Cover UTF-8 in single quoted strings X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" utf8_string() tests only double quoted strings. Cover single quoted strings, too: store the strings to test without quotes, then wrap them in either kind of quote. Signed-off-by: Markus Armbruster --- tests/check-qjson.c | 427 ++++++++++++++++++++++---------------------- 1 file changed, 214 insertions(+), 213 deletions(-) diff --git a/tests/check-qjson.c b/tests/check-qjson.c index f0e8967a53..75f0a9f18a 100644 --- a/tests/check-qjson.c +++ b/tests/check-qjson.c @@ -144,10 +144,14 @@ static void utf8_string(void) * http://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt */ static const struct { + /* Content of JSON string to parse with qobject_from_json() */ const char *json_in; + /* Expected parse output */ const char *utf8_out; - const char *json_out; /* defaults to @json_in */ - const char *utf8_in; /* defaults to @utf8_out */ + /* Expected unparse output, defaults to @json_in */ + const char *json_out; + /* Expected parse output for @json_out, defaults to @utf8_out */ + const char *utf8_in; } test_cases[] =3D { /* * Bug markers used here: @@ -165,72 +169,72 @@ static void utf8_string(void) /* 1 Some correct UTF-8 text */ { /* a bit of German */ - "\"Falsches \xC3\x9C" "ben von Xylophonmusik qu\xC3\xA4lt" - " jeden gr\xC3\xB6\xC3\x9F" "eren Zwerg.\"", "Falsches \xC3\x9C" "ben von Xylophonmusik qu\xC3\xA4lt" " jeden gr\xC3\xB6\xC3\x9F" "eren Zwerg.", - "\"Falsches \\u00DCben von Xylophonmusik qu\\u00E4lt" - " jeden gr\\u00F6\\u00DFeren Zwerg.\"", + "Falsches \xC3\x9C" "ben von Xylophonmusik qu\xC3\xA4lt" + " jeden gr\xC3\xB6\xC3\x9F" "eren Zwerg.", + "Falsches \\u00DCben von Xylophonmusik qu\\u00E4lt" + " jeden gr\\u00F6\\u00DFeren Zwerg.", }, { /* a bit of Greek */ - "\"\xCE\xBA\xE1\xBD\xB9\xCF\x83\xCE\xBC\xCE\xB5\"", "\xCE\xBA\xE1\xBD\xB9\xCF\x83\xCE\xBC\xCE\xB5", - "\"\\u03BA\\u1F79\\u03C3\\u03BC\\u03B5\"", + "\xCE\xBA\xE1\xBD\xB9\xCF\x83\xCE\xBC\xCE\xB5", + "\\u03BA\\u1F79\\u03C3\\u03BC\\u03B5", }, /* 2 Boundary condition test cases */ /* 2.1 First possible sequence of a certain length */ /* 2.1.1 1 byte U+0000 */ { - "\"\\u0000\"", + "\\u0000", "", /* bug: want overlong "\xC0\x80" */ - "\"\\u0000\"", + "\\u0000", "\xC0\x80", }, /* 2.1.2 2 bytes U+0080 */ { - "\"\xC2\x80\"", "\xC2\x80", - "\"\\u0080\"", + "\xC2\x80", + "\\u0080", }, /* 2.1.3 3 bytes U+0800 */ { - "\"\xE0\xA0\x80\"", "\xE0\xA0\x80", - "\"\\u0800\"", + "\xE0\xA0\x80", + "\\u0800", }, /* 2.1.4 4 bytes U+10000 */ { - "\"\xF0\x90\x80\x80\"", "\xF0\x90\x80\x80", - "\"\\uD800\\uDC00\"", + "\xF0\x90\x80\x80", + "\\uD800\\uDC00", }, /* 2.1.5 5 bytes U+200000 */ { - "\"\xF8\x88\x80\x80\x80\"", + "\xF8\x88\x80\x80\x80", NULL, /* bug: rejected */ - "\"\\uFFFD\"", + "\\uFFFD", "\xF8\x88\x80\x80\x80", }, /* 2.1.6 6 bytes U+4000000 */ { - "\"\xFC\x84\x80\x80\x80\x80\"", + "\xFC\x84\x80\x80\x80\x80", NULL, /* bug: rejected */ - "\"\\uFFFD\"", + "\\uFFFD", "\xFC\x84\x80\x80\x80\x80", }, /* 2.2 Last possible sequence of a certain length */ /* 2.2.1 1 byte U+007F */ { - "\"\x7F\"", "\x7F", - "\"\\u007F\"", + "\x7F", + "\\u007F", }, /* 2.2.2 2 bytes U+07FF */ { - "\"\xDF\xBF\"", "\xDF\xBF", - "\"\\u07FF\"", + "\xDF\xBF", + "\\u07FF", }, /* * 2.2.3 3 bytes U+FFFC @@ -242,122 +246,122 @@ static void utf8_string(void) * U+FFFC here. */ { - "\"\xEF\xBF\xBC\"", "\xEF\xBF\xBC", - "\"\\uFFFC\"", + "\xEF\xBF\xBC", + "\\uFFFC", }, /* 2.2.4 4 bytes U+1FFFFF */ { - "\"\xF7\xBF\xBF\xBF\"", + "\xF7\xBF\xBF\xBF", NULL, /* bug: rejected */ - "\"\\uFFFD\"", + "\\uFFFD", "\xF7\xBF\xBF\xBF", }, /* 2.2.5 5 bytes U+3FFFFFF */ { - "\"\xFB\xBF\xBF\xBF\xBF\"", + "\xFB\xBF\xBF\xBF\xBF", NULL, /* bug: rejected */ - "\"\\uFFFD\"", + "\\uFFFD", "\xFB\xBF\xBF\xBF\xBF", }, /* 2.2.6 6 bytes U+7FFFFFFF */ { - "\"\xFD\xBF\xBF\xBF\xBF\xBF\"", + "\xFD\xBF\xBF\xBF\xBF\xBF", NULL, /* bug: rejected */ - "\"\\uFFFD\"", + "\\uFFFD", "\xFD\xBF\xBF\xBF\xBF\xBF", }, /* 2.3 Other boundary conditions */ { /* last one before surrogate range: U+D7FF */ - "\"\xED\x9F\xBF\"", "\xED\x9F\xBF", - "\"\\uD7FF\"", + "\xED\x9F\xBF", + "\\uD7FF", }, { /* first one after surrogate range: U+E000 */ - "\"\xEE\x80\x80\"", "\xEE\x80\x80", - "\"\\uE000\"", + "\xEE\x80\x80", + "\\uE000", }, { /* last one in BMP: U+FFFD */ - "\"\xEF\xBF\xBD\"", "\xEF\xBF\xBD", - "\"\\uFFFD\"", + "\xEF\xBF\xBD", + "\\uFFFD", }, { /* last one in last plane: U+10FFFD */ - "\"\xF4\x8F\xBF\xBD\"", "\xF4\x8F\xBF\xBD", - "\"\\uDBFF\\uDFFD\"" + "\xF4\x8F\xBF\xBD", + "\\uDBFF\\uDFFD" }, { /* first one beyond Unicode range: U+110000 */ - "\"\xF4\x90\x80\x80\"", "\xF4\x90\x80\x80", - "\"\\uFFFD\"", + "\xF4\x90\x80\x80", + "\\uFFFD", }, /* 3 Malformed sequences */ /* 3.1 Unexpected continuation bytes */ /* 3.1.1 First continuation byte */ { - "\"\x80\"", + "\x80", "\x80", /* bug: not corrected */ - "\"\\uFFFD\"", + "\\uFFFD", }, /* 3.1.2 Last continuation byte */ { - "\"\xBF\"", + "\xBF", "\xBF", /* bug: not corrected */ - "\"\\uFFFD\"", + "\\uFFFD", }, /* 3.1.3 2 continuation bytes */ { - "\"\x80\xBF\"", + "\x80\xBF", "\x80\xBF", /* bug: not corrected */ - "\"\\uFFFD\\uFFFD\"", + "\\uFFFD\\uFFFD", }, /* 3.1.4 3 continuation bytes */ { - "\"\x80\xBF\x80\"", + "\x80\xBF\x80", "\x80\xBF\x80", /* bug: not corrected */ - "\"\\uFFFD\\uFFFD\\uFFFD\"", + "\\uFFFD\\uFFFD\\uFFFD", }, /* 3.1.5 4 continuation bytes */ { - "\"\x80\xBF\x80\xBF\"", + "\x80\xBF\x80\xBF", "\x80\xBF\x80\xBF", /* bug: not corrected */ - "\"\\uFFFD\\uFFFD\\uFFFD\\uFFFD\"", + "\\uFFFD\\uFFFD\\uFFFD\\uFFFD", }, /* 3.1.6 5 continuation bytes */ { - "\"\x80\xBF\x80\xBF\x80\"", + "\x80\xBF\x80\xBF\x80", "\x80\xBF\x80\xBF\x80", /* bug: not corrected */ - "\"\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\"", + "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD", }, /* 3.1.7 6 continuation bytes */ { - "\"\x80\xBF\x80\xBF\x80\xBF\"", + "\x80\xBF\x80\xBF\x80\xBF", "\x80\xBF\x80\xBF\x80\xBF", /* bug: not corrected */ - "\"\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\"", + "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD", }, /* 3.1.8 7 continuation bytes */ { - "\"\x80\xBF\x80\xBF\x80\xBF\x80\"", + "\x80\xBF\x80\xBF\x80\xBF\x80", "\x80\xBF\x80\xBF\x80\xBF\x80", /* bug: not corrected */ - "\"\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\"", + "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD", }, /* 3.1.9 Sequence of all 64 possible continuation bytes */ { - "\"\x80\x81\x82\x83\x84\x85\x86\x87" + "\x80\x81\x82\x83\x84\x85\x86\x87" "\x88\x89\x8A\x8B\x8C\x8D\x8E\x8F" "\x90\x91\x92\x93\x94\x95\x96\x97" "\x98\x99\x9A\x9B\x9C\x9D\x9E\x9F" "\xA0\xA1\xA2\xA3\xA4\xA5\xA6\xA7" "\xA8\xA9\xAA\xAB\xAC\xAD\xAE\xAF" "\xB0\xB1\xB2\xB3\xB4\xB5\xB6\xB7" - "\xB8\xB9\xBA\xBB\xBC\xBD\xBE\xBF\"", + "\xB8\xB9\xBA\xBB\xBC\xBD\xBE\xBF", /* bug: not corrected */ "\x80\x81\x82\x83\x84\x85\x86\x87" "\x88\x89\x8A\x8B\x8C\x8D\x8E\x8F" @@ -367,27 +371,27 @@ static void utf8_string(void) "\xA8\xA9\xAA\xAB\xAC\xAD\xAE\xAF" "\xB0\xB1\xB2\xB3\xB4\xB5\xB6\xB7" "\xB8\xB9\xBA\xBB\xBC\xBD\xBE\xBF", - "\"\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD" "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD" "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD" "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD" "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD" "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD" "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD" - "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\"" + "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD" + "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD" }, /* 3.2 Lonely start characters */ /* 3.2.1 All 32 first bytes of 2-byte sequences, followed by spac= e */ { - "\"\xC0 \xC1 \xC2 \xC3 \xC4 \xC5 \xC6 \xC7 " + "\xC0 \xC1 \xC2 \xC3 \xC4 \xC5 \xC6 \xC7 " "\xC8 \xC9 \xCA \xCB \xCC \xCD \xCE \xCF " "\xD0 \xD1 \xD2 \xD3 \xD4 \xD5 \xD6 \xD7 " - "\xD8 \xD9 \xDA \xDB \xDC \xDD \xDE \xDF \"", + "\xD8 \xD9 \xDA \xDB \xDC \xDD \xDE \xDF ", NULL, /* bug: rejected */ - "\"\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\u= FFFD " "\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFF= FD " "\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFF= FD " - "\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFF= FD \"", + "\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFF= FD " + "\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFF= FD ", "\xC0 \xC1 \xC2 \xC3 \xC4 \xC5 \xC6 \xC7 " "\xC8 \xC9 \xCA \xCB \xCC \xCD \xCE \xCF " "\xD0 \xD1 \xD2 \xD3 \xD4 \xD5 \xD6 \xD7 " @@ -395,159 +399,159 @@ static void utf8_string(void) }, /* 3.2.2 All 16 first bytes of 3-byte sequences, followed by spac= e */ { - "\"\xE0 \xE1 \xE2 \xE3 \xE4 \xE5 \xE6 \xE7 " - "\xE8 \xE9 \xEA \xEB \xEC \xED \xEE \xEF \"", + "\xE0 \xE1 \xE2 \xE3 \xE4 \xE5 \xE6 \xE7 " + "\xE8 \xE9 \xEA \xEB \xEC \xED \xEE \xEF ", /* bug: not corrected */ "\xE0 \xE1 \xE2 \xE3 \xE4 \xE5 \xE6 \xE7 " "\xE8 \xE9 \xEA \xEB \xEC \xED \xEE \xEF ", - "\"\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\u= FFFD " - "\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFF= FD \"", + "\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFF= FD " + "\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFF= FD ", }, /* 3.2.3 All 8 first bytes of 4-byte sequences, followed by space= */ { - "\"\xF0 \xF1 \xF2 \xF3 \xF4 \xF5 \xF6 \xF7 \"", + "\xF0 \xF1 \xF2 \xF3 \xF4 \xF5 \xF6 \xF7 ", NULL, /* bug: rejected */ - "\"\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\u= FFFD \"", + "\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFF= FD ", "\xF0 \xF1 \xF2 \xF3 \xF4 \xF5 \xF6 \xF7 ", }, /* 3.2.4 All 4 first bytes of 5-byte sequences, followed by space= */ { - "\"\xF8 \xF9 \xFA \xFB \"", + "\xF8 \xF9 \xFA \xFB ", NULL, /* bug: rejected */ - "\"\\uFFFD \\uFFFD \\uFFFD \\uFFFD \"", + "\\uFFFD \\uFFFD \\uFFFD \\uFFFD ", "\xF8 \xF9 \xFA \xFB ", }, /* 3.2.5 All 2 first bytes of 6-byte sequences, followed by space= */ { - "\"\xFC \xFD \"", + "\xFC \xFD ", NULL, /* bug: rejected */ - "\"\\uFFFD \\uFFFD \"", + "\\uFFFD \\uFFFD ", "\xFC \xFD ", }, /* 3.3 Sequences with last continuation byte missing */ /* 3.3.1 2-byte sequence with last byte missing (U+0000) */ { - "\"\xC0\"", + "\xC0", NULL, /* bug: rejected */ - "\"\\uFFFD\"", + "\\uFFFD", "\xC0", }, /* 3.3.2 3-byte sequence with last byte missing (U+0000) */ { - "\"\xE0\x80\"", + "\xE0\x80", "\xE0\x80", /* bug: not corrected */ - "\"\\uFFFD\"", + "\\uFFFD", }, /* 3.3.3 4-byte sequence with last byte missing (U+0000) */ { - "\"\xF0\x80\x80\"", + "\xF0\x80\x80", "\xF0\x80\x80", /* bug: not corrected */ - "\"\\uFFFD\"", + "\\uFFFD", }, /* 3.3.4 5-byte sequence with last byte missing (U+0000) */ { - "\"\xF8\x80\x80\x80\"", + "\xF8\x80\x80\x80", NULL, /* bug: rejected */ - "\"\\uFFFD\"", + "\\uFFFD", "\xF8\x80\x80\x80", }, /* 3.3.5 6-byte sequence with last byte missing (U+0000) */ { - "\"\xFC\x80\x80\x80\x80\"", + "\xFC\x80\x80\x80\x80", NULL, /* bug: rejected */ - "\"\\uFFFD\"", + "\\uFFFD", "\xFC\x80\x80\x80\x80", }, /* 3.3.6 2-byte sequence with last byte missing (U+07FF) */ { - "\"\xDF\"", + "\xDF", "\xDF", /* bug: not corrected */ - "\"\\uFFFD\"", + "\\uFFFD", }, /* 3.3.7 3-byte sequence with last byte missing (U+FFFF) */ { - "\"\xEF\xBF\"", + "\xEF\xBF", "\xEF\xBF", /* bug: not corrected */ - "\"\\uFFFD\"", + "\\uFFFD", }, /* 3.3.8 4-byte sequence with last byte missing (U+1FFFFF) */ { - "\"\xF7\xBF\xBF\"", + "\xF7\xBF\xBF", NULL, /* bug: rejected */ - "\"\\uFFFD\"", + "\\uFFFD", "\xF7\xBF\xBF", }, /* 3.3.9 5-byte sequence with last byte missing (U+3FFFFFF) */ { - "\"\xFB\xBF\xBF\xBF\"", + "\xFB\xBF\xBF\xBF", NULL, /* bug: rejected */ - "\"\\uFFFD\"", + "\\uFFFD", "\xFB\xBF\xBF\xBF", }, /* 3.3.10 6-byte sequence with last byte missing (U+7FFFFFFF) */ { - "\"\xFD\xBF\xBF\xBF\xBF\"", + "\xFD\xBF\xBF\xBF\xBF", NULL, /* bug: rejected */ - "\"\\uFFFD\"", + "\\uFFFD", "\xFD\xBF\xBF\xBF\xBF", }, /* 3.4 Concatenation of incomplete sequences */ { - "\"\xC0\xE0\x80\xF0\x80\x80\xF8\x80\x80\x80\xFC\x80\x80\x80\x8= 0" - "\xDF\xEF\xBF\xF7\xBF\xBF\xFB\xBF\xBF\xBF\xFD\xBF\xBF\xBF\xBF\= "", + "\xC0\xE0\x80\xF0\x80\x80\xF8\x80\x80\x80\xFC\x80\x80\x80\x80" + "\xDF\xEF\xBF\xF7\xBF\xBF\xFB\xBF\xBF\xBF\xFD\xBF\xBF\xBF\xBF", NULL, /* bug: rejected */ - "\"\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD" - "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\"", + "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD" + "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD", "\xC0\xE0\x80\xF0\x80\x80\xF8\x80\x80\x80\xFC\x80\x80\x80\x80" "\xDF\xEF\xBF\xF7\xBF\xBF\xFB\xBF\xBF\xBF\xFD\xBF\xBF\xBF\xBF", }, /* 3.5 Impossible bytes */ { - "\"\xFE\"", + "\xFE", NULL, /* bug: rejected */ - "\"\\uFFFD\"", + "\\uFFFD", "\xFE", }, { - "\"\xFF\"", + "\xFF", NULL, /* bug: rejected */ - "\"\\uFFFD\"", + "\\uFFFD", "\xFF", }, { - "\"\xFE\xFE\xFF\xFF\"", + "\xFE\xFE\xFF\xFF", NULL, /* bug: rejected */ - "\"\\uFFFD\\uFFFD\\uFFFD\\uFFFD\"", + "\\uFFFD\\uFFFD\\uFFFD\\uFFFD", "\xFE\xFE\xFF\xFF", }, /* 4 Overlong sequences */ /* 4.1 Overlong '/' */ { - "\"\xC0\xAF\"", + "\xC0\xAF", NULL, /* bug: rejected */ - "\"\\uFFFD\"", + "\\uFFFD", "\xC0\xAF", }, { - "\"\xE0\x80\xAF\"", + "\xE0\x80\xAF", "\xE0\x80\xAF", /* bug: not corrected */ - "\"\\uFFFD\"", + "\\uFFFD", }, { - "\"\xF0\x80\x80\xAF\"", + "\xF0\x80\x80\xAF", "\xF0\x80\x80\xAF", /* bug: not corrected */ - "\"\\uFFFD\"", + "\\uFFFD", }, { - "\"\xF8\x80\x80\x80\xAF\"", + "\xF8\x80\x80\x80\xAF", NULL, /* bug: rejected */ - "\"\\uFFFD\"", + "\\uFFFD", "\xF8\x80\x80\x80\xAF", }, { - "\"\xFC\x80\x80\x80\x80\xAF\"", + "\xFC\x80\x80\x80\x80\xAF", NULL, /* bug: rejected */ - "\"\\uFFFD\"", + "\\uFFFD", "\xFC\x80\x80\x80\x80\xAF", }, /* @@ -558,16 +562,16 @@ static void utf8_string(void) */ { /* \U+007F */ - "\"\xC1\xBF\"", + "\xC1\xBF", NULL, /* bug: rejected */ - "\"\\uFFFD\"", + "\\uFFFD", "\xC1\xBF", }, { /* \U+07FF */ - "\"\xE0\x9F\xBF\"", + "\xE0\x9F\xBF", "\xE0\x9F\xBF", /* bug: not corrected */ - "\"\\uFFFD\"", + "\\uFFFD", }, { /* @@ -576,181 +580,181 @@ static void utf8_string(void) * noncharacter. Testing U+FFFC seems more useful. See * also 2.2.3 */ - "\"\xF0\x8F\xBF\xBC\"", + "\xF0\x8F\xBF\xBC", "\xF0\x8F\xBF\xBC", /* bug: not corrected */ - "\"\\uFFFD\"", + "\\uFFFD", }, { /* \U+1FFFFF */ - "\"\xF8\x87\xBF\xBF\xBF\"", + "\xF8\x87\xBF\xBF\xBF", NULL, /* bug: rejected */ - "\"\\uFFFD\"", + "\\uFFFD", "\xF8\x87\xBF\xBF\xBF", }, { /* \U+3FFFFFF */ - "\"\xFC\x83\xBF\xBF\xBF\xBF\"", + "\xFC\x83\xBF\xBF\xBF\xBF", NULL, /* bug: rejected */ - "\"\\uFFFD\"", + "\\uFFFD", "\xFC\x83\xBF\xBF\xBF\xBF", }, /* 4.3 Overlong representation of the NUL character */ { /* \U+0000 */ - "\"\xC0\x80\"", + "\xC0\x80", NULL, /* bug: rejected */ - "\"\\u0000\"", + "\\u0000", "\xC0\x80", }, { /* \U+0000 */ - "\"\xE0\x80\x80\"", + "\xE0\x80\x80", "\xE0\x80\x80", /* bug: not corrected */ - "\"\\uFFFD\"", + "\\uFFFD", }, { /* \U+0000 */ - "\"\xF0\x80\x80\x80\"", + "\xF0\x80\x80\x80", "\xF0\x80\x80\x80", /* bug: not corrected */ - "\"\\uFFFD\"", + "\\uFFFD", }, { /* \U+0000 */ - "\"\xF8\x80\x80\x80\x80\"", + "\xF8\x80\x80\x80\x80", NULL, /* bug: rejected */ - "\"\\uFFFD\"", + "\\uFFFD", "\xF8\x80\x80\x80\x80", }, { /* \U+0000 */ - "\"\xFC\x80\x80\x80\x80\x80\"", + "\xFC\x80\x80\x80\x80\x80", NULL, /* bug: rejected */ - "\"\\uFFFD\"", + "\\uFFFD", "\xFC\x80\x80\x80\x80\x80", }, /* 5 Illegal code positions */ /* 5.1 Single UTF-16 surrogates */ { /* \U+D800 */ - "\"\xED\xA0\x80\"", + "\xED\xA0\x80", "\xED\xA0\x80", /* bug: not corrected */ - "\"\\uFFFD\"", + "\\uFFFD", }, { /* \U+DB7F */ - "\"\xED\xAD\xBF\"", + "\xED\xAD\xBF", "\xED\xAD\xBF", /* bug: not corrected */ - "\"\\uFFFD\"", + "\\uFFFD", }, { /* \U+DB80 */ - "\"\xED\xAE\x80\"", + "\xED\xAE\x80", "\xED\xAE\x80", /* bug: not corrected */ - "\"\\uFFFD\"", + "\\uFFFD", }, { /* \U+DBFF */ - "\"\xED\xAF\xBF\"", + "\xED\xAF\xBF", "\xED\xAF\xBF", /* bug: not corrected */ - "\"\\uFFFD\"", + "\\uFFFD", }, { /* \U+DC00 */ - "\"\xED\xB0\x80\"", + "\xED\xB0\x80", "\xED\xB0\x80", /* bug: not corrected */ - "\"\\uFFFD\"", + "\\uFFFD", }, { /* \U+DF80 */ - "\"\xED\xBE\x80\"", + "\xED\xBE\x80", "\xED\xBE\x80", /* bug: not corrected */ - "\"\\uFFFD\"", + "\\uFFFD", }, { /* \U+DFFF */ - "\"\xED\xBF\xBF\"", + "\xED\xBF\xBF", "\xED\xBF\xBF", /* bug: not corrected */ - "\"\\uFFFD\"", + "\\uFFFD", }, /* 5.2 Paired UTF-16 surrogates */ { /* \U+D800\U+DC00 */ - "\"\xED\xA0\x80\xED\xB0\x80\"", + "\xED\xA0\x80\xED\xB0\x80", "\xED\xA0\x80\xED\xB0\x80", /* bug: not corrected */ - "\"\\uFFFD\\uFFFD\"", + "\\uFFFD\\uFFFD", }, { /* \U+D800\U+DFFF */ - "\"\xED\xA0\x80\xED\xBF\xBF\"", + "\xED\xA0\x80\xED\xBF\xBF", "\xED\xA0\x80\xED\xBF\xBF", /* bug: not corrected */ - "\"\\uFFFD\\uFFFD\"", + "\\uFFFD\\uFFFD", }, { /* \U+DB7F\U+DC00 */ - "\"\xED\xAD\xBF\xED\xB0\x80\"", + "\xED\xAD\xBF\xED\xB0\x80", "\xED\xAD\xBF\xED\xB0\x80", /* bug: not corrected */ - "\"\\uFFFD\\uFFFD\"", + "\\uFFFD\\uFFFD", }, { /* \U+DB7F\U+DFFF */ - "\"\xED\xAD\xBF\xED\xBF\xBF\"", + "\xED\xAD\xBF\xED\xBF\xBF", "\xED\xAD\xBF\xED\xBF\xBF", /* bug: not corrected */ - "\"\\uFFFD\\uFFFD\"", + "\\uFFFD\\uFFFD", }, { /* \U+DB80\U+DC00 */ - "\"\xED\xAE\x80\xED\xB0\x80\"", + "\xED\xAE\x80\xED\xB0\x80", "\xED\xAE\x80\xED\xB0\x80", /* bug: not corrected */ - "\"\\uFFFD\\uFFFD\"", + "\\uFFFD\\uFFFD", }, { /* \U+DB80\U+DFFF */ - "\"\xED\xAE\x80\xED\xBF\xBF\"", + "\xED\xAE\x80\xED\xBF\xBF", "\xED\xAE\x80\xED\xBF\xBF", /* bug: not corrected */ - "\"\\uFFFD\\uFFFD\"", + "\\uFFFD\\uFFFD", }, { /* \U+DBFF\U+DC00 */ - "\"\xED\xAF\xBF\xED\xB0\x80\"", + "\xED\xAF\xBF\xED\xB0\x80", "\xED\xAF\xBF\xED\xB0\x80", /* bug: not corrected */ - "\"\\uFFFD\\uFFFD\"", + "\\uFFFD\\uFFFD", }, { /* \U+DBFF\U+DFFF */ - "\"\xED\xAF\xBF\xED\xBF\xBF\"", + "\xED\xAF\xBF\xED\xBF\xBF", "\xED\xAF\xBF\xED\xBF\xBF", /* bug: not corrected */ - "\"\\uFFFD\\uFFFD\"", + "\\uFFFD\\uFFFD", }, /* 5.3 Other illegal code positions */ /* BMP noncharacters */ { /* \U+FFFE */ - "\"\xEF\xBF\xBE\"", + "\xEF\xBF\xBE", "\xEF\xBF\xBE", /* bug: not corrected */ - "\"\\uFFFD\"", + "\\uFFFD", }, { /* \U+FFFF */ - "\"\xEF\xBF\xBF\"", + "\xEF\xBF\xBF", "\xEF\xBF\xBF", /* bug: not corrected */ - "\"\\uFFFD\"", + "\\uFFFD", }, { /* U+FDD0 */ - "\"\xEF\xB7\x90\"", + "\xEF\xB7\x90", "\xEF\xB7\x90", /* bug: not corrected */ - "\"\\uFFFD\"", + "\\uFFFD", }, { /* U+FDEF */ - "\"\xEF\xB7\xAF\"", + "\xEF\xB7\xAF", "\xEF\xB7\xAF", /* bug: not corrected */ - "\"\\uFFFD\"", + "\\uFFFD", }, /* Plane 1 .. 16 noncharacters */ { /* U+1FFFE U+1FFFF U+2FFFE U+2FFFF ... U+10FFFE U+10FFFF */ - "\"\xF0\x9F\xBF\xBE\xF0\x9F\xBF\xBF" + "\xF0\x9F\xBF\xBE\xF0\x9F\xBF\xBF" "\xF0\xAF\xBF\xBE\xF0\xAF\xBF\xBF" "\xF0\xBF\xBF\xBE\xF0\xBF\xBF\xBF" "\xF1\x8F\xBF\xBE\xF1\x8F\xBF\xBF" @@ -765,7 +769,7 @@ static void utf8_string(void) "\xF3\x9F\xBF\xBE\xF3\x9F\xBF\xBF" "\xF3\xAF\xBF\xBE\xF3\xAF\xBF\xBF" "\xF3\xBF\xBF\xBE\xF3\xBF\xBF\xBF" - "\xF4\x8F\xBF\xBE\xF4\x8F\xBF\xBF\"", + "\xF4\x8F\xBF\xBE\xF4\x8F\xBF\xBF", /* bug: not corrected */ "\xF0\x9F\xBF\xBE\xF0\x9F\xBF\xBF" "\xF0\xAF\xBF\xBE\xF0\xAF\xBF\xBF" @@ -783,55 +787,52 @@ static void utf8_string(void) "\xF3\xAF\xBF\xBE\xF3\xAF\xBF\xBF" "\xF3\xBF\xBF\xBE\xF3\xBF\xBF\xBF" "\xF4\x8F\xBF\xBE\xF4\x8F\xBF\xBF", - "\"\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD" "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD" "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD" - "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\"", + "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD" + "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD", }, {} }; - int i; - QObject *obj; + int i, j; QString *str; const char *json_in, *utf8_out, *utf8_in, *json_out; + char *jstr; =20 for (i =3D 0; test_cases[i].json_in; i++) { - json_in =3D test_cases[i].json_in; - utf8_out =3D test_cases[i].utf8_out; - utf8_in =3D test_cases[i].utf8_in ?: test_cases[i].utf8_out; - json_out =3D test_cases[i].json_out ?: test_cases[i].json_in; + for (j =3D 0; j < 2; j++) { + json_in =3D test_cases[i].json_in; + utf8_out =3D test_cases[i].utf8_out; + utf8_in =3D test_cases[i].utf8_in ?: test_cases[i].utf8_out; + json_out =3D test_cases[i].json_out ?: test_cases[i].json_in; =20 - obj =3D qobject_from_json(json_in, utf8_out ? &error_abort : NULL); - if (utf8_out) { - str =3D qobject_to(QString, obj); - g_assert(str); - g_assert_cmpstr(qstring_get_str(str), =3D=3D, utf8_out); - } else { - g_assert(!obj); - } - qobject_unref(obj); + /* Parse @json_in, expect @utf8_out */ + if (utf8_out) { + str =3D from_json_str(json_in, &error_abort, j); + g_assert_cmpstr(qstring_get_try_str(str), =3D=3D, utf8_out= ); + qobject_unref(str); + } else { + str =3D from_json_str(json_in, NULL, j); + g_assert(!str); + } =20 - obj =3D QOBJECT(qstring_from_str(utf8_in)); - str =3D qobject_to_json(obj); - if (json_out) { - g_assert(str); - g_assert_cmpstr(qstring_get_str(str), =3D=3D, json_out); - } else { - g_assert(!str); - } - qobject_unref(str); - qobject_unref(obj); + /* Unparse @utf8_in, expect @json_out */ + str =3D qstring_from_str(utf8_in); + jstr =3D to_json_str(str); + g_assert_cmpstr(jstr, =3D=3D, json_out); + qobject_unref(str); + g_free(jstr); =20 - /* - * Disabled, because qobject_from_json() is buggy, and I can't - * be bothered to add the expected incorrect results. - * FIXME Enable once these bugs have been fixed. - */ - if (0 && json_out !=3D json_in) { - obj =3D qobject_from_json(json_out, &error_abort); - str =3D qobject_to(QString, obj); - g_assert(str); - g_assert_cmpstr(qstring_get_str(str), =3D=3D, utf8_out); + /* + * Parse @json_out right back + * Disabled, because qobject_from_json() is buggy, and I can't + * be bothered to add the expected incorrect results. + * FIXME Enable once these bugs have been fixed. + */ + if (0 && json_out !=3D json_in) { + str =3D from_json_str(json_out, &error_abort, j); + g_assert_cmpstr(qstring_get_try_str(str), =3D=3D, utf8_out= ); + } } } } --=20 2.17.1 From nobody Wed Nov 5 07:53:13 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533731680040456.26476999494287; Wed, 8 Aug 2018 05:34:40 -0700 (PDT) Received: from localhost ([::1]:43350 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNfd-00084v-NQ for importer@patchew.org; Wed, 08 Aug 2018 08:34:33 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52580) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBq-0006jv-56 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:55 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBi-0008Ri-Nl for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:46 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:47684 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBi-0008Qy-Gc for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:38 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 0D5937A7EE; Wed, 8 Aug 2018 12:03:38 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id B969A1040840; Wed, 8 Aug 2018 12:03:37 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 346F11138669; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:02:50 +0200 Message-Id: <20180808120334.10970-13-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) for IP:'10.11.54.3' DOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 12/56] check-qjson: Simplify utf8_string() X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The previous commit made utf8_string()'s test_cases[].utf8_in superfluous: we can use .json_in instead. Except for the case testing U+0000. \x00 doesn't work in C strings, so it tests \\u0000 instead. But testing \\uXXXX is escaped_string()'s job. It's covered there. Test U+0001 here, and drop .utf8_in. Signed-off-by: Markus Armbruster Reviewed-by: Eric Blake --- tests/check-qjson.c | 53 ++++++++------------------------------------- 1 file changed, 9 insertions(+), 44 deletions(-) diff --git a/tests/check-qjson.c b/tests/check-qjson.c index 75f0a9f18a..5ba09e5ab6 100644 --- a/tests/check-qjson.c +++ b/tests/check-qjson.c @@ -150,8 +150,6 @@ static void utf8_string(void) const char *utf8_out; /* Expected unparse output, defaults to @json_in */ const char *json_out; - /* Expected parse output for @json_out, defaults to @utf8_out */ - const char *utf8_in; } test_cases[] =3D { /* * Bug markers used here: @@ -160,10 +158,6 @@ static void utf8_string(void) * - bug: rejected * JSON parser rejects invalid sequence(s) * We may choose to define this as feature - * - bug: want "..." - * JSON parser produces incorrect result, this is the - * correct one, assuming replacement character U+FFFF - * We may choose to reject instead of replace */ =20 /* 1 Some correct UTF-8 text */ @@ -184,12 +178,15 @@ static void utf8_string(void) }, /* 2 Boundary condition test cases */ /* 2.1 First possible sequence of a certain length */ - /* 2.1.1 1 byte U+0000 */ + /* + * 2.1.1 1 byte U+0001 + * \x00 is impossible, test \x01 instead. Other + * representations of U+0000 are covered under 4.3. + */ { - "\\u0000", - "", /* bug: want overlong "\xC0\x80" */ - "\\u0000", - "\xC0\x80", + "\x01", + "\x01", + "\\u0001", }, /* 2.1.2 2 bytes U+0080 */ { @@ -214,14 +211,12 @@ static void utf8_string(void) "\xF8\x88\x80\x80\x80", NULL, /* bug: rejected */ "\\uFFFD", - "\xF8\x88\x80\x80\x80", }, /* 2.1.6 6 bytes U+4000000 */ { "\xFC\x84\x80\x80\x80\x80", NULL, /* bug: rejected */ "\\uFFFD", - "\xFC\x84\x80\x80\x80\x80", }, /* 2.2 Last possible sequence of a certain length */ /* 2.2.1 1 byte U+007F */ @@ -255,21 +250,18 @@ static void utf8_string(void) "\xF7\xBF\xBF\xBF", NULL, /* bug: rejected */ "\\uFFFD", - "\xF7\xBF\xBF\xBF", }, /* 2.2.5 5 bytes U+3FFFFFF */ { "\xFB\xBF\xBF\xBF\xBF", NULL, /* bug: rejected */ "\\uFFFD", - "\xFB\xBF\xBF\xBF\xBF", }, /* 2.2.6 6 bytes U+7FFFFFFF */ { "\xFD\xBF\xBF\xBF\xBF\xBF", NULL, /* bug: rejected */ "\\uFFFD", - "\xFD\xBF\xBF\xBF\xBF\xBF", }, /* 2.3 Other boundary conditions */ { @@ -392,10 +384,6 @@ static void utf8_string(void) "\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFF= FD " "\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFF= FD " "\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFF= FD ", - "\xC0 \xC1 \xC2 \xC3 \xC4 \xC5 \xC6 \xC7 " - "\xC8 \xC9 \xCA \xCB \xCC \xCD \xCE \xCF " - "\xD0 \xD1 \xD2 \xD3 \xD4 \xD5 \xD6 \xD7 " - "\xD8 \xD9 \xDA \xDB \xDC \xDD \xDE \xDF ", }, /* 3.2.2 All 16 first bytes of 3-byte sequences, followed by spac= e */ { @@ -412,21 +400,18 @@ static void utf8_string(void) "\xF0 \xF1 \xF2 \xF3 \xF4 \xF5 \xF6 \xF7 ", NULL, /* bug: rejected */ "\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFF= FD ", - "\xF0 \xF1 \xF2 \xF3 \xF4 \xF5 \xF6 \xF7 ", }, /* 3.2.4 All 4 first bytes of 5-byte sequences, followed by space= */ { "\xF8 \xF9 \xFA \xFB ", NULL, /* bug: rejected */ "\\uFFFD \\uFFFD \\uFFFD \\uFFFD ", - "\xF8 \xF9 \xFA \xFB ", }, /* 3.2.5 All 2 first bytes of 6-byte sequences, followed by space= */ { "\xFC \xFD ", NULL, /* bug: rejected */ "\\uFFFD \\uFFFD ", - "\xFC \xFD ", }, /* 3.3 Sequences with last continuation byte missing */ /* 3.3.1 2-byte sequence with last byte missing (U+0000) */ @@ -434,7 +419,6 @@ static void utf8_string(void) "\xC0", NULL, /* bug: rejected */ "\\uFFFD", - "\xC0", }, /* 3.3.2 3-byte sequence with last byte missing (U+0000) */ { @@ -453,14 +437,12 @@ static void utf8_string(void) "\xF8\x80\x80\x80", NULL, /* bug: rejected */ "\\uFFFD", - "\xF8\x80\x80\x80", }, /* 3.3.5 6-byte sequence with last byte missing (U+0000) */ { "\xFC\x80\x80\x80\x80", NULL, /* bug: rejected */ "\\uFFFD", - "\xFC\x80\x80\x80\x80", }, /* 3.3.6 2-byte sequence with last byte missing (U+07FF) */ { @@ -479,21 +461,18 @@ static void utf8_string(void) "\xF7\xBF\xBF", NULL, /* bug: rejected */ "\\uFFFD", - "\xF7\xBF\xBF", }, /* 3.3.9 5-byte sequence with last byte missing (U+3FFFFFF) */ { "\xFB\xBF\xBF\xBF", NULL, /* bug: rejected */ "\\uFFFD", - "\xFB\xBF\xBF\xBF", }, /* 3.3.10 6-byte sequence with last byte missing (U+7FFFFFFF) */ { "\xFD\xBF\xBF\xBF\xBF", NULL, /* bug: rejected */ "\\uFFFD", - "\xFD\xBF\xBF\xBF\xBF", }, /* 3.4 Concatenation of incomplete sequences */ { @@ -502,27 +481,22 @@ static void utf8_string(void) NULL, /* bug: rejected */ "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD" "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD", - "\xC0\xE0\x80\xF0\x80\x80\xF8\x80\x80\x80\xFC\x80\x80\x80\x80" - "\xDF\xEF\xBF\xF7\xBF\xBF\xFB\xBF\xBF\xBF\xFD\xBF\xBF\xBF\xBF", }, /* 3.5 Impossible bytes */ { "\xFE", NULL, /* bug: rejected */ "\\uFFFD", - "\xFE", }, { "\xFF", NULL, /* bug: rejected */ "\\uFFFD", - "\xFF", }, { "\xFE\xFE\xFF\xFF", NULL, /* bug: rejected */ "\\uFFFD\\uFFFD\\uFFFD\\uFFFD", - "\xFE\xFE\xFF\xFF", }, /* 4 Overlong sequences */ /* 4.1 Overlong '/' */ @@ -530,7 +504,6 @@ static void utf8_string(void) "\xC0\xAF", NULL, /* bug: rejected */ "\\uFFFD", - "\xC0\xAF", }, { "\xE0\x80\xAF", @@ -546,13 +519,11 @@ static void utf8_string(void) "\xF8\x80\x80\x80\xAF", NULL, /* bug: rejected */ "\\uFFFD", - "\xF8\x80\x80\x80\xAF", }, { "\xFC\x80\x80\x80\x80\xAF", NULL, /* bug: rejected */ "\\uFFFD", - "\xFC\x80\x80\x80\x80\xAF", }, /* * 4.2 Maximum overlong sequences @@ -565,7 +536,6 @@ static void utf8_string(void) "\xC1\xBF", NULL, /* bug: rejected */ "\\uFFFD", - "\xC1\xBF", }, { /* \U+07FF */ @@ -589,14 +559,12 @@ static void utf8_string(void) "\xF8\x87\xBF\xBF\xBF", NULL, /* bug: rejected */ "\\uFFFD", - "\xF8\x87\xBF\xBF\xBF", }, { /* \U+3FFFFFF */ "\xFC\x83\xBF\xBF\xBF\xBF", NULL, /* bug: rejected */ "\\uFFFD", - "\xFC\x83\xBF\xBF\xBF\xBF", }, /* 4.3 Overlong representation of the NUL character */ { @@ -604,7 +572,6 @@ static void utf8_string(void) "\xC0\x80", NULL, /* bug: rejected */ "\\u0000", - "\xC0\x80", }, { /* \U+0000 */ @@ -623,14 +590,12 @@ static void utf8_string(void) "\xF8\x80\x80\x80\x80", NULL, /* bug: rejected */ "\\uFFFD", - "\xF8\x80\x80\x80\x80", }, { /* \U+0000 */ "\xFC\x80\x80\x80\x80\x80", NULL, /* bug: rejected */ "\\uFFFD", - "\xFC\x80\x80\x80\x80\x80", }, /* 5 Illegal code positions */ /* 5.1 Single UTF-16 surrogates */ @@ -803,7 +768,7 @@ static void utf8_string(void) for (j =3D 0; j < 2; j++) { json_in =3D test_cases[i].json_in; utf8_out =3D test_cases[i].utf8_out; - utf8_in =3D test_cases[i].utf8_in ?: test_cases[i].utf8_out; + utf8_in =3D test_cases[i].utf8_out ?: test_cases[i].json_in; json_out =3D test_cases[i].json_out ?: test_cases[i].json_in; =20 /* Parse @json_in, expect @utf8_out */ --=20 2.17.1 From nobody Wed Nov 5 07:53:13 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533731201160296.7708349010783; Wed, 8 Aug 2018 05:26:41 -0700 (PDT) Received: from localhost ([::1]:43310 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNY0-0001Lt-1n for importer@patchew.org; Wed, 08 Aug 2018 08:26:40 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52393) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBl-0006f9-M0 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:52 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBi-0008SY-UL for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:41 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:51054 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBi-0008RE-L5 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:38 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 2CD25402346C; Wed, 8 Aug 2018 12:03:38 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D64422166BA0; Wed, 8 Aug 2018 12:03:37 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 37D8B113866C; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:02:51 +0200 Message-Id: <20180808120334.10970-14-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 13/56] check-qjson: Fix utf8_string() to test all invalid sequences X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Some of utf8_string()'s test_cases[] contain multiple invalid sequences. Testing that qobject_from_json() fails only tests we reject at least one invalid sequence. That's incomplete. Additionally test each non-space sequence in isolation. This demonstrates that the JSON parser accepts invalid sequences starting with \xC2..\xF4. Add a FIXME comment. Signed-off-by: Markus Armbruster Reviewed-by: Eric Blake --- tests/check-qjson.c | 33 ++++++++++++++++++++++++++++----- 1 file changed, 28 insertions(+), 5 deletions(-) diff --git a/tests/check-qjson.c b/tests/check-qjson.c index 5ba09e5ab6..5f3334322b 100644 --- a/tests/check-qjson.c +++ b/tests/check-qjson.c @@ -20,6 +20,7 @@ #include "qapi/qmp/qnull.h" #include "qapi/qmp/qnum.h" #include "qapi/qmp/qstring.h" +#include "qemu/unicode.h" #include "qemu-common.h" =20 static QString *from_json_str(const char *jstr, Error **errp, bool single) @@ -379,7 +380,7 @@ static void utf8_string(void) "\xC8 \xC9 \xCA \xCB \xCC \xCD \xCE \xCF " "\xD0 \xD1 \xD2 \xD3 \xD4 \xD5 \xD6 \xD7 " "\xD8 \xD9 \xDA \xDB \xDC \xDD \xDE \xDF ", - NULL, /* bug: rejected */ + NULL, /* bug: rejected (partly, see FIXME below)= */ "\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFF= FD " "\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFF= FD " "\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFF= FD " @@ -398,7 +399,7 @@ static void utf8_string(void) /* 3.2.3 All 8 first bytes of 4-byte sequences, followed by space= */ { "\xF0 \xF1 \xF2 \xF3 \xF4 \xF5 \xF6 \xF7 ", - NULL, /* bug: rejected */ + NULL, /* bug: rejected (partly, see FIXME below)= */ "\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFF= FD ", }, /* 3.2.4 All 4 first bytes of 5-byte sequences, followed by space= */ @@ -478,7 +479,7 @@ static void utf8_string(void) { "\xC0\xE0\x80\xF0\x80\x80\xF8\x80\x80\x80\xFC\x80\x80\x80\x80" "\xDF\xEF\xBF\xF7\xBF\xBF\xFB\xBF\xBF\xBF\xFD\xBF\xBF\xBF\xBF", - NULL, /* bug: rejected */ + NULL, /* bug: rejected (partly, see FIXME below)= */ "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD" "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD", }, @@ -761,8 +762,8 @@ static void utf8_string(void) }; int i, j; QString *str; - const char *json_in, *utf8_out, *utf8_in, *json_out; - char *jstr; + const char *json_in, *utf8_out, *utf8_in, *json_out, *tail; + char *end, *in, *jstr; =20 for (i =3D 0; test_cases[i].json_in; i++) { for (j =3D 0; j < 2; j++) { @@ -779,6 +780,28 @@ static void utf8_string(void) } else { str =3D from_json_str(json_in, NULL, j); g_assert(!str); + /* + * Failure may be due to any sequence, but *all* sequences + * are expected to fail. Test each one in isolation. + */ + for (tail =3D json_in; *tail; tail =3D end) { + mod_utf8_codepoint(tail, 6, &end); + if (*end =3D=3D ' ') { + end++; + } + in =3D strndup(tail, end - tail); + str =3D from_json_str(in, NULL, j); + /* + * FIXME JSON parser accepts invalid sequence + * starting with \xC2..\xF4 + */ + if (*in >=3D '\xC2' && *in <=3D '\xF4') { + g_free(str); + str =3D NULL; + } + g_assert(!str); + g_free(in); + } } =20 /* Unparse @utf8_in, expect @json_out */ --=20 2.17.1 From nobody Wed Nov 5 07:53:13 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 15337308237971.5299829764369406; Wed, 8 Aug 2018 05:20:23 -0700 (PDT) Received: from localhost ([::1]:43269 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNRn-0003yf-NA for importer@patchew.org; Wed, 08 Aug 2018 08:20:15 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52372) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBl-0006eb-33 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:46 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBi-0008SS-Tz for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:40 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:47690 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBi-0008RG-Kh for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:38 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 2F67E7A7E7; Wed, 8 Aug 2018 12:03:38 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id DB1C720180F5; Wed, 8 Aug 2018 12:03:37 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 3B6FB113866D; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:02:52 +0200 Message-Id: <20180808120334.10970-15-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 14/56] check-qjson qmp-test: Cover control characters more thoroughly X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" RFC 7159 requires control characters in strings to be escaped. Demonstrate the JSON parser accepts U+0001 .. U+001F unescaped. Signed-off-by: Markus Armbruster Reviewed-by: Eric Blake --- tests/check-qjson.c | 36 ++++++++++++++++++++++++++++++------ tests/qmp-test.c | 14 ++++++++++++++ 2 files changed, 44 insertions(+), 6 deletions(-) diff --git a/tests/check-qjson.c b/tests/check-qjson.c index 5f3334322b..33bd5854fc 100644 --- a/tests/check-qjson.c +++ b/tests/check-qjson.c @@ -161,6 +161,26 @@ static void utf8_string(void) * We may choose to define this as feature */ =20 + /* 0 Control characters */ + { + /* + * Note: \x00 is impossible, other representations of + * U+0000 are covered under 4.3 + */ + "\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0A\x0B\x0C\x0D\x0E\x0F" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1A\x1B\x1C\x1D\x1E\x1F", + /* bug: not corrected (valid UTF-8, but invalid JSON) */ + "\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0A\x0B\x0C\x0D\x0E\x0F" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1A\x1B\x1C\x1D\x1E\x1F", + "\\u0001\\u0002\\u0003\\u0004\\u0005\\u0006\\u0007" + "\\b\\t\\n\\u000B\\f\\r\\u000E\\u000F" + "\\u0010\\u0011\\u0012\\u0013\\u0014\\u0015\\u0016\\u0017" + "\\u0018\\u0019\\u001A\\u001B\\u001C\\u001D\\u001E\\u001F", + }, /* 1 Some correct UTF-8 text */ { /* a bit of German */ @@ -180,14 +200,14 @@ static void utf8_string(void) /* 2 Boundary condition test cases */ /* 2.1 First possible sequence of a certain length */ /* - * 2.1.1 1 byte U+0001 - * \x00 is impossible, test \x01 instead. Other - * representations of U+0000 are covered under 4.3. + * 2.1.1 1 byte U+0020 + * Control characters are already covered by their own test + * case under 0. Test the first 1 byte non-control character + * here. */ { - "\x01", - "\x01", - "\\u0001", + " ", + " ", }, /* 2.1.2 2 bytes U+0080 */ { @@ -1302,6 +1322,10 @@ static void junk_input(void) g_assert(!err); /* BUG */ g_assert(obj =3D=3D NULL); =20 + obj =3D qobject_from_json("{\x01", &err); + g_assert(!err); /* BUG */ + g_assert(obj =3D=3D NULL); + obj =3D qobject_from_json("[0\xFF]", &err); error_free_or_abort(&err); g_assert(obj =3D=3D NULL); diff --git a/tests/qmp-test.c b/tests/qmp-test.c index 5e56be105e..5117a1ab25 100644 --- a/tests/qmp-test.c +++ b/tests/qmp-test.c @@ -71,6 +71,13 @@ static void test_malformed(QTestState *qts) qobject_unref(resp); g_assert(recovered(qts)); =20 + /* lexical error: funny control character outside string */ + qtest_qmp_send_raw(qts, "{\x01"); + resp =3D qtest_qmp_receive(qts); + g_assert_cmpstr(get_error_class(resp), =3D=3D, "GenericError"); + qobject_unref(resp); + g_assert(recovered(qts)); + /* lexical error: impossible byte in string */ qtest_qmp_send_raw(qts, "{'bad \xFF"); resp =3D qtest_qmp_receive(qts); @@ -78,6 +85,13 @@ static void test_malformed(QTestState *qts) qobject_unref(resp); g_assert(recovered(qts)); =20 + /* lexical error: control character in string */ + qtest_qmp_send_raw(qts, "{'execute': 'nonexistent', 'id':'\n'}"); + resp =3D qtest_qmp_receive(qts); + g_assert_cmpstr(get_error_class(resp), =3D=3D, "CommandNotFound"); /* = BUG */ + qobject_unref(resp); + g_assert(recovered(qts)); + /* lexical error: interpolation */ qtest_qmp_send_raw(qts, "%%p\n"); resp =3D qtest_qmp_receive(qts); --=20 2.17.1 From nobody Wed Nov 5 07:53:13 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533730170555401.5188226372745; Wed, 8 Aug 2018 05:09:30 -0700 (PDT) Received: from localhost ([::1]:43147 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNHN-0002vY-5O for importer@patchew.org; Wed, 08 Aug 2018 08:09:29 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52411) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBm-0006fh-6X for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:51 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBi-0008Sj-VP for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:42 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:51746 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBi-0008RM-Nh for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:38 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 3975D40241CB; Wed, 8 Aug 2018 12:03:38 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id DE86A2166BA8; Wed, 8 Aug 2018 12:03:37 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 44EC911386AB; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:02:53 +0200 Message-Id: <20180808120334.10970-16-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.7]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.7]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 15/56] check-qjson: Cover interpolation more thoroughly X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Signed-off-by: Markus Armbruster Reviewed-by: Eric Blake --- tests/check-qjson.c | 141 ++++++++++++++++++++++++-------------------- 1 file changed, 77 insertions(+), 64 deletions(-) diff --git a/tests/check-qjson.c b/tests/check-qjson.c index 33bd5854fc..fda2b014a3 100644 --- a/tests/check-qjson.c +++ b/tests/check-qjson.c @@ -845,30 +845,6 @@ static void utf8_string(void) } } =20 -static void vararg_string(void) -{ - int i; - struct { - const char *decoded; - } test_cases[] =3D { - { "hello world" }, - { "the quick brown fox jumped over the fence" }, - {} - }; - - for (i =3D 0; test_cases[i].decoded; i++) { - QString *str; - - str =3D qobject_to(QString, - qobject_from_jsonf_nofail("%s", - test_cases[i].decoded)); - g_assert(str); - g_assert(strcmp(qstring_get_str(str), test_cases[i].decoded) =3D= =3D 0); - - qobject_unref(str); - } -} - static void simple_number(void) { int i; @@ -986,29 +962,6 @@ static void float_number(void) } } =20 -static void vararg_number(void) -{ - QNum *qnum; - int value =3D 0x2342; - long long value_ll =3D 0x2342342343LL; - double valuef =3D 2.323423423; - int64_t val; - - qnum =3D qobject_to(QNum, qobject_from_jsonf_nofail("%d", value)); - g_assert(qnum_get_try_int(qnum, &val)); - g_assert_cmpint(val, =3D=3D, value); - qobject_unref(qnum); - - qnum =3D qobject_to(QNum, qobject_from_jsonf_nofail("%lld", value_ll)); - g_assert(qnum_get_try_int(qnum, &val)); - g_assert_cmpint(val, =3D=3D, value_ll); - qobject_unref(qnum); - - qnum =3D qobject_to(QNum, qobject_from_jsonf_nofail("%f", valuef)); - g_assert(qnum_get_double(qnum) =3D=3D valuef); - qobject_unref(qnum); -} - static void keyword_literal(void) { QObject *obj; @@ -1038,17 +991,6 @@ static void keyword_literal(void) =20 qobject_unref(qbool); =20 - qbool =3D qobject_to(QBool, qobject_from_jsonf_nofail("%i", false)); - g_assert(qbool); - g_assert(qbool_get_bool(qbool) =3D=3D false); - qobject_unref(qbool); - - /* Test that non-zero values other than 1 get collapsed to true */ - qbool =3D qobject_to(QBool, qobject_from_jsonf_nofail("%i", 2)); - g_assert(qbool); - g_assert(qbool_get_bool(qbool) =3D=3D true); - qobject_unref(qbool); - obj =3D qobject_from_json("null", &error_abort); g_assert(obj !=3D NULL); g_assert(qobject_type(obj) =3D=3D QTYPE_QNULL); @@ -1060,6 +1002,78 @@ static void keyword_literal(void) qobject_unref(null); } =20 +static void interpolation(void) +{ + long long value_lld =3D 0x123456789abcdefLL; + long value_ld =3D (long)value_lld; + int value_d =3D (int)value_lld; + unsigned long long value_llu =3D 0xfedcba9876543210ULL; + unsigned long value_lu =3D (unsigned long)value_llu; + unsigned value_u =3D (unsigned)value_llu; + double value_f =3D 2.323423423; + const char *value_s =3D "hello world"; + QObject *value_p =3D QOBJECT(qnull()); + QBool *qbool; + QNum *qnum; + QString *qstr; + QObject *qobj; + + /* bool */ + + qbool =3D qobject_to(QBool, qobject_from_jsonf_nofail("%i", false)); + g_assert(qbool); + g_assert(qbool_get_bool(qbool) =3D=3D false); + qobject_unref(qbool); + + /* Test that non-zero values other than 1 get collapsed to true */ + qbool =3D qobject_to(QBool, qobject_from_jsonf_nofail("%i", 2)); + g_assert(qbool); + g_assert(qbool_get_bool(qbool) =3D=3D true); + qobject_unref(qbool); + + /* number */ + + qnum =3D qobject_to(QNum, qobject_from_jsonf_nofail("%d", value_d)); + g_assert_cmpint(qnum_get_int(qnum), =3D=3D, value_d); + qobject_unref(qnum); + + qnum =3D qobject_to(QNum, qobject_from_jsonf_nofail("%ld", value_ld)); + g_assert_cmpint(qnum_get_int(qnum), =3D=3D, value_ld); + qobject_unref(qnum); + + qnum =3D qobject_to(QNum, qobject_from_jsonf_nofail("%lld", value_lld)= ); + g_assert_cmpint(qnum_get_int(qnum), =3D=3D, value_lld); + qobject_unref(qnum); + + qnum =3D qobject_to(QNum, qobject_from_jsonf_nofail("%u", value_u)); + g_assert_cmpuint(qnum_get_uint(qnum), =3D=3D, value_u); + qobject_unref(qnum); + + qnum =3D qobject_to(QNum, qobject_from_jsonf_nofail("%lu", value_lu)); + g_assert_cmpuint(qnum_get_uint(qnum), =3D=3D, value_lu); + qobject_unref(qnum); + + qnum =3D qobject_to(QNum, qobject_from_jsonf_nofail("%llu", value_llu)= ); + g_assert_cmpuint(qnum_get_uint(qnum), =3D=3D, value_llu); + qobject_unref(qnum); + + qnum =3D qobject_to(QNum, qobject_from_jsonf_nofail("%f", value_f)); + g_assert(qnum_get_double(qnum) =3D=3D value_f); + qobject_unref(qnum); + + /* string */ + + qstr =3D qobject_to(QString, + qobject_from_jsonf_nofail("%s", value_s)); + g_assert_cmpstr(qstring_get_try_str(qstr), =3D=3D, value_s); + qobject_unref(qstr); + + /* object */ + + qobj =3D qobject_from_jsonf_nofail("%p", value_p); + g_assert(qobj =3D=3D value_p); +} + static void simple_dict(void) { int i; @@ -1278,7 +1292,7 @@ static void simple_whitespace(void) } } =20 -static void simple_varargs(void) +static void simple_interpolation(void) { QObject *embedded_obj; QObject *obj; @@ -1470,22 +1484,21 @@ int main(int argc, char **argv) =20 g_test_add_func("/literals/string/escaped", escaped_string); g_test_add_func("/literals/string/utf8", utf8_string); - g_test_add_func("/literals/string/vararg", vararg_string); =20 g_test_add_func("/literals/number/simple", simple_number); g_test_add_func("/literals/number/large", large_number); g_test_add_func("/literals/number/float", float_number); - g_test_add_func("/literals/number/vararg", vararg_number); =20 g_test_add_func("/literals/keyword", keyword_literal); =20 + g_test_add_func("/literals/interpolation", interpolation); + g_test_add_func("/dicts/simple_dict", simple_dict); g_test_add_func("/dicts/large_dict", large_dict); g_test_add_func("/lists/simple_list", simple_list); =20 - g_test_add_func("/whitespace/simple_whitespace", simple_whitespace); - - g_test_add_func("/varargs/simple_varargs", simple_varargs); + g_test_add_func("/mixed/simple_whitespace", simple_whitespace); + g_test_add_func("/mixed/interpolation", simple_interpolation); =20 g_test_add_func("/errors/empty", empty_input); g_test_add_func("/errors/blank", blank_input); --=20 2.17.1 From nobody Wed Nov 5 07:53:13 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533730626754254.5727674603047; Wed, 8 Aug 2018 05:17:06 -0700 (PDT) Received: from localhost ([::1]:43227 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNOj-0001Pu-IO for importer@patchew.org; Wed, 08 Aug 2018 08:17:05 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52369) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBk-0006eL-Rj for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:45 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBi-0008SD-RU for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:40 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:45370 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBi-0008R5-JQ for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:38 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 1ED5F81663CF; Wed, 8 Aug 2018 12:03:38 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E331920180F6; Wed, 8 Aug 2018 12:03:37 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 4B9B511386AE; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:02:54 +0200 Message-Id: <20180808120334.10970-17-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 16/56] json: Fix lexer to include the bad character in JSON_ERROR token X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" json_lexer[] maps (lexer state, input character) to the new lexer state. The input character is consumed unless the new state is terminal and the input character doesn't belong to this token, i.e. the state transition uses look-ahead. When this is the case, input character '\0' would result in the same state transition. TERMINAL_NEEDED_LOOKAHEAD() exploits this. Except this is wrong for transitions to IN_ERROR. There, the offending input character is in fact consumed: case IN_ERROR returns. It isn't added to the JSON_ERROR token, though. Fix that by making TERMINAL_NEEDED_LOOKAHEAD() return false for transitions to IN_ERROR. There's a slight complication. json_lexer_flush() passes input character '\0' to flush an incomplete token. If this results in JSON_ERROR, we'd now add the '\0' to the token. Suppress that. Signed-off-by: Markus Armbruster Reviewed-by: Eric Blake --- qobject/json-lexer.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/qobject/json-lexer.c b/qobject/json-lexer.c index 980ba159d6..7c0875d225 100644 --- a/qobject/json-lexer.c +++ b/qobject/json-lexer.c @@ -76,7 +76,7 @@ QEMU_BUILD_BUG_ON((int)JSON_MIN <=3D (int)IN_START); from OLD_STATE required lookahead. This happens whenever the table below uses the TERMINAL macro. */ #define TERMINAL_NEEDED_LOOKAHEAD(old_state, terminal) \ - (json_lexer[(old_state)][0] =3D=3D (terminal)) + (terminal !=3D IN_ERROR && json_lexer[(old_state)][0] =3D=3D (terminal= )) =20 static const uint8_t json_lexer[][256] =3D { /* Relies on default initialization to IN_ERROR! */ @@ -304,7 +304,7 @@ static int json_lexer_feed_char(JSONLexer *lexer, char = ch, bool flush) assert(lexer->state <=3D ARRAY_SIZE(json_lexer)); new_state =3D json_lexer[lexer->state][(uint8_t)ch]; char_consumed =3D !TERMINAL_NEEDED_LOOKAHEAD(lexer->state, new_sta= te); - if (char_consumed) { + if (char_consumed && !flush) { g_string_append_c(lexer->token, ch); } =20 --=20 2.17.1 From nobody Wed Nov 5 07:53:13 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533731712082165.38365827692166; Wed, 8 Aug 2018 05:35:12 -0700 (PDT) Received: from localhost ([::1]:43351 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNgA-0008W3-8O for importer@patchew.org; Wed, 08 Aug 2018 08:35:06 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52429) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBm-0006g0-HI for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:51 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBj-0008TD-7Z for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:42 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:47698 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBi-0008RW-R3 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:39 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 568507A7F1; Wed, 8 Aug 2018 12:03:38 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 2ECC41C72E; Wed, 8 Aug 2018 12:03:38 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 5097611386B2; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:02:55 +0200 Message-Id: <20180808120334.10970-18-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) for IP:'10.11.54.5' DOMAIN:'int-mx05.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 17/56] json: Reject unescaped control characters X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Fix the lexer to reject unescaped control characters in JSON strings, in accordance with RFC 7159. Bonus: we now recover more nicely from unclosed strings. E.g. {"one: 1}\n{"two": 2} now recovers cleanly after the newline, where before the lexer remained confused until the next unpaired double quote or lexical error. Signed-off-by: Markus Armbruster --- qobject/json-lexer.c | 4 ++-- tests/check-qjson.c | 6 +----- tests/qmp-test.c | 4 ++-- 3 files changed, 5 insertions(+), 9 deletions(-) diff --git a/qobject/json-lexer.c b/qobject/json-lexer.c index 7c0875d225..e85e9a78ff 100644 --- a/qobject/json-lexer.c +++ b/qobject/json-lexer.c @@ -115,7 +115,7 @@ static const uint8_t json_lexer[][256] =3D { ['u'] =3D IN_DQ_UCODE0, }, [IN_DQ_STRING] =3D { - [1 ... 0xBF] =3D IN_DQ_STRING, + [0x20 ... 0xBF] =3D IN_DQ_STRING, [0xC2 ... 0xF4] =3D IN_DQ_STRING, ['\\'] =3D IN_DQ_STRING_ESCAPE, ['"'] =3D JSON_STRING, @@ -155,7 +155,7 @@ static const uint8_t json_lexer[][256] =3D { ['u'] =3D IN_SQ_UCODE0, }, [IN_SQ_STRING] =3D { - [1 ... 0xBF] =3D IN_SQ_STRING, + [0x20 ... 0xBF] =3D IN_SQ_STRING, [0xC2 ... 0xF4] =3D IN_SQ_STRING, ['\\'] =3D IN_SQ_STRING_ESCAPE, ['\''] =3D JSON_STRING, diff --git a/tests/check-qjson.c b/tests/check-qjson.c index fda2b014a3..7d8ce5c68d 100644 --- a/tests/check-qjson.c +++ b/tests/check-qjson.c @@ -171,11 +171,7 @@ static void utf8_string(void) "\x08\x09\x0A\x0B\x0C\x0D\x0E\x0F" "\x10\x11\x12\x13\x14\x15\x16\x17" "\x18\x19\x1A\x1B\x1C\x1D\x1E\x1F", - /* bug: not corrected (valid UTF-8, but invalid JSON) */ - "\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0A\x0B\x0C\x0D\x0E\x0F" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1A\x1B\x1C\x1D\x1E\x1F", + NULL, "\\u0001\\u0002\\u0003\\u0004\\u0005\\u0006\\u0007" "\\b\\t\\n\\u000B\\f\\r\\u000E\\u000F" "\\u0010\\u0011\\u0012\\u0013\\u0014\\u0015\\u0016\\u0017" diff --git a/tests/qmp-test.c b/tests/qmp-test.c index 5117a1ab25..b77987b644 100644 --- a/tests/qmp-test.c +++ b/tests/qmp-test.c @@ -86,9 +86,9 @@ static void test_malformed(QTestState *qts) g_assert(recovered(qts)); =20 /* lexical error: control character in string */ - qtest_qmp_send_raw(qts, "{'execute': 'nonexistent', 'id':'\n'}"); + qtest_qmp_send_raw(qts, "{'execute': 'nonexistent', 'id':'\n"); resp =3D qtest_qmp_receive(qts); - g_assert_cmpstr(get_error_class(resp), =3D=3D, "CommandNotFound"); /* = BUG */ + g_assert_cmpstr(get_error_class(resp), =3D=3D, "GenericError"); qobject_unref(resp); g_assert(recovered(qts)); =20 --=20 2.17.1 From nobody Wed Nov 5 07:53:13 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533731035140103.8182901556479; Wed, 8 Aug 2018 05:23:55 -0700 (PDT) Received: from localhost ([::1]:43292 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNVJ-0007EI-SF for importer@patchew.org; Wed, 08 Aug 2018 08:23:53 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52484) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBn-0006hF-Q2 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:52 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBj-0008Tn-De for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:43 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:47706 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBj-0008Rn-1x for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:39 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 7A6727A7F4; Wed, 8 Aug 2018 12:03:38 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 32521111AF00; Wed, 8 Aug 2018 12:03:38 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 55AC011386B6; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:02:56 +0200 Message-Id: <20180808120334.10970-19-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) for IP:'10.11.54.3' DOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 18/56] json: Revamp lexer documentation X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Signed-off-by: Markus Armbruster Reviewed-by: Eric Blake --- qobject/json-lexer.c | 80 +++++++++++++++++++++++++++++++++++++++----- 1 file changed, 71 insertions(+), 9 deletions(-) diff --git a/qobject/json-lexer.c b/qobject/json-lexer.c index e85e9a78ff..109a7d8bb8 100644 --- a/qobject/json-lexer.c +++ b/qobject/json-lexer.c @@ -18,21 +18,83 @@ #define MAX_TOKEN_SIZE (64ULL << 20) =20 /* - * Required by JSON (RFC 7159): + * From RFC 7159 "The JavaScript Object Notation (JSON) Data + * Interchange Format", with [comments in brackets]: * - * \"([^\\\"]|\\[\"'\\/bfnrt]|\\u[0-9a-fA-F]{4})*\" - * -?(0|[1-9][0-9]*)(.[0-9]+)?([eE][-+]?[0-9]+)? - * [{}\[\],:] - * [a-z]+ # covers null, true, false + * The set of tokens includes six structural characters, strings, + * numbers, and three literal names. * - * Extension of '' strings: + * These are the six structural characters: * - * '([^\\']|\\[\"'\\/bfnrt]|\\u[0-9a-fA-F]{4})*' + * begin-array =3D ws %x5B ws ; [ left square bracket + * begin-object =3D ws %x7B ws ; { left curly bracket + * end-array =3D ws %x5D ws ; ] right square bracket + * end-object =3D ws %x7D ws ; } right curly bracket + * name-separator =3D ws %x3A ws ; : colon + * value-separator =3D ws %x2C ws ; , comma * - * Extension for vararg handling in JSON construction: + * Insignificant whitespace is allowed before or after any of the six + * structural characters. + * [This lexer accepts it before or after any token, which is actually + * the same, as the grammar always has structural characters between + * other tokens.] * - * %((l|ll|I64)?d|[ipsf]) + * ws =3D *( + * %x20 / ; Space + * %x09 / ; Horizontal tab + * %x0A / ; Line feed or New line + * %x0D ) ; Carriage return * + * [...] three literal names: + * false null true + * [This lexer accepts [a-z]+, and leaves rejecting unknown literal + * names to the parser.] + * + * [Numbers:] + * + * number =3D [ minus ] int [ frac ] [ exp ] + * decimal-point =3D %x2E ; . + * digit1-9 =3D %x31-39 ; 1-9 + * e =3D %x65 / %x45 ; e E + * exp =3D e [ minus / plus ] 1*DIGIT + * frac =3D decimal-point 1*DIGIT + * int =3D zero / ( digit1-9 *DIGIT ) + * minus =3D %x2D ; - + * plus =3D %x2B ; + + * zero =3D %x30 ; 0 + * + * [Strings:] + * string =3D quotation-mark *char quotation-mark + * + * char =3D unescaped / + * escape ( + * %x22 / ; " quotation mark U+0022 + * %x5C / ; \ reverse solidus U+005C + * %x2F / ; / solidus U+002F + * %x62 / ; b backspace U+0008 + * %x66 / ; f form feed U+000C + * %x6E / ; n line feed U+000A + * %x72 / ; r carriage return U+000D + * %x74 / ; t tab U+0009 + * %x75 4HEXDIG ) ; uXXXX U+XXXX + * escape =3D %x5C ; \ + * quotation-mark =3D %x22 ; " + * unescaped =3D %x20-21 / %x23-5B / %x5D-10FFFF + * + * + * Extensions over RFC 7159: + * - Extra escape sequence in strings: + * 0x27 (apostrophe) is recognized after escape, too + * - Single-quoted strings: + * Like double-quoted strings, except they're delimited by %x27 + * (apostrophe) instead of %x22 (quotation mark), and can't contain + * unescaped apostrophe, but can contain unescaped quotation mark. + * - Interpolation: + * interpolation =3D %((l|ll|I64)[du]|[ipsf]) + * + * Note: + * - Input must be encoded in UTF-8. + * - Decoding and validating is left to the parser. */ =20 enum json_lexer_state { --=20 2.17.1 From nobody Wed Nov 5 07:53:13 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533731853183859.4170197357553; Wed, 8 Aug 2018 05:37:33 -0700 (PDT) Received: from localhost ([::1]:43369 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNiW-0002Eh-1s for importer@patchew.org; Wed, 08 Aug 2018 08:37:32 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52566) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBp-0006jd-Sq for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:56 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBj-0008TM-6k for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:45 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:45378 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBi-0008Rc-SR for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:39 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 5E10B81663F3; Wed, 8 Aug 2018 12:03:38 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 35EC31C72F; Wed, 8 Aug 2018 12:03:38 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 5AC3911386B9; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:02:57 +0200 Message-Id: <20180808120334.10970-20-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) for IP:'10.11.54.5' DOMAIN:'int-mx05.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 19/56] json: Tighten and simplify qstring_from_escaped_str()'s loop X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Simplify loop control, and assert that the string ends with the appropriate quote (the lexer ensures it does). Signed-off-by: Markus Armbruster Reviewed-by: Eric Blake --- qobject/json-parser.c | 35 +++++++++-------------------------- 1 file changed, 9 insertions(+), 26 deletions(-) diff --git a/qobject/json-parser.c b/qobject/json-parser.c index a5aa790d62..e00405745f 100644 --- a/qobject/json-parser.c +++ b/qobject/json-parser.c @@ -132,66 +132,50 @@ static QString *qstring_from_escaped_str(JSONParserCo= ntext *ctxt, { const char *ptr =3D token->str; QString *str; - int double_quote =3D 1; - - if (*ptr =3D=3D '"') { - double_quote =3D 1; - } else { - double_quote =3D 0; - } - ptr++; + char quote; =20 + assert(*ptr =3D=3D '"' || *ptr =3D=3D '\''); + quote =3D *ptr; str =3D qstring_new(); - while (*ptr &&=20 - ((double_quote && *ptr !=3D '"') || (!double_quote && *ptr !=3D= '\''))) { + + while (*++ptr !=3D quote) { + assert(*ptr); if (*ptr =3D=3D '\\') { - ptr++; - - switch (*ptr) { + switch (*++ptr) { case '"': qstring_append(str, "\""); - ptr++; break; case '\'': qstring_append(str, "'"); - ptr++; break; case '\\': qstring_append(str, "\\"); - ptr++; break; case '/': qstring_append(str, "/"); - ptr++; break; case 'b': qstring_append(str, "\b"); - ptr++; break; case 'f': qstring_append(str, "\f"); - ptr++; break; case 'n': qstring_append(str, "\n"); - ptr++; break; case 'r': qstring_append(str, "\r"); - ptr++; break; case 't': qstring_append(str, "\t"); - ptr++; break; case 'u': { uint16_t unicode_char =3D 0; char utf8_char[4]; int i =3D 0; =20 - ptr++; - for (i =3D 0; i < 4; i++) { + ptr++; if (qemu_isxdigit(*ptr)) { unicode_char |=3D hex2decimal(*ptr) << ((3 - i) * = 4); } else { @@ -199,7 +183,6 @@ static QString *qstring_from_escaped_str(JSONParserCont= ext *ctxt, "invalid hex escape sequence in string= "); goto out; } - ptr++; } =20 wchar_to_utf8(unicode_char, utf8_char, sizeof(utf8_char)); @@ -212,7 +195,7 @@ static QString *qstring_from_escaped_str(JSONParserCont= ext *ctxt, } else { char dummy[2]; =20 - dummy[0] =3D *ptr++; + dummy[0] =3D *ptr; dummy[1] =3D 0; =20 qstring_append(str, dummy); --=20 2.17.1 From nobody Wed Nov 5 07:53:13 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533731132162882.2700097871178; Wed, 8 Aug 2018 05:25:32 -0700 (PDT) Received: from localhost ([::1]:43301 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNWs-0000HB-RR for importer@patchew.org; Wed, 08 Aug 2018 08:25:30 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52533) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBp-0006iq-7v for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:54 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBj-0008UG-IU for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:45 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:45386 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBj-0008SM-75 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:39 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id A3C6481663F8; Wed, 8 Aug 2018 12:03:38 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 3A9EF1040840; Wed, 8 Aug 2018 12:03:38 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 5FEED11386BF; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:02:58 +0200 Message-Id: <20180808120334.10970-21-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) for IP:'10.11.54.3' DOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 20/56] check-qjson: Document we expect invalid UTF-8 to be rejected X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The JSON parser rejects some invalid sequences, but accepts others without correcting the problem. We should either reject all invalid sequences, or minimize overlong sequences and replace all other invalid sequences by a suitable replacement character. A common choice for replacement is U+FFFD. I'm going to implement the former. Update the comments in utf8_string() to expect this. Signed-off-by: Markus Armbruster --- tests/check-qjson.c | 151 +++++++++++++++++++++----------------------- 1 file changed, 71 insertions(+), 80 deletions(-) diff --git a/tests/check-qjson.c b/tests/check-qjson.c index 7d8ce5c68d..8ce047fad0 100644 --- a/tests/check-qjson.c +++ b/tests/check-qjson.c @@ -126,13 +126,7 @@ static void utf8_string(void) * They're all marked "bug:" below, and are to be replaced by * correct ones as the bugs get fixed. * - * The JSON parser rejects some invalid sequences, but accepts - * others without correcting the problem. - * - * We should either reject all invalid sequences, or minimize - * overlong sequences and replace all other invalid sequences by a - * suitable replacement character. A common choice for - * replacement is U+FFFD. + * The JSON parser rejects some, but not all invalid sequences. * * Problem: we can't easily deal with embedded U+0000. Parsing * the JSON string "this \\u0000" is fun" yields "this \0 is fun", @@ -154,11 +148,8 @@ static void utf8_string(void) } test_cases[] =3D { /* * Bug markers used here: - * - bug: not corrected - * JSON parser fails to correct invalid sequence(s) - * - bug: rejected - * JSON parser rejects invalid sequence(s) - * We may choose to define this as feature + * - bug: not rejected + * JSON parser fails to reject invalid sequence(s) */ =20 /* 0 Control characters */ @@ -226,13 +217,13 @@ static void utf8_string(void) /* 2.1.5 5 bytes U+200000 */ { "\xF8\x88\x80\x80\x80", - NULL, /* bug: rejected */ + NULL, "\\uFFFD", }, /* 2.1.6 6 bytes U+4000000 */ { "\xFC\x84\x80\x80\x80\x80", - NULL, /* bug: rejected */ + NULL, "\\uFFFD", }, /* 2.2 Last possible sequence of a certain length */ @@ -265,19 +256,19 @@ static void utf8_string(void) /* 2.2.4 4 bytes U+1FFFFF */ { "\xF7\xBF\xBF\xBF", - NULL, /* bug: rejected */ + NULL, "\\uFFFD", }, /* 2.2.5 5 bytes U+3FFFFFF */ { "\xFB\xBF\xBF\xBF\xBF", - NULL, /* bug: rejected */ + NULL, "\\uFFFD", }, /* 2.2.6 6 bytes U+7FFFFFFF */ { "\xFD\xBF\xBF\xBF\xBF\xBF", - NULL, /* bug: rejected */ + NULL, "\\uFFFD", }, /* 2.3 Other boundary conditions */ @@ -316,49 +307,49 @@ static void utf8_string(void) /* 3.1.1 First continuation byte */ { "\x80", - "\x80", /* bug: not corrected */ + "\x80", /* bug: not rejected */ "\\uFFFD", }, /* 3.1.2 Last continuation byte */ { "\xBF", - "\xBF", /* bug: not corrected */ + "\xBF", /* bug: not rejected */ "\\uFFFD", }, /* 3.1.3 2 continuation bytes */ { "\x80\xBF", - "\x80\xBF", /* bug: not corrected */ + "\x80\xBF", /* bug: not rejected */ "\\uFFFD\\uFFFD", }, /* 3.1.4 3 continuation bytes */ { "\x80\xBF\x80", - "\x80\xBF\x80", /* bug: not corrected */ + "\x80\xBF\x80", /* bug: not rejected */ "\\uFFFD\\uFFFD\\uFFFD", }, /* 3.1.5 4 continuation bytes */ { "\x80\xBF\x80\xBF", - "\x80\xBF\x80\xBF", /* bug: not corrected */ + "\x80\xBF\x80\xBF", /* bug: not rejected */ "\\uFFFD\\uFFFD\\uFFFD\\uFFFD", }, /* 3.1.6 5 continuation bytes */ { "\x80\xBF\x80\xBF\x80", - "\x80\xBF\x80\xBF\x80", /* bug: not corrected */ + "\x80\xBF\x80\xBF\x80", /* bug: not rejected */ "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD", }, /* 3.1.7 6 continuation bytes */ { "\x80\xBF\x80\xBF\x80\xBF", - "\x80\xBF\x80\xBF\x80\xBF", /* bug: not corrected */ + "\x80\xBF\x80\xBF\x80\xBF", /* bug: not rejected */ "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD", }, /* 3.1.8 7 continuation bytes */ { "\x80\xBF\x80\xBF\x80\xBF\x80", - "\x80\xBF\x80\xBF\x80\xBF\x80", /* bug: not corrected */ + "\x80\xBF\x80\xBF\x80\xBF\x80", /* bug: not rejected */ "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD", }, /* 3.1.9 Sequence of all 64 possible continuation bytes */ @@ -371,7 +362,7 @@ static void utf8_string(void) "\xA8\xA9\xAA\xAB\xAC\xAD\xAE\xAF" "\xB0\xB1\xB2\xB3\xB4\xB5\xB6\xB7" "\xB8\xB9\xBA\xBB\xBC\xBD\xBE\xBF", - /* bug: not corrected */ + /* bug: not rejected */ "\x80\x81\x82\x83\x84\x85\x86\x87" "\x88\x89\x8A\x8B\x8C\x8D\x8E\x8F" "\x90\x91\x92\x93\x94\x95\x96\x97" @@ -396,7 +387,7 @@ static void utf8_string(void) "\xC8 \xC9 \xCA \xCB \xCC \xCD \xCE \xCF " "\xD0 \xD1 \xD2 \xD3 \xD4 \xD5 \xD6 \xD7 " "\xD8 \xD9 \xDA \xDB \xDC \xDD \xDE \xDF ", - NULL, /* bug: rejected (partly, see FIXME below)= */ + NULL, /* bug: accepted partly, see FIXME below */ "\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFF= FD " "\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFF= FD " "\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFF= FD " @@ -406,7 +397,7 @@ static void utf8_string(void) { "\xE0 \xE1 \xE2 \xE3 \xE4 \xE5 \xE6 \xE7 " "\xE8 \xE9 \xEA \xEB \xEC \xED \xEE \xEF ", - /* bug: not corrected */ + /* bug: not rejected */ "\xE0 \xE1 \xE2 \xE3 \xE4 \xE5 \xE6 \xE7 " "\xE8 \xE9 \xEA \xEB \xEC \xED \xEE \xEF ", "\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFF= FD " @@ -415,131 +406,131 @@ static void utf8_string(void) /* 3.2.3 All 8 first bytes of 4-byte sequences, followed by space= */ { "\xF0 \xF1 \xF2 \xF3 \xF4 \xF5 \xF6 \xF7 ", - NULL, /* bug: rejected (partly, see FIXME below)= */ + NULL, /* bug: accepted partly, see FIXME below */ "\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFF= FD ", }, /* 3.2.4 All 4 first bytes of 5-byte sequences, followed by space= */ { "\xF8 \xF9 \xFA \xFB ", - NULL, /* bug: rejected */ + NULL, "\\uFFFD \\uFFFD \\uFFFD \\uFFFD ", }, /* 3.2.5 All 2 first bytes of 6-byte sequences, followed by space= */ { "\xFC \xFD ", - NULL, /* bug: rejected */ + NULL, "\\uFFFD \\uFFFD ", }, /* 3.3 Sequences with last continuation byte missing */ /* 3.3.1 2-byte sequence with last byte missing (U+0000) */ { "\xC0", - NULL, /* bug: rejected */ + NULL, "\\uFFFD", }, /* 3.3.2 3-byte sequence with last byte missing (U+0000) */ { "\xE0\x80", - "\xE0\x80", /* bug: not corrected */ + "\xE0\x80", /* bug: not rejected */ "\\uFFFD", }, /* 3.3.3 4-byte sequence with last byte missing (U+0000) */ { "\xF0\x80\x80", - "\xF0\x80\x80", /* bug: not corrected */ + "\xF0\x80\x80", /* bug: not rejected */ "\\uFFFD", }, /* 3.3.4 5-byte sequence with last byte missing (U+0000) */ { "\xF8\x80\x80\x80", - NULL, /* bug: rejected */ + NULL, "\\uFFFD", }, /* 3.3.5 6-byte sequence with last byte missing (U+0000) */ { "\xFC\x80\x80\x80\x80", - NULL, /* bug: rejected */ + NULL, "\\uFFFD", }, /* 3.3.6 2-byte sequence with last byte missing (U+07FF) */ { "\xDF", - "\xDF", /* bug: not corrected */ + "\xDF", /* bug: not rejected */ "\\uFFFD", }, /* 3.3.7 3-byte sequence with last byte missing (U+FFFF) */ { "\xEF\xBF", - "\xEF\xBF", /* bug: not corrected */ + "\xEF\xBF", /* bug: not rejected */ "\\uFFFD", }, /* 3.3.8 4-byte sequence with last byte missing (U+1FFFFF) */ { "\xF7\xBF\xBF", - NULL, /* bug: rejected */ + NULL, "\\uFFFD", }, /* 3.3.9 5-byte sequence with last byte missing (U+3FFFFFF) */ { "\xFB\xBF\xBF\xBF", - NULL, /* bug: rejected */ + NULL, "\\uFFFD", }, /* 3.3.10 6-byte sequence with last byte missing (U+7FFFFFFF) */ { "\xFD\xBF\xBF\xBF\xBF", - NULL, /* bug: rejected */ + NULL, "\\uFFFD", }, /* 3.4 Concatenation of incomplete sequences */ { "\xC0\xE0\x80\xF0\x80\x80\xF8\x80\x80\x80\xFC\x80\x80\x80\x80" "\xDF\xEF\xBF\xF7\xBF\xBF\xFB\xBF\xBF\xBF\xFD\xBF\xBF\xBF\xBF", - NULL, /* bug: rejected (partly, see FIXME below)= */ + NULL, /* bug: accepted partly, see FIXME below */ "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD" "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD", }, /* 3.5 Impossible bytes */ { "\xFE", - NULL, /* bug: rejected */ + NULL, "\\uFFFD", }, { "\xFF", - NULL, /* bug: rejected */ + NULL, "\\uFFFD", }, { "\xFE\xFE\xFF\xFF", - NULL, /* bug: rejected */ + NULL, "\\uFFFD\\uFFFD\\uFFFD\\uFFFD", }, /* 4 Overlong sequences */ /* 4.1 Overlong '/' */ { "\xC0\xAF", - NULL, /* bug: rejected */ + NULL, "\\uFFFD", }, { "\xE0\x80\xAF", - "\xE0\x80\xAF", /* bug: not corrected */ + "\xE0\x80\xAF", /* bug: not rejected */ "\\uFFFD", }, { "\xF0\x80\x80\xAF", - "\xF0\x80\x80\xAF", /* bug: not corrected */ + "\xF0\x80\x80\xAF", /* bug: not rejected */ "\\uFFFD", }, { "\xF8\x80\x80\x80\xAF", - NULL, /* bug: rejected */ + NULL, "\\uFFFD", }, { "\xFC\x80\x80\x80\x80\xAF", - NULL, /* bug: rejected */ + NULL, "\\uFFFD", }, /* @@ -551,13 +542,13 @@ static void utf8_string(void) { /* \U+007F */ "\xC1\xBF", - NULL, /* bug: rejected */ + NULL, "\\uFFFD", }, { /* \U+07FF */ "\xE0\x9F\xBF", - "\xE0\x9F\xBF", /* bug: not corrected */ + "\xE0\x9F\xBF", /* bug: not rejected */ "\\uFFFD", }, { @@ -568,50 +559,50 @@ static void utf8_string(void) * also 2.2.3 */ "\xF0\x8F\xBF\xBC", - "\xF0\x8F\xBF\xBC", /* bug: not corrected */ + "\xF0\x8F\xBF\xBC", /* bug: not rejected */ "\\uFFFD", }, { /* \U+1FFFFF */ "\xF8\x87\xBF\xBF\xBF", - NULL, /* bug: rejected */ + NULL, "\\uFFFD", }, { /* \U+3FFFFFF */ "\xFC\x83\xBF\xBF\xBF\xBF", - NULL, /* bug: rejected */ + NULL, "\\uFFFD", }, /* 4.3 Overlong representation of the NUL character */ { /* \U+0000 */ "\xC0\x80", - NULL, /* bug: rejected */ + NULL, "\\u0000", }, { /* \U+0000 */ "\xE0\x80\x80", - "\xE0\x80\x80", /* bug: not corrected */ + "\xE0\x80\x80", /* bug: not rejected */ "\\uFFFD", }, { /* \U+0000 */ "\xF0\x80\x80\x80", - "\xF0\x80\x80\x80", /* bug: not corrected */ + "\xF0\x80\x80\x80", /* bug: not rejected */ "\\uFFFD", }, { /* \U+0000 */ "\xF8\x80\x80\x80\x80", - NULL, /* bug: rejected */ + NULL, "\\uFFFD", }, { /* \U+0000 */ "\xFC\x80\x80\x80\x80\x80", - NULL, /* bug: rejected */ + NULL, "\\uFFFD", }, /* 5 Illegal code positions */ @@ -619,92 +610,92 @@ static void utf8_string(void) { /* \U+D800 */ "\xED\xA0\x80", - "\xED\xA0\x80", /* bug: not corrected */ + "\xED\xA0\x80", /* bug: not rejected */ "\\uFFFD", }, { /* \U+DB7F */ "\xED\xAD\xBF", - "\xED\xAD\xBF", /* bug: not corrected */ + "\xED\xAD\xBF", /* bug: not rejected */ "\\uFFFD", }, { /* \U+DB80 */ "\xED\xAE\x80", - "\xED\xAE\x80", /* bug: not corrected */ + "\xED\xAE\x80", /* bug: not rejected */ "\\uFFFD", }, { /* \U+DBFF */ "\xED\xAF\xBF", - "\xED\xAF\xBF", /* bug: not corrected */ + "\xED\xAF\xBF", /* bug: not rejected */ "\\uFFFD", }, { /* \U+DC00 */ "\xED\xB0\x80", - "\xED\xB0\x80", /* bug: not corrected */ + "\xED\xB0\x80", /* bug: not rejected */ "\\uFFFD", }, { /* \U+DF80 */ "\xED\xBE\x80", - "\xED\xBE\x80", /* bug: not corrected */ + "\xED\xBE\x80", /* bug: not rejected */ "\\uFFFD", }, { /* \U+DFFF */ "\xED\xBF\xBF", - "\xED\xBF\xBF", /* bug: not corrected */ + "\xED\xBF\xBF", /* bug: not rejected */ "\\uFFFD", }, /* 5.2 Paired UTF-16 surrogates */ { /* \U+D800\U+DC00 */ "\xED\xA0\x80\xED\xB0\x80", - "\xED\xA0\x80\xED\xB0\x80", /* bug: not corrected */ + "\xED\xA0\x80\xED\xB0\x80", /* bug: not rejected */ "\\uFFFD\\uFFFD", }, { /* \U+D800\U+DFFF */ "\xED\xA0\x80\xED\xBF\xBF", - "\xED\xA0\x80\xED\xBF\xBF", /* bug: not corrected */ + "\xED\xA0\x80\xED\xBF\xBF", /* bug: not rejected */ "\\uFFFD\\uFFFD", }, { /* \U+DB7F\U+DC00 */ "\xED\xAD\xBF\xED\xB0\x80", - "\xED\xAD\xBF\xED\xB0\x80", /* bug: not corrected */ + "\xED\xAD\xBF\xED\xB0\x80", /* bug: not rejected */ "\\uFFFD\\uFFFD", }, { /* \U+DB7F\U+DFFF */ "\xED\xAD\xBF\xED\xBF\xBF", - "\xED\xAD\xBF\xED\xBF\xBF", /* bug: not corrected */ + "\xED\xAD\xBF\xED\xBF\xBF", /* bug: not rejected */ "\\uFFFD\\uFFFD", }, { /* \U+DB80\U+DC00 */ "\xED\xAE\x80\xED\xB0\x80", - "\xED\xAE\x80\xED\xB0\x80", /* bug: not corrected */ + "\xED\xAE\x80\xED\xB0\x80", /* bug: not rejected */ "\\uFFFD\\uFFFD", }, { /* \U+DB80\U+DFFF */ "\xED\xAE\x80\xED\xBF\xBF", - "\xED\xAE\x80\xED\xBF\xBF", /* bug: not corrected */ + "\xED\xAE\x80\xED\xBF\xBF", /* bug: not rejected */ "\\uFFFD\\uFFFD", }, { /* \U+DBFF\U+DC00 */ "\xED\xAF\xBF\xED\xB0\x80", - "\xED\xAF\xBF\xED\xB0\x80", /* bug: not corrected */ + "\xED\xAF\xBF\xED\xB0\x80", /* bug: not rejected */ "\\uFFFD\\uFFFD", }, { /* \U+DBFF\U+DFFF */ "\xED\xAF\xBF\xED\xBF\xBF", - "\xED\xAF\xBF\xED\xBF\xBF", /* bug: not corrected */ + "\xED\xAF\xBF\xED\xBF\xBF", /* bug: not rejected */ "\\uFFFD\\uFFFD", }, /* 5.3 Other illegal code positions */ @@ -712,25 +703,25 @@ static void utf8_string(void) { /* \U+FFFE */ "\xEF\xBF\xBE", - "\xEF\xBF\xBE", /* bug: not corrected */ + "\xEF\xBF\xBE", /* bug: not rejected */ "\\uFFFD", }, { /* \U+FFFF */ "\xEF\xBF\xBF", - "\xEF\xBF\xBF", /* bug: not corrected */ + "\xEF\xBF\xBF", /* bug: not rejected */ "\\uFFFD", }, { /* U+FDD0 */ "\xEF\xB7\x90", - "\xEF\xB7\x90", /* bug: not corrected */ + "\xEF\xB7\x90", /* bug: not rejected */ "\\uFFFD", }, { /* U+FDEF */ "\xEF\xB7\xAF", - "\xEF\xB7\xAF", /* bug: not corrected */ + "\xEF\xB7\xAF", /* bug: not rejected */ "\\uFFFD", }, /* Plane 1 .. 16 noncharacters */ @@ -752,7 +743,7 @@ static void utf8_string(void) "\xF3\xAF\xBF\xBE\xF3\xAF\xBF\xBF" "\xF3\xBF\xBF\xBE\xF3\xBF\xBF\xBF" "\xF4\x8F\xBF\xBE\xF4\x8F\xBF\xBF", - /* bug: not corrected */ + /* bug: not rejected */ "\xF0\x9F\xBF\xBE\xF0\x9F\xBF\xBF" "\xF0\xAF\xBF\xBE\xF0\xAF\xBF\xBF" "\xF0\xBF\xBF\xBE\xF0\xBF\xBF\xBF" --=20 2.17.1 From nobody Wed Nov 5 07:53:13 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533732576798703.3790571702058; Wed, 8 Aug 2018 05:49:36 -0700 (PDT) Received: from localhost ([::1]:43460 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNuB-0004Hf-H5 for importer@patchew.org; Wed, 08 Aug 2018 08:49:35 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52660) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBt-0006ok-2i for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:59 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBj-0008Uo-LU for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:49 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:47712 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBj-0008Si-9I for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:39 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id B939272620; Wed, 8 Aug 2018 12:03:38 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 4C86221568A1; Wed, 8 Aug 2018 12:03:38 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 650AA11386C0; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:02:59 +0200 Message-Id: <20180808120334.10970-22-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 21/56] json: Reject invalid UTF-8 sequences X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" We reject bytes that can't occur in valid UTF-8 (\xC0..\xC1, \xF5..\xFF in the lexer. That's insufficient; there's plenty of invalid UTF-8 not containing these bytes, as demonstrated by check-qjson: * Malformed sequences - Unexpected continuation bytes - Missing continuation bytes after start bytes other than \xC0..\xC1, \xF5..\xFD. * Overlong sequences with start bytes other than \xC0..\xC1, \xF5..\xFD. * Invalid code points Fixing this in the lexer would be bothersome. Fixing it in the parser is straightforward, so do that. Signed-off-by: Markus Armbruster --- include/qemu/unicode.h | 1 + qobject/json-parser.c | 20 ++++-- tests/check-qjson.c | 137 ++++++++++++++--------------------------- util/unicode.c | 69 ++++++++++++++++++--- 4 files changed, 122 insertions(+), 105 deletions(-) diff --git a/include/qemu/unicode.h b/include/qemu/unicode.h index 71c72db461..7fa10b8e60 100644 --- a/include/qemu/unicode.h +++ b/include/qemu/unicode.h @@ -2,5 +2,6 @@ #define QEMU_UNICODE_H =20 int mod_utf8_codepoint(const char *s, size_t n, char **end); +ssize_t mod_utf8_encode(char buf[], size_t bufsz, int codepoint); =20 #endif diff --git a/qobject/json-parser.c b/qobject/json-parser.c index e00405745f..6b60b07e09 100644 --- a/qobject/json-parser.c +++ b/qobject/json-parser.c @@ -13,6 +13,7 @@ =20 #include "qemu/osdep.h" #include "qemu/cutils.h" +#include "qemu/unicode.h" #include "qapi/error.h" #include "qemu-common.h" #include "qapi/qmp/qbool.h" @@ -133,6 +134,10 @@ static QString *qstring_from_escaped_str(JSONParserCon= text *ctxt, const char *ptr =3D token->str; QString *str; char quote; + int cp; + char *end; + ssize_t len; + char utf8_buf[5]; =20 assert(*ptr =3D=3D '"' || *ptr =3D=3D '\''); quote =3D *ptr; @@ -193,12 +198,15 @@ static QString *qstring_from_escaped_str(JSONParserCo= ntext *ctxt, goto out; } } else { - char dummy[2]; - - dummy[0] =3D *ptr; - dummy[1] =3D 0; - - qstring_append(str, dummy); + cp =3D mod_utf8_codepoint(ptr, 6, &end); + if (cp <=3D 0) { + parse_error(ctxt, token, "invalid UTF-8 sequence in string= "); + goto out; + } + ptr =3D end - 1; + len =3D mod_utf8_encode(utf8_buf, sizeof(utf8_buf), cp); + assert(len >=3D 0); + qstring_append(str, utf8_buf); } } =20 diff --git a/tests/check-qjson.c b/tests/check-qjson.c index 8ce047fad0..e00be5b023 100644 --- a/tests/check-qjson.c +++ b/tests/check-qjson.c @@ -121,13 +121,6 @@ static void escaped_string(void) static void utf8_string(void) { /* - * FIXME Current behavior for invalid UTF-8 sequences is - * incorrect. This test expects current, incorrect results. - * They're all marked "bug:" below, and are to be replaced by - * correct ones as the bugs get fixed. - * - * The JSON parser rejects some, but not all invalid sequences. - * * Problem: we can't easily deal with embedded U+0000. Parsing * the JSON string "this \\u0000" is fun" yields "this \0 is fun", * which gets misinterpreted as NUL-terminated "this ". We should @@ -146,12 +139,6 @@ static void utf8_string(void) /* Expected unparse output, defaults to @json_in */ const char *json_out; } test_cases[] =3D { - /* - * Bug markers used here: - * - bug: not rejected - * JSON parser fails to reject invalid sequence(s) - */ - /* 0 Control characters */ { /* @@ -299,7 +286,7 @@ static void utf8_string(void) { /* first one beyond Unicode range: U+110000 */ "\xF4\x90\x80\x80", - "\xF4\x90\x80\x80", + NULL, "\\uFFFD", }, /* 3 Malformed sequences */ @@ -307,49 +294,49 @@ static void utf8_string(void) /* 3.1.1 First continuation byte */ { "\x80", - "\x80", /* bug: not rejected */ + NULL, "\\uFFFD", }, /* 3.1.2 Last continuation byte */ { "\xBF", - "\xBF", /* bug: not rejected */ + NULL, "\\uFFFD", }, /* 3.1.3 2 continuation bytes */ { "\x80\xBF", - "\x80\xBF", /* bug: not rejected */ + NULL, "\\uFFFD\\uFFFD", }, /* 3.1.4 3 continuation bytes */ { "\x80\xBF\x80", - "\x80\xBF\x80", /* bug: not rejected */ + NULL, "\\uFFFD\\uFFFD\\uFFFD", }, /* 3.1.5 4 continuation bytes */ { "\x80\xBF\x80\xBF", - "\x80\xBF\x80\xBF", /* bug: not rejected */ + NULL, "\\uFFFD\\uFFFD\\uFFFD\\uFFFD", }, /* 3.1.6 5 continuation bytes */ { "\x80\xBF\x80\xBF\x80", - "\x80\xBF\x80\xBF\x80", /* bug: not rejected */ + NULL, "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD", }, /* 3.1.7 6 continuation bytes */ { "\x80\xBF\x80\xBF\x80\xBF", - "\x80\xBF\x80\xBF\x80\xBF", /* bug: not rejected */ + NULL, "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD", }, /* 3.1.8 7 continuation bytes */ { "\x80\xBF\x80\xBF\x80\xBF\x80", - "\x80\xBF\x80\xBF\x80\xBF\x80", /* bug: not rejected */ + NULL, "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD", }, /* 3.1.9 Sequence of all 64 possible continuation bytes */ @@ -362,16 +349,7 @@ static void utf8_string(void) "\xA8\xA9\xAA\xAB\xAC\xAD\xAE\xAF" "\xB0\xB1\xB2\xB3\xB4\xB5\xB6\xB7" "\xB8\xB9\xBA\xBB\xBC\xBD\xBE\xBF", - /* bug: not rejected */ - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8A\x8B\x8C\x8D\x8E\x8F" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9A\x9B\x9C\x9D\x9E\x9F" - "\xA0\xA1\xA2\xA3\xA4\xA5\xA6\xA7" - "\xA8\xA9\xAA\xAB\xAC\xAD\xAE\xAF" - "\xB0\xB1\xB2\xB3\xB4\xB5\xB6\xB7" - "\xB8\xB9\xBA\xBB\xBC\xBD\xBE\xBF", - "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD" + NULL, "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD" "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD" "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD" @@ -379,6 +357,7 @@ static void utf8_string(void) "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD" "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD" "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD" + "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD", }, /* 3.2 Lonely start characters */ /* 3.2.1 All 32 first bytes of 2-byte sequences, followed by spac= e */ @@ -387,7 +366,7 @@ static void utf8_string(void) "\xC8 \xC9 \xCA \xCB \xCC \xCD \xCE \xCF " "\xD0 \xD1 \xD2 \xD3 \xD4 \xD5 \xD6 \xD7 " "\xD8 \xD9 \xDA \xDB \xDC \xDD \xDE \xDF ", - NULL, /* bug: accepted partly, see FIXME below */ + NULL, "\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFF= FD " "\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFF= FD " "\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFF= FD " @@ -397,16 +376,14 @@ static void utf8_string(void) { "\xE0 \xE1 \xE2 \xE3 \xE4 \xE5 \xE6 \xE7 " "\xE8 \xE9 \xEA \xEB \xEC \xED \xEE \xEF ", - /* bug: not rejected */ - "\xE0 \xE1 \xE2 \xE3 \xE4 \xE5 \xE6 \xE7 " - "\xE8 \xE9 \xEA \xEB \xEC \xED \xEE \xEF ", + NULL, "\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFF= FD " "\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFF= FD ", }, /* 3.2.3 All 8 first bytes of 4-byte sequences, followed by space= */ { "\xF0 \xF1 \xF2 \xF3 \xF4 \xF5 \xF6 \xF7 ", - NULL, /* bug: accepted partly, see FIXME below */ + NULL, "\\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFFFD \\uFF= FD ", }, /* 3.2.4 All 4 first bytes of 5-byte sequences, followed by space= */ @@ -431,13 +408,13 @@ static void utf8_string(void) /* 3.3.2 3-byte sequence with last byte missing (U+0000) */ { "\xE0\x80", - "\xE0\x80", /* bug: not rejected */ + NULL, "\\uFFFD", }, /* 3.3.3 4-byte sequence with last byte missing (U+0000) */ { "\xF0\x80\x80", - "\xF0\x80\x80", /* bug: not rejected */ + NULL, "\\uFFFD", }, /* 3.3.4 5-byte sequence with last byte missing (U+0000) */ @@ -455,13 +432,13 @@ static void utf8_string(void) /* 3.3.6 2-byte sequence with last byte missing (U+07FF) */ { "\xDF", - "\xDF", /* bug: not rejected */ + NULL, "\\uFFFD", }, /* 3.3.7 3-byte sequence with last byte missing (U+FFFF) */ { "\xEF\xBF", - "\xEF\xBF", /* bug: not rejected */ + NULL, "\\uFFFD", }, /* 3.3.8 4-byte sequence with last byte missing (U+1FFFFF) */ @@ -486,7 +463,7 @@ static void utf8_string(void) { "\xC0\xE0\x80\xF0\x80\x80\xF8\x80\x80\x80\xFC\x80\x80\x80\x80" "\xDF\xEF\xBF\xF7\xBF\xBF\xFB\xBF\xBF\xBF\xFD\xBF\xBF\xBF\xBF", - NULL, /* bug: accepted partly, see FIXME below */ + NULL, "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD" "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD", }, @@ -515,12 +492,12 @@ static void utf8_string(void) }, { "\xE0\x80\xAF", - "\xE0\x80\xAF", /* bug: not rejected */ + NULL, "\\uFFFD", }, { "\xF0\x80\x80\xAF", - "\xF0\x80\x80\xAF", /* bug: not rejected */ + NULL, "\\uFFFD", }, { @@ -548,7 +525,7 @@ static void utf8_string(void) { /* \U+07FF */ "\xE0\x9F\xBF", - "\xE0\x9F\xBF", /* bug: not rejected */ + NULL, "\\uFFFD", }, { @@ -559,7 +536,7 @@ static void utf8_string(void) * also 2.2.3 */ "\xF0\x8F\xBF\xBC", - "\xF0\x8F\xBF\xBC", /* bug: not rejected */ + NULL, "\\uFFFD", }, { @@ -584,13 +561,13 @@ static void utf8_string(void) { /* \U+0000 */ "\xE0\x80\x80", - "\xE0\x80\x80", /* bug: not rejected */ + NULL, "\\uFFFD", }, { /* \U+0000 */ "\xF0\x80\x80\x80", - "\xF0\x80\x80\x80", /* bug: not rejected */ + NULL, "\\uFFFD", }, { @@ -610,92 +587,92 @@ static void utf8_string(void) { /* \U+D800 */ "\xED\xA0\x80", - "\xED\xA0\x80", /* bug: not rejected */ + NULL, "\\uFFFD", }, { /* \U+DB7F */ "\xED\xAD\xBF", - "\xED\xAD\xBF", /* bug: not rejected */ + NULL, "\\uFFFD", }, { /* \U+DB80 */ "\xED\xAE\x80", - "\xED\xAE\x80", /* bug: not rejected */ + NULL, "\\uFFFD", }, { /* \U+DBFF */ "\xED\xAF\xBF", - "\xED\xAF\xBF", /* bug: not rejected */ + NULL, "\\uFFFD", }, { /* \U+DC00 */ "\xED\xB0\x80", - "\xED\xB0\x80", /* bug: not rejected */ + NULL, "\\uFFFD", }, { /* \U+DF80 */ "\xED\xBE\x80", - "\xED\xBE\x80", /* bug: not rejected */ + NULL, "\\uFFFD", }, { /* \U+DFFF */ "\xED\xBF\xBF", - "\xED\xBF\xBF", /* bug: not rejected */ + NULL, "\\uFFFD", }, /* 5.2 Paired UTF-16 surrogates */ { /* \U+D800\U+DC00 */ "\xED\xA0\x80\xED\xB0\x80", - "\xED\xA0\x80\xED\xB0\x80", /* bug: not rejected */ + NULL, "\\uFFFD\\uFFFD", }, { /* \U+D800\U+DFFF */ "\xED\xA0\x80\xED\xBF\xBF", - "\xED\xA0\x80\xED\xBF\xBF", /* bug: not rejected */ + NULL, "\\uFFFD\\uFFFD", }, { /* \U+DB7F\U+DC00 */ "\xED\xAD\xBF\xED\xB0\x80", - "\xED\xAD\xBF\xED\xB0\x80", /* bug: not rejected */ + NULL, "\\uFFFD\\uFFFD", }, { /* \U+DB7F\U+DFFF */ "\xED\xAD\xBF\xED\xBF\xBF", - "\xED\xAD\xBF\xED\xBF\xBF", /* bug: not rejected */ + NULL, "\\uFFFD\\uFFFD", }, { /* \U+DB80\U+DC00 */ "\xED\xAE\x80\xED\xB0\x80", - "\xED\xAE\x80\xED\xB0\x80", /* bug: not rejected */ + NULL, "\\uFFFD\\uFFFD", }, { /* \U+DB80\U+DFFF */ "\xED\xAE\x80\xED\xBF\xBF", - "\xED\xAE\x80\xED\xBF\xBF", /* bug: not rejected */ + NULL, "\\uFFFD\\uFFFD", }, { /* \U+DBFF\U+DC00 */ "\xED\xAF\xBF\xED\xB0\x80", - "\xED\xAF\xBF\xED\xB0\x80", /* bug: not rejected */ + NULL, "\\uFFFD\\uFFFD", }, { /* \U+DBFF\U+DFFF */ "\xED\xAF\xBF\xED\xBF\xBF", - "\xED\xAF\xBF\xED\xBF\xBF", /* bug: not rejected */ + NULL, "\\uFFFD\\uFFFD", }, /* 5.3 Other illegal code positions */ @@ -703,25 +680,25 @@ static void utf8_string(void) { /* \U+FFFE */ "\xEF\xBF\xBE", - "\xEF\xBF\xBE", /* bug: not rejected */ + NULL, "\\uFFFD", }, { /* \U+FFFF */ "\xEF\xBF\xBF", - "\xEF\xBF\xBF", /* bug: not rejected */ + NULL, "\\uFFFD", }, { /* U+FDD0 */ "\xEF\xB7\x90", - "\xEF\xB7\x90", /* bug: not rejected */ + NULL, "\\uFFFD", }, { /* U+FDEF */ "\xEF\xB7\xAF", - "\xEF\xB7\xAF", /* bug: not rejected */ + NULL, "\\uFFFD", }, /* Plane 1 .. 16 noncharacters */ @@ -743,23 +720,7 @@ static void utf8_string(void) "\xF3\xAF\xBF\xBE\xF3\xAF\xBF\xBF" "\xF3\xBF\xBF\xBE\xF3\xBF\xBF\xBF" "\xF4\x8F\xBF\xBE\xF4\x8F\xBF\xBF", - /* bug: not rejected */ - "\xF0\x9F\xBF\xBE\xF0\x9F\xBF\xBF" - "\xF0\xAF\xBF\xBE\xF0\xAF\xBF\xBF" - "\xF0\xBF\xBF\xBE\xF0\xBF\xBF\xBF" - "\xF1\x8F\xBF\xBE\xF1\x8F\xBF\xBF" - "\xF1\x9F\xBF\xBE\xF1\x9F\xBF\xBF" - "\xF1\xAF\xBF\xBE\xF1\xAF\xBF\xBF" - "\xF1\xBF\xBF\xBE\xF1\xBF\xBF\xBF" - "\xF2\x8F\xBF\xBE\xF2\x8F\xBF\xBF" - "\xF2\x9F\xBF\xBE\xF2\x9F\xBF\xBF" - "\xF2\xAF\xBF\xBE\xF2\xAF\xBF\xBF" - "\xF2\xBF\xBF\xBE\xF2\xBF\xBF\xBF" - "\xF3\x8F\xBF\xBE\xF3\x8F\xBF\xBF" - "\xF3\x9F\xBF\xBE\xF3\x9F\xBF\xBF" - "\xF3\xAF\xBF\xBE\xF3\xAF\xBF\xBF" - "\xF3\xBF\xBF\xBE\xF3\xBF\xBF\xBF" - "\xF4\x8F\xBF\xBE\xF4\x8F\xBF\xBF", + NULL, "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD" "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD" "\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD\\uFFFD" @@ -798,14 +759,6 @@ static void utf8_string(void) } in =3D strndup(tail, end - tail); str =3D from_json_str(in, NULL, j); - /* - * FIXME JSON parser accepts invalid sequence - * starting with \xC2..\xF4 - */ - if (*in >=3D '\xC2' && *in <=3D '\xF4') { - g_free(str); - str =3D NULL; - } g_assert(!str); g_free(in); } diff --git a/util/unicode.c b/util/unicode.c index a812a35171..8580bc598b 100644 --- a/util/unicode.c +++ b/util/unicode.c @@ -13,6 +13,21 @@ #include "qemu/osdep.h" #include "qemu/unicode.h" =20 +static bool is_valid_codepoint(int codepoint) +{ + if (codepoint > 0x10FFFFu) { + return false; /* beyond Unicode range */ + } + if ((codepoint >=3D 0xFDD0 && codepoint <=3D 0xFDEF) + || (codepoint & 0xFFFE) =3D=3D 0xFFFE) { + return false; /* noncharacter */ + } + if (codepoint >=3D 0xD800 && codepoint <=3D 0xDFFF) { + return false; /* surrogate code point */ + } + return true; +} + /** * mod_utf8_codepoint: * @s: string encoded in modified UTF-8 @@ -83,13 +98,8 @@ int mod_utf8_codepoint(const char *s, size_t n, char **e= nd) cp <<=3D 6; cp |=3D byte & 0x3F; } - if (cp > 0x10FFFF) { - cp =3D -1; /* beyond Unicode range */ - } else if ((cp >=3D 0xFDD0 && cp <=3D 0xFDEF) - || (cp & 0xFFFE) =3D=3D 0xFFFE) { - cp =3D -1; /* noncharacter */ - } else if (cp >=3D 0xD800 && cp <=3D 0xDFFF) { - cp =3D -1; /* surrogate code point */ + if (!is_valid_codepoint(cp)) { + cp =3D -1; } else if (cp < min_cp[len - 2] && !(cp =3D=3D 0 && len =3D=3D 2))= { cp =3D -1; /* overlong, not \xC0\x80 */ } @@ -99,3 +109,48 @@ out: *end =3D (char *)p; return cp; } + +/** + * mod_utf8_encode: + * @buf: Destination buffer + * @bufsz: size of @buf, at least 5. + * @codepoint: Unicode codepoint to encode + * + * Convert Unicode codepoint @codepoint to modified UTF-8. + * + * Returns: the length of the UTF-8 sequence on success, -1 when + * @codepoint is invalid. + */ +ssize_t mod_utf8_encode(char buf[], size_t bufsz, int codepoint) +{ + assert(bufsz >=3D 5); + + if (!is_valid_codepoint(codepoint)) { + return -1; + } + + if (codepoint > 0 && codepoint <=3D 0x7F) { + buf[0] =3D codepoint & 0x7F; + buf[1] =3D 0; + return 1; + } + if (codepoint <=3D 0x7FF) { + buf[0] =3D 0xC0 | ((codepoint >> 6) & 0x1F); + buf[1] =3D 0x80 | (codepoint & 0x3F); + buf[2] =3D 0; + return 2; + } + if (codepoint <=3D 0xFFFF) { + buf[0] =3D 0xE0 | ((codepoint >> 12) & 0x0F); + buf[1] =3D 0x80 | ((codepoint >> 6) & 0x3F); + buf[2] =3D 0x80 | (codepoint & 0x3F); + buf[3] =3D 0; + return 3; + } + buf[0] =3D 0xF0 | ((codepoint >> 18) & 0x07); + buf[1] =3D 0x80 | ((codepoint >> 12) & 0x3F); + buf[2] =3D 0x80 | ((codepoint >> 6) & 0x3F); + buf[3] =3D 0x80 | (codepoint & 0x3F); + buf[4] =3D 0; + return 4; +} --=20 2.17.1 From nobody Wed Nov 5 07:53:13 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533730498444495.9461518805283; Wed, 8 Aug 2018 05:14:58 -0700 (PDT) Received: from localhost ([::1]:43212 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNMf-0007o2-6Q for importer@patchew.org; Wed, 08 Aug 2018 08:14:57 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52453) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBn-0006ga-4n for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:51 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBj-0008TZ-A3 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:43 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:51068 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBj-0008Rk-28 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:39 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 7CB004023461; Wed, 8 Aug 2018 12:03:38 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 5331320180F5; Wed, 8 Aug 2018 12:03:38 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 6A23A11386C1; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:00 +0200 Message-Id: <20180808120334.10970-23-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 22/56] json: Report first rather than last parse error X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Quiz time! When a parser reports multiple errors, but the user gets to see just one, which one is (on average) the least useful one? Yes, you're right, it's the last one! You're clearly familiar with compilers. Which one does QEMU report? Right again, the last one! You're clearly familiar with QEMU. Reproducer: feeding {"abc\xC2ijk": 1}\n to QMP produces {"error": {"class": "GenericError", "desc": "JSON parse error, key is n= ot a string in object"}} Report the first error instead. The reproducer now produces {"error": {"class": "GenericError", "desc": "JSON parse error, invalid = UTF-8 sequence in string"}} Signed-off-by: Markus Armbruster Reviewed-by: Eric Blake --- qobject/json-parser.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/qobject/json-parser.c b/qobject/json-parser.c index 6b60b07e09..b3a95be3c8 100644 --- a/qobject/json-parser.c +++ b/qobject/json-parser.c @@ -54,13 +54,13 @@ static void GCC_FMT_ATTR(3, 4) parse_error(JSONParserCo= ntext *ctxt, { va_list ap; char message[1024]; + + if (ctxt->err) { + return; + } va_start(ap, msg); vsnprintf(message, sizeof(message), msg, ap); va_end(ap); - if (ctxt->err) { - error_free(ctxt->err); - ctxt->err =3D NULL; - } error_setg(&ctxt->err, "JSON parse error, %s", message); } =20 --=20 2.17.1 From nobody Wed Nov 5 07:53:13 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533730422423391.0177127376967; Wed, 8 Aug 2018 05:13:42 -0700 (PDT) Received: from localhost ([::1]:43206 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNLR-0006kb-42 for importer@patchew.org; Wed, 08 Aug 2018 08:13:41 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52464) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBn-0006gn-Ad for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:52 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBj-0008Tp-Ci for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:43 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:51070 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBj-0008S9-2i for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:39 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 97CC3402346D; Wed, 8 Aug 2018 12:03:38 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 689C821568A5; Wed, 8 Aug 2018 12:03:38 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 6F2CE11386C2; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:01 +0200 Message-Id: <20180808120334.10970-24-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 23/56] json: Leave rejecting invalid UTF-8 to parser X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Both the lexer and the parser (attempt to) validate UTF-8 in JSON strings. The lexer rejects bytes that can't occur in valid UTF-8: \xC0..\xC1, \xF5..\xFF. This rejects some, but not all invalid UTF-8. It also rejects ASCII control characters \x00..\x1F, in accordance with RFC 7159 (see recent commit "json: Reject unescaped control characters"). When the lexer rejects, it ends the token right after the first bad byte. Good when the bad byte is a newline. Not so good when it's something like an overlong sequence in the middle of a string. For instance, input {"abc\xC0\xAFijk": 1}\n produces the tokens JSON_LCURLY { JSON_ERROR "abc\xC0 JSON_ERROR \xAF JSON_KEYWORD ijk JSON_ERROR ": 1}\n The parser then reports four errors Invalid JSON syntax Invalid JSON syntax JSON parse error, invalid keyword 'ijk' Invalid JSON syntax before it recovers at the newline. The commit before previous made the parser reject invalid UTF-8 sequences. Since then, anything the lexer rejects, the parser would reject as well. Thus, the lexer's rejecting is unnecessary for correctness, and harmful for error reporting. However, we want to keep rejecting ASCII control characters in the lexer, because that produces the behavior we want for unclosed strings. We also need to keep rejecting \xFF in the lexer, because we documented that as a way to reset the JSON parser (docs/interop/qmp-spec.txt section 2.6 QGA Synchronization), which means we can't change how we recover from this error now. I wish we hadn't done that. I think we should treat \xFE the same as \xFF. Change the lexer to accept \xC0..\xC1 and \xF5..\xFD. It now rejects only \x00..\x1F and \xFE..\xFF. Error reporting for invalid UTF-8 in strings is much improved, except for \xFE and \xFF. For the example above, the lexer now produces JSON_LCURLY { JSON_STRING "abc\xC0\xAFijk" JSON_COLON : JSON_INTEGER 1 JSON_RCURLY and the parser reports just JSON parse error, invalid UTF-8 sequence in string Signed-off-by: Markus Armbruster Reviewed-by: Eric Blake --- qobject/json-lexer.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/qobject/json-lexer.c b/qobject/json-lexer.c index 109a7d8bb8..ca1e0e2c03 100644 --- a/qobject/json-lexer.c +++ b/qobject/json-lexer.c @@ -177,8 +177,7 @@ static const uint8_t json_lexer[][256] =3D { ['u'] =3D IN_DQ_UCODE0, }, [IN_DQ_STRING] =3D { - [0x20 ... 0xBF] =3D IN_DQ_STRING, - [0xC2 ... 0xF4] =3D IN_DQ_STRING, + [0x20 ... 0xFD] =3D IN_DQ_STRING, ['\\'] =3D IN_DQ_STRING_ESCAPE, ['"'] =3D JSON_STRING, }, @@ -217,8 +216,7 @@ static const uint8_t json_lexer[][256] =3D { ['u'] =3D IN_SQ_UCODE0, }, [IN_SQ_STRING] =3D { - [0x20 ... 0xBF] =3D IN_SQ_STRING, - [0xC2 ... 0xF4] =3D IN_SQ_STRING, + [0x20 ... 0xFD] =3D IN_SQ_STRING, ['\\'] =3D IN_SQ_STRING_ESCAPE, ['\''] =3D JSON_STRING, }, --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533730685166679.8979964558371; Wed, 8 Aug 2018 05:18:05 -0700 (PDT) Received: from localhost ([::1]:43236 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNPg-00027f-48 for importer@patchew.org; Wed, 08 Aug 2018 08:18:04 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52457) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBn-0006gd-6S for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:51 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBj-0008Tx-Cj for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:43 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:47708 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBj-0008S8-64 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:39 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 99B7079D41; Wed, 8 Aug 2018 12:03:38 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 6DC8220180F8; Wed, 8 Aug 2018 12:03:38 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 7436011386C3; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:02 +0200 Message-Id: <20180808120334.10970-25-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 24/56] json: Accept overlong \xC0\x80 as U+0000 ("modified UTF-8") X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This is consistent with qobject_to_json(). See commit e2ec3f97680. Signed-off-by: Markus Armbruster --- qobject/json-lexer.c | 2 +- qobject/json-parser.c | 2 +- tests/check-qjson.c | 8 +------- 3 files changed, 3 insertions(+), 9 deletions(-) diff --git a/qobject/json-lexer.c b/qobject/json-lexer.c index ca1e0e2c03..36fb665b12 100644 --- a/qobject/json-lexer.c +++ b/qobject/json-lexer.c @@ -93,7 +93,7 @@ * interpolation =3D %((l|ll|I64)[du]|[ipsf]) * * Note: - * - Input must be encoded in UTF-8. + * - Input must be encoded in modified UTF-8. * - Decoding and validating is left to the parser. */ =20 diff --git a/qobject/json-parser.c b/qobject/json-parser.c index b3a95be3c8..14225c3c09 100644 --- a/qobject/json-parser.c +++ b/qobject/json-parser.c @@ -199,7 +199,7 @@ static QString *qstring_from_escaped_str(JSONParserCont= ext *ctxt, } } else { cp =3D mod_utf8_codepoint(ptr, 6, &end); - if (cp <=3D 0) { + if (cp < 0) { parse_error(ctxt, token, "invalid UTF-8 sequence in string= "); goto out; } diff --git a/tests/check-qjson.c b/tests/check-qjson.c index e00be5b023..e5a7cb6bf6 100644 --- a/tests/check-qjson.c +++ b/tests/check-qjson.c @@ -121,12 +121,6 @@ static void escaped_string(void) static void utf8_string(void) { /* - * Problem: we can't easily deal with embedded U+0000. Parsing - * the JSON string "this \\u0000" is fun" yields "this \0 is fun", - * which gets misinterpreted as NUL-terminated "this ". We should - * consider using overlong encoding \xC0\x80 for U+0000 ("modified - * UTF-8"). - * * Most test cases are scraped from Markus Kuhn's UTF-8 decoder * capability and stress test at * http://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt @@ -555,7 +549,7 @@ static void utf8_string(void) { /* \U+0000 */ "\xC0\x80", - NULL, + "\xC0\x80", "\\u0000", }, { --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533732045690999.602039676095; Wed, 8 Aug 2018 05:40:45 -0700 (PDT) Received: from localhost ([::1]:43385 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNlc-00052s-EK for importer@patchew.org; Wed, 08 Aug 2018 08:40:44 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52503) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBo-0006hw-D9 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBj-0008Ub-Jf for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:44 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:45388 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBj-0008Sf-9c for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:39 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id B768681663FD; Wed, 8 Aug 2018 12:03:38 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 706131C72F; Wed, 8 Aug 2018 12:03:38 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 797AE11386C8; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:03 +0200 Message-Id: <20180808120334.10970-26-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) for IP:'10.11.54.5' DOMAIN:'int-mx05.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 25/56] json: Leave rejecting invalid escape sequences to parser X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Both lexer and parser reject invalid escape sequences in strings. The parser's check is useless. The lexer ends the token right after the first non-well-formed byte. This tends to lead to suboptimal error reporting. For instance, input {"abc\@ijk": 1} produces the tokens JSON_LCURLY { JSON_ERROR "abc\@ JSON_KEYWORD ijk JSON_ERROR ": 1}\n The parser then reports three errors Invalid JSON syntax JSON parse error, invalid keyword 'ijk' Invalid JSON syntax before it recovers at the newline. Drop the lexer's escape sequence checking, and make it accept the same characters after '\' it accepts elsewhere in strings. It now produces JSON_LCURLY { JSON_STRING "abc\@ijk" JSON_COLON : JSON_INTEGER 1 JSON_RCURLY and the parser reports just JSON parse error, invalid escape sequence in string While there, fix parse_string()'s inaccurate function comment. Signed-off-by: Markus Armbruster --- qobject/json-lexer.c | 72 +++---------------------------------------- qobject/json-parser.c | 56 +++++++++++++++++++-------------- 2 files changed, 37 insertions(+), 91 deletions(-) diff --git a/qobject/json-lexer.c b/qobject/json-lexer.c index 36fb665b12..af0a7fdb8a 100644 --- a/qobject/json-lexer.c +++ b/qobject/json-lexer.c @@ -80,6 +80,8 @@ * escape =3D %x5C ; \ * quotation-mark =3D %x22 ; " * unescaped =3D %x20-21 / %x23-5B / %x5D-10FFFF + * [This lexer accepts any non-control character after escape, and + * leaves rejecting invalid ones to the parser.] * * * Extensions over RFC 7159: @@ -99,16 +101,8 @@ =20 enum json_lexer_state { IN_ERROR =3D 0, /* must really be 0, see json_lexer[] */ - IN_DQ_UCODE3, - IN_DQ_UCODE2, - IN_DQ_UCODE1, - IN_DQ_UCODE0, IN_DQ_STRING_ESCAPE, IN_DQ_STRING, - IN_SQ_UCODE3, - IN_SQ_UCODE2, - IN_SQ_UCODE1, - IN_SQ_UCODE0, IN_SQ_STRING_ESCAPE, IN_SQ_STRING, IN_ZERO, @@ -144,37 +138,8 @@ static const uint8_t json_lexer[][256] =3D { /* Relies on default initialization to IN_ERROR! */ =20 /* double quote string */ - [IN_DQ_UCODE3] =3D { - ['0' ... '9'] =3D IN_DQ_STRING, - ['a' ... 'f'] =3D IN_DQ_STRING, - ['A' ... 'F'] =3D IN_DQ_STRING, - }, - [IN_DQ_UCODE2] =3D { - ['0' ... '9'] =3D IN_DQ_UCODE3, - ['a' ... 'f'] =3D IN_DQ_UCODE3, - ['A' ... 'F'] =3D IN_DQ_UCODE3, - }, - [IN_DQ_UCODE1] =3D { - ['0' ... '9'] =3D IN_DQ_UCODE2, - ['a' ... 'f'] =3D IN_DQ_UCODE2, - ['A' ... 'F'] =3D IN_DQ_UCODE2, - }, - [IN_DQ_UCODE0] =3D { - ['0' ... '9'] =3D IN_DQ_UCODE1, - ['a' ... 'f'] =3D IN_DQ_UCODE1, - ['A' ... 'F'] =3D IN_DQ_UCODE1, - }, [IN_DQ_STRING_ESCAPE] =3D { - ['b'] =3D IN_DQ_STRING, - ['f'] =3D IN_DQ_STRING, - ['n'] =3D IN_DQ_STRING, - ['r'] =3D IN_DQ_STRING, - ['t'] =3D IN_DQ_STRING, - ['/'] =3D IN_DQ_STRING, - ['\\'] =3D IN_DQ_STRING, - ['\''] =3D IN_DQ_STRING, - ['\"'] =3D IN_DQ_STRING, - ['u'] =3D IN_DQ_UCODE0, + [0x20 ... 0xFD] =3D IN_DQ_STRING, }, [IN_DQ_STRING] =3D { [0x20 ... 0xFD] =3D IN_DQ_STRING, @@ -183,37 +148,8 @@ static const uint8_t json_lexer[][256] =3D { }, =20 /* single quote string */ - [IN_SQ_UCODE3] =3D { - ['0' ... '9'] =3D IN_SQ_STRING, - ['a' ... 'f'] =3D IN_SQ_STRING, - ['A' ... 'F'] =3D IN_SQ_STRING, - }, - [IN_SQ_UCODE2] =3D { - ['0' ... '9'] =3D IN_SQ_UCODE3, - ['a' ... 'f'] =3D IN_SQ_UCODE3, - ['A' ... 'F'] =3D IN_SQ_UCODE3, - }, - [IN_SQ_UCODE1] =3D { - ['0' ... '9'] =3D IN_SQ_UCODE2, - ['a' ... 'f'] =3D IN_SQ_UCODE2, - ['A' ... 'F'] =3D IN_SQ_UCODE2, - }, - [IN_SQ_UCODE0] =3D { - ['0' ... '9'] =3D IN_SQ_UCODE1, - ['a' ... 'f'] =3D IN_SQ_UCODE1, - ['A' ... 'F'] =3D IN_SQ_UCODE1, - }, [IN_SQ_STRING_ESCAPE] =3D { - ['b'] =3D IN_SQ_STRING, - ['f'] =3D IN_SQ_STRING, - ['n'] =3D IN_SQ_STRING, - ['r'] =3D IN_SQ_STRING, - ['t'] =3D IN_SQ_STRING, - ['/'] =3D IN_SQ_STRING, - ['\\'] =3D IN_SQ_STRING, - ['\''] =3D IN_SQ_STRING, - ['\"'] =3D IN_SQ_STRING, - ['u'] =3D IN_SQ_UCODE0, + [0x20 ... 0xFD] =3D IN_SQ_STRING, }, [IN_SQ_STRING] =3D { [0x20 ... 0xFD] =3D IN_SQ_STRING, diff --git a/qobject/json-parser.c b/qobject/json-parser.c index 14225c3c09..d469004616 100644 --- a/qobject/json-parser.c +++ b/qobject/json-parser.c @@ -106,30 +106,40 @@ static int hex2decimal(char ch) } =20 /** - * parse_string(): Parse a json string and return a QObject + * parse_string(): Parse a JSON string * - * string - * "" - * " chars " - * chars - * char - * char chars - * char - * any-Unicode-character- - * except-"-or-\-or- - * control-character - * \" - * \\ - * \/ - * \b - * \f - * \n - * \r - * \t - * \u four-hex-digits=20 + * From RFC 7159 "The JavaScript Object Notation (JSON) Data + * Interchange Format": + * + * char =3D unescaped / + * escape ( + * %x22 / ; " quotation mark U+0022 + * %x5C / ; \ reverse solidus U+005C + * %x2F / ; / solidus U+002F + * %x62 / ; b backspace U+0008 + * %x66 / ; f form feed U+000C + * %x6E / ; n line feed U+000A + * %x72 / ; r carriage return U+000D + * %x74 / ; t tab U+0009 + * %x75 4HEXDIG ) ; uXXXX U+XXXX + * escape =3D %x5C ; \ + * quotation-mark =3D %x22 ; " + * unescaped =3D %x20-21 / %x23-5B / %x5D-10FFFF + * + * Extensions over RFC 7159: + * - Extra escape sequence in strings: + * 0x27 (apostrophe) is recognized after escape, too + * - Single-quoted strings: + * Like double-quoted strings, except they're delimited by %x27 + * (apostrophe) instead of %x22 (quotation mark), and can't contain + * unescaped apostrophe, but can contain unescaped quotation mark. + * + * Note: + * - Encoding is modified UTF-8. + * - Invalid Unicode characters are rejected. + * - Control characters are rejected by the lexer. */ -static QString *qstring_from_escaped_str(JSONParserContext *ctxt, - JSONToken *token) +static QString *parse_string(JSONParserContext *ctxt, JSONToken *token) { const char *ptr =3D token->str; QString *str; @@ -494,7 +504,7 @@ static QObject *parse_literal(JSONParserContext *ctxt) =20 switch (token->type) { case JSON_STRING: - return QOBJECT(qstring_from_escaped_str(ctxt, token)); + return QOBJECT(parse_string(ctxt, token)); case JSON_INTEGER: { /* * Represent JSON_INTEGER as QNUM_I64 if possible, else as --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533731549431309.6126445821185; Wed, 8 Aug 2018 05:32:29 -0700 (PDT) Received: from localhost ([::1]:43340 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNdc-0006SH-BK for importer@patchew.org; Wed, 08 Aug 2018 08:32:28 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52611) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBq-0006ld-Lp for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBj-0008UL-I5 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:46 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:51076 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBj-0008SP-8e for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:39 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id A41B8402347F; Wed, 8 Aug 2018 12:03:38 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 774AF1018CA8; Wed, 8 Aug 2018 12:03:38 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 8336011386C9; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:04 +0200 Message-Id: <20180808120334.10970-27-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:38 +0000 (UTC) for IP:'10.11.54.3' DOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 26/56] json: Simplify parse_string() X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Signed-off-by: Markus Armbruster --- qobject/json-parser.c | 42 +++++++++++++++++++----------------------- 1 file changed, 19 insertions(+), 23 deletions(-) diff --git a/qobject/json-parser.c b/qobject/json-parser.c index d469004616..f26e5b7511 100644 --- a/qobject/json-parser.c +++ b/qobject/json-parser.c @@ -101,8 +101,7 @@ static int hex2decimal(char ch) } else if (ch >=3D 'A' && ch <=3D 'F') { return 10 + (ch - 'A'); } - - return -1; + abort(); } =20 /** @@ -144,7 +143,7 @@ static QString *parse_string(JSONParserContext *ctxt, J= SONToken *token) const char *ptr =3D token->str; QString *str; char quote; - int cp; + int cp, i; char *end; ssize_t len; char utf8_buf[5]; @@ -158,51 +157,48 @@ static QString *parse_string(JSONParserContext *ctxt,= JSONToken *token) if (*ptr =3D=3D '\\') { switch (*++ptr) { case '"': - qstring_append(str, "\""); + qstring_append_chr(str, '"'); break; case '\'': - qstring_append(str, "'"); + qstring_append_chr(str, '\''); break; case '\\': - qstring_append(str, "\\"); + qstring_append_chr(str, '\\'); break; case '/': - qstring_append(str, "/"); + qstring_append_chr(str, '/'); break; case 'b': - qstring_append(str, "\b"); + qstring_append_chr(str, '\b'); break; case 'f': - qstring_append(str, "\f"); + qstring_append_chr(str, '\f'); break; case 'n': - qstring_append(str, "\n"); + qstring_append_chr(str, '\n'); break; case 'r': - qstring_append(str, "\r"); + qstring_append_chr(str, '\r'); break; case 't': - qstring_append(str, "\t"); + qstring_append_chr(str, '\t'); break; - case 'u': { - uint16_t unicode_char =3D 0; - char utf8_char[4]; - int i =3D 0; - + case 'u': + cp =3D 0; for (i =3D 0; i < 4; i++) { ptr++; - if (qemu_isxdigit(*ptr)) { - unicode_char |=3D hex2decimal(*ptr) << ((3 - i) * = 4); - } else { + if (!qemu_isxdigit(*ptr)) { parse_error(ctxt, token, "invalid hex escape sequence in string= "); goto out; } + cp <<=3D 4; + cp |=3D hex2decimal(*ptr); } =20 - wchar_to_utf8(unicode_char, utf8_char, sizeof(utf8_char)); - qstring_append(str, utf8_char); - } break; + wchar_to_utf8(cp, utf8_buf, sizeof(utf8_buf)); + qstring_append(str, utf8_buf); + break; default: parse_error(ctxt, token, "invalid escape sequence in strin= g"); goto out; --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533731153412264.6978808790857; Wed, 8 Aug 2018 05:25:53 -0700 (PDT) Received: from localhost ([::1]:43308 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNXE-0000dc-2v for importer@patchew.org; Wed, 08 Aug 2018 08:25:52 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52491) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBn-0006hS-V6 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:52 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBk-0008Vg-87 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:43 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:47718 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBj-0008UJ-Rn for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:40 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 5354D72624; Wed, 8 Aug 2018 12:03:39 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 0BA05104083A; Wed, 8 Aug 2018 12:03:39 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 8A26211386CA; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:05 +0200 Message-Id: <20180808120334.10970-28-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 08 Aug 2018 12:03:39 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 08 Aug 2018 12:03:39 +0000 (UTC) for IP:'10.11.54.3' DOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 27/56] json: Reject invalid \uXXXX, fix \u0000 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The JSON parser translates invalid \uXXXX to garbage instead of rejecting it, and swallows \u0000. Fix by using mod_utf8_encode() instead of flawed wchar_to_utf8(). Valid surrogate pairs are now differently broken: they're rejected instead of translated to garbage. The next commit will fix them. Signed-off-by: Markus Armbruster --- qobject/json-parser.c | 35 ++++++----------------------------- tests/check-qjson.c | 32 +++++++++----------------------- 2 files changed, 15 insertions(+), 52 deletions(-) diff --git a/qobject/json-parser.c b/qobject/json-parser.c index f26e5b7511..bb54886809 100644 --- a/qobject/json-parser.c +++ b/qobject/json-parser.c @@ -64,34 +64,6 @@ static void GCC_FMT_ATTR(3, 4) parse_error(JSONParserCon= text *ctxt, error_setg(&ctxt->err, "JSON parse error, %s", message); } =20 -/** - * String helpers - * - * These helpers are used to unescape strings. - */ -static void wchar_to_utf8(uint16_t wchar, char *buffer, size_t buffer_leng= th) -{ - if (wchar <=3D 0x007F) { - BUG_ON(buffer_length < 2); - - buffer[0] =3D wchar & 0x7F; - buffer[1] =3D 0; - } else if (wchar <=3D 0x07FF) { - BUG_ON(buffer_length < 3); - - buffer[0] =3D 0xC0 | ((wchar >> 6) & 0x1F); - buffer[1] =3D 0x80 | (wchar & 0x3F); - buffer[2] =3D 0; - } else { - BUG_ON(buffer_length < 4); - - buffer[0] =3D 0xE0 | ((wchar >> 12) & 0x0F); - buffer[1] =3D 0x80 | ((wchar >> 6) & 0x3F); - buffer[2] =3D 0x80 | (wchar & 0x3F); - buffer[3] =3D 0; - } -} - static int hex2decimal(char ch) { if (ch >=3D '0' && ch <=3D '9') { @@ -196,7 +168,12 @@ static QString *parse_string(JSONParserContext *ctxt, = JSONToken *token) cp |=3D hex2decimal(*ptr); } =20 - wchar_to_utf8(cp, utf8_buf, sizeof(utf8_buf)); + if (mod_utf8_encode(utf8_buf, sizeof(utf8_buf), cp) < 0) { + parse_error(ctxt, token, + "\\u%.4s is not a valid Unicode character", + ptr - 3); + goto out; + } qstring_append(str, utf8_buf); break; default: diff --git a/tests/check-qjson.c b/tests/check-qjson.c index e5a7cb6bf6..422697459f 100644 --- a/tests/check-qjson.c +++ b/tests/check-qjson.c @@ -62,7 +62,7 @@ static void escaped_string(void) { "triple byte utf-8 \\u20AC", "triple byte utf-8 \xe2\x82\xac" }, { "quadruple byte utf-8 \\uD834\\uDD1E", /* U+1D11E */ /* bug: want \xF0\x9D\x84\x9E */ - "quadruple byte utf-8 \xED\xA0\xB4\xED\xB4\x9E", .skip =3D 1 }, + NULL }, { "\\z", NULL }, { "\\ux", NULL }, { "\\u1x", NULL }, @@ -70,28 +70,14 @@ static void escaped_string(void) { "\\u123x", NULL }, { "\\u12345", "\341\210\2645" }, { "\\u12345", "\341\210\2645" }, - { "\\u0000x", "x", .skip =3D 1}, /* bug: want \xC0\x80x */ - { "unpaired leading surrogate \\uD800\\uD800", - /* bug: not rejected */ - "unpaired leading surrogate \355\240\200\355\240\200", .skip =3D= 1 }, - { "unpaired trailing surrogate \\uDC00\\uDC00", - /* bug: not rejected */ - "unpaired trailing surrogate \355\260\200\355\260\200", .skip = =3D 1}, - { "backward surrogate pair \\uDC00\\uD800", - /* bug: not rejected */ - "backward surrogate pair \355\260\200\355\240\200", .skip =3D 1}, - { "noncharacter U+FDD0 \\uFDD0", - /* bug: not rejected */ - "noncharacter U+FDD0 \xEF\xB7\x90", .skip =3D 1}, - { "noncharacter U+FDEF \\uFDEF", - /* bug: not rejected */ - "noncharacter U+FDEF \xEF\xB7\xAF", .skip =3D 1}, - { "noncharacter U+1FFFE \\uD87F\\uDFFE", - /* bug: not rejected */ - "noncharacter U+1FFFE \xED\xA1\xBF\xED\xBF\xBE", .skip =3D 1}, - { "noncharacter U+10FFFF \\uDC3F\\uDFFF", - /* bug: not rejected */ - "noncharacter U+10FFFF \xED\xB0\xBF\xED\xBF\xBF", .skip =3D 1}, + { "\\u0000x", "\xC0\x80x" }, + { "unpaired leading surrogate \\uD800\\uD800", NULL }, + { "unpaired trailing surrogate \\uDC00\\uDC00", NULL }, + { "backward surrogate pair \\uDC00\\uD800", NULL }, + { "noncharacter U+FDD0 \\uFDD0", NULL }, + { "noncharacter U+FDEF \\uFDEF", NULL }, + { "noncharacter U+1FFFE \\uD87F\\uDFFE", NULL }, + { "noncharacter U+10FFFF \\uDC3F\\uDFFF", NULL }, {} }; int i, j; --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533732210985710.8322677922141; Wed, 8 Aug 2018 05:43:30 -0700 (PDT) Received: from localhost ([::1]:43400 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNoH-0007fP-SR for importer@patchew.org; Wed, 08 Aug 2018 08:43:29 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52644) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBs-0006ny-2H for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBk-0008VN-1K for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:48 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:45392 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBj-0008U1-O0 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:39 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 37E1181663CE; Wed, 8 Aug 2018 12:03:39 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 0E59D2166BA0; Wed, 8 Aug 2018 12:03:39 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 9042A11386CB; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:06 +0200 Message-Id: <20180808120334.10970-29-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 08 Aug 2018 12:03:39 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 08 Aug 2018 12:03:39 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 28/56] json: Fix \uXXXX for surrogate pairs X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The JSON parser treats each half of a surrogate pair as unpaired surrogate. Fix it to recognize surrogate pairs. Signed-off-by: Markus Armbruster --- qobject/json-parser.c | 16 +++++++++++++++- tests/check-qjson.c | 3 +-- 2 files changed, 16 insertions(+), 3 deletions(-) diff --git a/qobject/json-parser.c b/qobject/json-parser.c index bb54886809..703065fa2b 100644 --- a/qobject/json-parser.c +++ b/qobject/json-parser.c @@ -115,7 +115,7 @@ static QString *parse_string(JSONParserContext *ctxt, J= SONToken *token) const char *ptr =3D token->str; QString *str; char quote; - int cp, i; + int cp, i, leading_surrogate; char *end; ssize_t len; char utf8_buf[5]; @@ -156,6 +156,8 @@ static QString *parse_string(JSONParserContext *ctxt, J= SONToken *token) qstring_append_chr(str, '\t'); break; case 'u': + leading_surrogate =3D 0; + hex: cp =3D 0; for (i =3D 0; i < 4; i++) { ptr++; @@ -168,6 +170,18 @@ static QString *parse_string(JSONParserContext *ctxt, = JSONToken *token) cp |=3D hex2decimal(*ptr); } =20 + if (cp >=3D 0xD800 && cp <=3D 0xDBFF && !leading_surrogate + && ptr[1] =3D=3D '\\' && ptr[2] =3D=3D 'u') { + ptr +=3D 2; + leading_surrogate =3D cp; + goto hex; + } + if (cp >=3D 0xDC00 && cp <=3D 0xDFFF && leading_surrogate)= { + cp &=3D 0x3FF; + cp |=3D (leading_surrogate & 0x3FF) << 10; + cp +=3D 0x010000; + } + if (mod_utf8_encode(utf8_buf, sizeof(utf8_buf), cp) < 0) { parse_error(ctxt, token, "\\u%.4s is not a valid Unicode character", diff --git a/tests/check-qjson.c b/tests/check-qjson.c index 422697459f..3d3a3f105f 100644 --- a/tests/check-qjson.c +++ b/tests/check-qjson.c @@ -61,8 +61,7 @@ static void escaped_string(void) { "double byte utf-8 \\u00A2", "double byte utf-8 \xc2\xa2" }, { "triple byte utf-8 \\u20AC", "triple byte utf-8 \xe2\x82\xac" }, { "quadruple byte utf-8 \\uD834\\uDD1E", /* U+1D11E */ - /* bug: want \xF0\x9D\x84\x9E */ - NULL }, + "quadruple byte utf-8 \xF0\x9D\x84\x9E" }, { "\\z", NULL }, { "\\ux", NULL }, { "\\u1x", NULL }, --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533732782533198.9078113499353; Wed, 8 Aug 2018 05:53:02 -0700 (PDT) Received: from localhost ([::1]:43496 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNxV-0007Fv-EK for importer@patchew.org; Wed, 08 Aug 2018 08:53:01 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52745) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBv-0006p4-N6 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:58 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBl-00007Z-GI for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:51 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:47736 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBl-000052-19 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:41 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 486E672622; Wed, 8 Aug 2018 12:03:40 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 1CA3020180F5; Wed, 8 Aug 2018 12:03:40 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 9652411386CC; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:07 +0200 Message-Id: <20180808120334.10970-30-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 08 Aug 2018 12:03:40 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 08 Aug 2018 12:03:40 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 29/56] check-qjson: Fix and enable utf8_string()'s disabled part X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Signed-off-by: Markus Armbruster --- tests/check-qjson.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/tests/check-qjson.c b/tests/check-qjson.c index 3d3a3f105f..c8c0ad95a6 100644 --- a/tests/check-qjson.c +++ b/tests/check-qjson.c @@ -750,15 +750,10 @@ static void utf8_string(void) qobject_unref(str); g_free(jstr); =20 - /* - * Parse @json_out right back - * Disabled, because qobject_from_json() is buggy, and I can't - * be bothered to add the expected incorrect results. - * FIXME Enable once these bugs have been fixed. - */ - if (0 && json_out !=3D json_in) { + /* Parse @json_out right back, unless it has replacements */ + if (!strstr(json_out, "\\uFFFD")) { str =3D from_json_str(json_out, &error_abort, j); - g_assert_cmpstr(qstring_get_try_str(str), =3D=3D, utf8_out= ); + g_assert_cmpstr(qstring_get_try_str(str), =3D=3D, utf8_in); } } } --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533731317407556.6279651285373; Wed, 8 Aug 2018 05:28:37 -0700 (PDT) Received: from localhost ([::1]:43321 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNZo-0002zJ-ST for importer@patchew.org; Wed, 08 Aug 2018 08:28:32 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52541) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBp-0006ix-Bq for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBk-00004y-FN for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:45 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:51086 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBj-0008Us-Uk for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:40 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 6F6D7402347E; Wed, 8 Aug 2018 12:03:39 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 25D6F2166BA8; Wed, 8 Aug 2018 12:03:39 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 9C96611386CD; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:08 +0200 Message-Id: <20180808120334.10970-31-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:39 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:39 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 30/56] json: remove useless return value from lexer/parser X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" From: Marc-Andr=C3=A9 Lureau The lexer always returns 0 when char feeding. Furthermore, none of the caller care about the return value. Signed-off-by: Marc-Andr=C3=A9 Lureau Message-Id: <20180326150916.9602-10-marcandre.lureau@redhat.com> Reviewed-by: Markus Armbruster Reviewed-by: Thomas Huth Signed-off-by: Markus Armbruster --- include/qapi/qmp/json-lexer.h | 4 ++-- include/qapi/qmp/json-streamer.h | 4 ++-- qobject/json-lexer.c | 23 ++++++++--------------- qobject/json-streamer.c | 8 ++++---- 4 files changed, 16 insertions(+), 23 deletions(-) diff --git a/include/qapi/qmp/json-lexer.h b/include/qapi/qmp/json-lexer.h index afee7828cd..66ccf0357c 100644 --- a/include/qapi/qmp/json-lexer.h +++ b/include/qapi/qmp/json-lexer.h @@ -47,9 +47,9 @@ struct JSONLexer =20 void json_lexer_init(JSONLexer *lexer, JSONLexerEmitter func); =20 -int json_lexer_feed(JSONLexer *lexer, const char *buffer, size_t size); +void json_lexer_feed(JSONLexer *lexer, const char *buffer, size_t size); =20 -int json_lexer_flush(JSONLexer *lexer); +void json_lexer_flush(JSONLexer *lexer); =20 void json_lexer_destroy(JSONLexer *lexer); =20 diff --git a/include/qapi/qmp/json-streamer.h b/include/qapi/qmp/json-strea= mer.h index 00d8a23af8..cb808cf27d 100644 --- a/include/qapi/qmp/json-streamer.h +++ b/include/qapi/qmp/json-streamer.h @@ -36,10 +36,10 @@ typedef struct JSONMessageParser void json_message_parser_init(JSONMessageParser *parser, void (*func)(JSONMessageParser *, GQueue *)); =20 -int json_message_parser_feed(JSONMessageParser *parser, +void json_message_parser_feed(JSONMessageParser *parser, const char *buffer, size_t size); =20 -int json_message_parser_flush(JSONMessageParser *parser); +void json_message_parser_flush(JSONMessageParser *parser); =20 void json_message_parser_destroy(JSONMessageParser *parser); =20 diff --git a/qobject/json-lexer.c b/qobject/json-lexer.c index af0a7fdb8a..87cdd41f29 100644 --- a/qobject/json-lexer.c +++ b/qobject/json-lexer.c @@ -286,7 +286,7 @@ void json_lexer_init(JSONLexer *lexer, JSONLexerEmitter= func) lexer->x =3D lexer->y =3D 0; } =20 -static int json_lexer_feed_char(JSONLexer *lexer, char ch, bool flush) +static void json_lexer_feed_char(JSONLexer *lexer, char ch, bool flush) { int char_consumed, new_state; =20 @@ -340,7 +340,7 @@ static int json_lexer_feed_char(JSONLexer *lexer, char = ch, bool flush) g_string_truncate(lexer->token, 0); new_state =3D IN_START; lexer->state =3D new_state; - return 0; + return; default: break; } @@ -355,29 +355,22 @@ static int json_lexer_feed_char(JSONLexer *lexer, cha= r ch, bool flush) g_string_truncate(lexer->token, 0); lexer->state =3D IN_START; } - - return 0; } =20 -int json_lexer_feed(JSONLexer *lexer, const char *buffer, size_t size) +void json_lexer_feed(JSONLexer *lexer, const char *buffer, size_t size) { size_t i; =20 for (i =3D 0; i < size; i++) { - int err; - - err =3D json_lexer_feed_char(lexer, buffer[i], false); - if (err < 0) { - return err; - } + json_lexer_feed_char(lexer, buffer[i], false); } - - return 0; } =20 -int json_lexer_flush(JSONLexer *lexer) +void json_lexer_flush(JSONLexer *lexer) { - return lexer->state =3D=3D IN_START ? 0 : json_lexer_feed_char(lexer, = 0, true); + if (lexer->state !=3D IN_START) { + json_lexer_feed_char(lexer, 0, true); + } } =20 void json_lexer_destroy(JSONLexer *lexer) diff --git a/qobject/json-streamer.c b/qobject/json-streamer.c index c51c2021f9..78dfff2aa0 100644 --- a/qobject/json-streamer.c +++ b/qobject/json-streamer.c @@ -118,15 +118,15 @@ void json_message_parser_init(JSONMessageParser *pars= er, json_lexer_init(&parser->lexer, json_message_process_token); } =20 -int json_message_parser_feed(JSONMessageParser *parser, +void json_message_parser_feed(JSONMessageParser *parser, const char *buffer, size_t size) { - return json_lexer_feed(&parser->lexer, buffer, size); + json_lexer_feed(&parser->lexer, buffer, size); } =20 -int json_message_parser_flush(JSONMessageParser *parser) +void json_message_parser_flush(JSONMessageParser *parser) { - return json_lexer_flush(&parser->lexer); + json_lexer_flush(&parser->lexer); } =20 void json_message_parser_destroy(JSONMessageParser *parser) --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533731732118287.381833215645; Wed, 8 Aug 2018 05:35:32 -0700 (PDT) Received: from localhost ([::1]:43352 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNgZ-0000Xs-31 for importer@patchew.org; Wed, 08 Aug 2018 08:35:31 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52610) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBq-0006lc-Lk for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBk-0008VT-4p for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:46 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:45394 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBj-0008UI-Qv for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:39 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 5397F81663EE; Wed, 8 Aug 2018 12:03:39 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 2A70B111AF00; Wed, 8 Aug 2018 12:03:39 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id A2D0911386D0; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:09 +0200 Message-Id: <20180808120334.10970-32-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 08 Aug 2018 12:03:39 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 08 Aug 2018 12:03:39 +0000 (UTC) for IP:'10.11.54.3' DOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 31/56] json-parser: simplify and avoid JSONParserContext allocation X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" From: Marc-Andr=C3=A9 Lureau parser_context_new/free() are only used from json_parser_parse(). We can fold the code there and avoid an allocation altogether. Signed-off-by: Marc-Andr=C3=A9 Lureau Message-Id: <20180719184111.5129-9-marcandre.lureau@redhat.com> Reviewed-by: Markus Armbruster --- qobject/json-parser.c | 41 +++++++++-------------------------------- 1 file changed, 9 insertions(+), 32 deletions(-) diff --git a/qobject/json-parser.c b/qobject/json-parser.c index 703065fa2b..b14336e653 100644 --- a/qobject/json-parser.c +++ b/qobject/json-parser.c @@ -232,33 +232,6 @@ static JSONToken *parser_context_peek_token(JSONParser= Context *ctxt) return g_queue_peek_head(ctxt->buf); } =20 -static JSONParserContext *parser_context_new(GQueue *tokens) -{ - JSONParserContext *ctxt; - - if (!tokens) { - return NULL; - } - - ctxt =3D g_malloc0(sizeof(JSONParserContext)); - ctxt->buf =3D tokens; - - return ctxt; -} - -/* to support error propagation, ctxt->err must be freed separately */ -static void parser_context_free(JSONParserContext *ctxt) -{ - if (ctxt) { - while (!g_queue_is_empty(ctxt->buf)) { - parser_context_pop_token(ctxt); - } - g_free(ctxt->current); - g_queue_free(ctxt->buf); - g_free(ctxt); - } -} - /** * Parsing rules */ @@ -570,18 +543,22 @@ QObject *json_parser_parse(GQueue *tokens, va_list *a= p) =20 QObject *json_parser_parse_err(GQueue *tokens, va_list *ap, Error **errp) { - JSONParserContext *ctxt =3D parser_context_new(tokens); + JSONParserContext ctxt =3D { .buf =3D tokens }; QObject *result; =20 - if (!ctxt) { + if (!tokens) { return NULL; } =20 - result =3D parse_value(ctxt, ap); + result =3D parse_value(&ctxt, ap); =20 - error_propagate(errp, ctxt->err); + error_propagate(errp, ctxt.err); =20 - parser_context_free(ctxt); + while (!g_queue_is_empty(ctxt.buf)) { + parser_context_pop_token(&ctxt); + } + g_free(ctxt.current); + g_queue_free(ctxt.buf); =20 return result; } --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533732022812301.7442022632056; Wed, 8 Aug 2018 05:40:22 -0700 (PDT) Received: from localhost ([::1]:43382 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNlF-0004fL-HT for importer@patchew.org; Wed, 08 Aug 2018 08:40:21 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52563) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBp-0006jW-Qw for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:56 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBk-00005F-LO for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:45 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:51090 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBk-0008V1-2r for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:40 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 79BC04023820; Wed, 8 Aug 2018 12:03:39 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 2DD782156897; Wed, 8 Aug 2018 12:03:39 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id A8FB911386D1; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:10 +0200 Message-Id: <20180808120334.10970-33-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:39 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:39 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 32/56] json: Have lexer call streamer directly X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" json_lexer_init() takes the function to process a token as an argument. It's always json_message_process_token(). Makes the code harder to understand for no actual gain. Drop the indirection. Signed-off-by: Markus Armbruster Reviewed-by: Eric Blake --- include/qapi/qmp/json-lexer.h | 13 +++---------- include/qapi/qmp/json-streamer.h | 3 +++ qobject/json-lexer.c | 13 ++++++++----- qobject/json-streamer.c | 6 +++--- 4 files changed, 17 insertions(+), 18 deletions(-) diff --git a/include/qapi/qmp/json-lexer.h b/include/qapi/qmp/json-lexer.h index 66ccf0357c..44bcf2ca64 100644 --- a/include/qapi/qmp/json-lexer.h +++ b/include/qapi/qmp/json-lexer.h @@ -32,20 +32,13 @@ typedef enum json_token_type { JSON_ERROR, } JSONTokenType; =20 -typedef struct JSONLexer JSONLexer; - -typedef void (JSONLexerEmitter)(JSONLexer *, GString *, - JSONTokenType, int x, int y); - -struct JSONLexer -{ - JSONLexerEmitter *emit; +typedef struct JSONLexer { int state; GString *token; int x, y; -}; +} JSONLexer; =20 -void json_lexer_init(JSONLexer *lexer, JSONLexerEmitter func); +void json_lexer_init(JSONLexer *lexer); =20 void json_lexer_feed(JSONLexer *lexer, const char *buffer, size_t size); =20 diff --git a/include/qapi/qmp/json-streamer.h b/include/qapi/qmp/json-strea= mer.h index cb808cf27d..7922e185a5 100644 --- a/include/qapi/qmp/json-streamer.h +++ b/include/qapi/qmp/json-streamer.h @@ -33,6 +33,9 @@ typedef struct JSONMessageParser uint64_t token_size; } JSONMessageParser; =20 +void json_message_process_token(JSONLexer *lexer, GString *input, + JSONTokenType type, int x, int y); + void json_message_parser_init(JSONMessageParser *parser, void (*func)(JSONMessageParser *, GQueue *)); =20 diff --git a/qobject/json-lexer.c b/qobject/json-lexer.c index 87cdd41f29..0b54b1af56 100644 --- a/qobject/json-lexer.c +++ b/qobject/json-lexer.c @@ -14,6 +14,7 @@ #include "qemu/osdep.h" #include "qemu-common.h" #include "qapi/qmp/json-lexer.h" +#include "qapi/qmp/json-streamer.h" =20 #define MAX_TOKEN_SIZE (64ULL << 20) =20 @@ -278,9 +279,8 @@ static const uint8_t json_lexer[][256] =3D { }, }; =20 -void json_lexer_init(JSONLexer *lexer, JSONLexerEmitter func) +void json_lexer_init(JSONLexer *lexer) { - lexer->emit =3D func; lexer->state =3D IN_START; lexer->token =3D g_string_sized_new(3); lexer->x =3D lexer->y =3D 0; @@ -316,7 +316,8 @@ static void json_lexer_feed_char(JSONLexer *lexer, char= ch, bool flush) case JSON_FLOAT: case JSON_KEYWORD: case JSON_STRING: - lexer->emit(lexer, lexer->token, new_state, lexer->x, lexer->y= ); + json_message_process_token(lexer, lexer->token, new_state, + lexer->x, lexer->y); /* fall through */ case JSON_SKIP: g_string_truncate(lexer->token, 0); @@ -336,7 +337,8 @@ static void json_lexer_feed_char(JSONLexer *lexer, char= ch, bool flush) * never a valid ASCII/UTF-8 sequence, so this should reliably * induce an error/flush state. */ - lexer->emit(lexer, lexer->token, JSON_ERROR, lexer->x, lexer->= y); + json_message_process_token(lexer, lexer->token, JSON_ERROR, + lexer->x, lexer->y); g_string_truncate(lexer->token, 0); new_state =3D IN_START; lexer->state =3D new_state; @@ -351,7 +353,8 @@ static void json_lexer_feed_char(JSONLexer *lexer, char= ch, bool flush) * this is a security consideration. */ if (lexer->token->len > MAX_TOKEN_SIZE) { - lexer->emit(lexer, lexer->token, lexer->state, lexer->x, lexer->y); + json_message_process_token(lexer, lexer->token, lexer->state, + lexer->x, lexer->y); g_string_truncate(lexer->token, 0); lexer->state =3D IN_START; } diff --git a/qobject/json-streamer.c b/qobject/json-streamer.c index 78dfff2aa0..9f57ebf2bd 100644 --- a/qobject/json-streamer.c +++ b/qobject/json-streamer.c @@ -34,8 +34,8 @@ static void json_message_free_tokens(JSONMessageParser *p= arser) } } =20 -static void json_message_process_token(JSONLexer *lexer, GString *input, - JSONTokenType type, int x, int y) +void json_message_process_token(JSONLexer *lexer, GString *input, + JSONTokenType type, int x, int y) { JSONMessageParser *parser =3D container_of(lexer, JSONMessageParser, l= exer); JSONToken *token; @@ -115,7 +115,7 @@ void json_message_parser_init(JSONMessageParser *parser, parser->tokens =3D g_queue_new(); parser->token_size =3D 0; =20 - json_lexer_init(&parser->lexer, json_message_process_token); + json_lexer_init(&parser->lexer); } =20 void json_message_parser_feed(JSONMessageParser *parser, --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533732383314130.52479325600746; Wed, 8 Aug 2018 05:46:23 -0700 (PDT) Received: from localhost ([::1]:43418 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNr4-0001lc-2G for importer@patchew.org; Wed, 08 Aug 2018 08:46:22 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52603) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBq-0006lO-I7 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:55 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBk-00005T-La for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:46 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:51092 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBk-0008V8-36 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:40 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 93CDB4023826; Wed, 8 Aug 2018 12:03:39 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 2E0921C71D; Wed, 8 Aug 2018 12:03:39 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id AF20111386D2; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:11 +0200 Message-Id: <20180808120334.10970-34-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:39 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:39 +0000 (UTC) for IP:'10.11.54.5' DOMAIN:'int-mx05.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 33/56] json: Redesign the callback to consume JSON values X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The classical way to structure parser and lexer is to have the client call the parser to get an abstract syntax tree, the parser call the lexer to get the next token, and the lexer call some function to get input characters. Another way to structure them would be to have the client feed characters to the lexer, the lexer feed tokens to the parser, and the parser feed abstract syntax trees to some callback provided by the client. This way is more easily integrated into an event loop that dispatches input characters as they arrive. Our JSON parser is kind of between the two. The lexer feeds tokens to a "streamer" instead of a real parser. The streamer accumulates tokens until it got the sequence of tokens that comprise a single JSON value (it counts curly braces and square brackets to decide). It feeds those token sequences to a callback provided by the client. The callback passes each token sequence to the parser, and gets back an abstract syntax tree. I figure it was done that way to make a straightforward recursive descent parser possible. "Get next token" becomes "pop the first token off the token sequence". Drawback: we need to store a complete token sequence. Each token eats 13 + input characters + malloc overhead bytes. Observations: 1. This is not the only way to use recursive descent. If we replaced "get next token" by a coroutine yield, we could do without a streamer. 2. The lexer reports errors by passing a JSON_ERROR token to the streamer. This communicates the offending input characters and their location, but no more. 3. The streamer reports errors by passing a null token sequence to the callback. The (already poor) lexical error information is thrown away. 4. Having the callback receive a token sequence duplicates the code to convert token sequence to abstract syntax tree in every callback. 5. Known bug: the streamer silently drops incomplete token sequences. This commit rectifies 4. by lifting the call of the parser from the callbacks into the streamer. Later commits will address 3. and 5. The lifting removes a bug from qjson.c's parse_json(): it passed a pointer to a non-null Error * in certain cases, as demonstrated by check-qjson.c. json_parser_parse() is now unused. It's a stupid wrapper around json_parser_parse_err(). Drop it, and rename json_parser_parse_err() to json_parser_parse(). Signed-off-by: Markus Armbruster --- include/qapi/qmp/json-parser.h | 3 +-- include/qapi/qmp/json-streamer.h | 8 ++++++-- monitor.c | 18 ++++++++---------- qapi/qmp-dispatch.c | 1 - qga/main.c | 12 +++--------- qobject/json-parser.c | 7 +------ qobject/json-streamer.c | 19 +++++++++++-------- qobject/qjson.c | 14 +++++--------- tests/check-qjson.c | 1 - tests/libqtest.c | 9 +++------ 10 files changed, 38 insertions(+), 54 deletions(-) diff --git a/include/qapi/qmp/json-parser.h b/include/qapi/qmp/json-parser.h index 102f5c0068..a34209db7a 100644 --- a/include/qapi/qmp/json-parser.h +++ b/include/qapi/qmp/json-parser.h @@ -16,7 +16,6 @@ =20 #include "qemu-common.h" =20 -QObject *json_parser_parse(GQueue *tokens, va_list *ap); -QObject *json_parser_parse_err(GQueue *tokens, va_list *ap, Error **errp); +QObject *json_parser_parse(GQueue *tokens, va_list *ap, Error **errp); =20 #endif diff --git a/include/qapi/qmp/json-streamer.h b/include/qapi/qmp/json-strea= mer.h index 7922e185a5..e162fd01da 100644 --- a/include/qapi/qmp/json-streamer.h +++ b/include/qapi/qmp/json-streamer.h @@ -25,7 +25,9 @@ typedef struct JSONToken { =20 typedef struct JSONMessageParser { - void (*emit)(struct JSONMessageParser *parser, GQueue *tokens); + void (*emit)(void *opaque, QObject *json, Error *err); + void *opaque; + va_list *ap; JSONLexer lexer; int brace_count; int bracket_count; @@ -37,7 +39,9 @@ void json_message_process_token(JSONLexer *lexer, GString= *input, JSONTokenType type, int x, int y); =20 void json_message_parser_init(JSONMessageParser *parser, - void (*func)(JSONMessageParser *, GQueue *)); + void (*emit)(void *opaque, QObject *json, + Error *err), + void *opaque, va_list *ap); =20 void json_message_parser_feed(JSONMessageParser *parser, const char *buffer, size_t size); diff --git a/monitor.c b/monitor.c index 77861e96af..71658d9905 100644 --- a/monitor.c +++ b/monitor.c @@ -59,7 +59,6 @@ #include "qapi/qmp/qstring.h" #include "qapi/qmp/qjson.h" #include "qapi/qmp/json-streamer.h" -#include "qapi/qmp/json-parser.h" #include "qapi/qmp/qlist.h" #include "qom/object_interfaces.h" #include "trace-root.h" @@ -4245,18 +4244,15 @@ static void monitor_qmp_bh_dispatcher(void *data) =20 #define QMP_REQ_QUEUE_LEN_MAX (8) =20 -static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens) +static void handle_qmp_command(void *opaque, QObject *req, Error *err) { - QObject *req, *id =3D NULL; + Monitor *mon =3D opaque; + QObject *id =3D NULL; QDict *qdict; - MonitorQMP *mon_qmp =3D container_of(parser, MonitorQMP, parser); - Monitor *mon =3D container_of(mon_qmp, Monitor, qmp); - Error *err =3D NULL; QMPRequest *req_obj; =20 - req =3D json_parser_parse_err(tokens, NULL, &err); if (!req && !err) { - /* json_parser_parse_err() sucks: can fail without setting @err */ + /* json_parser_parse() sucks: can fail without setting @err */ error_setg(&err, QERR_JSON_PARSING); } =20 @@ -4452,7 +4448,8 @@ static void monitor_qmp_event(void *opaque, int event) monitor_qmp_response_flush(mon); monitor_qmp_cleanup_queues(mon); json_message_parser_destroy(&mon->qmp.parser); - json_message_parser_init(&mon->qmp.parser, handle_qmp_command); + json_message_parser_init(&mon->qmp.parser, handle_qmp_command, + mon, NULL); mon_refcount--; monitor_fdsets_cleanup(); break; @@ -4670,7 +4667,8 @@ void monitor_init(Chardev *chr, int flags) =20 if (monitor_is_qmp(mon)) { qemu_chr_fe_set_echo(&mon->chr, true); - json_message_parser_init(&mon->qmp.parser, handle_qmp_command); + json_message_parser_init(&mon->qmp.parser, handle_qmp_command, + mon, NULL); if (mon->use_io_thread) { /* * Make sure the old iowatch is gone. It's possible when diff --git a/qapi/qmp-dispatch.c b/qapi/qmp-dispatch.c index 6f2d466596..d8da1a62de 100644 --- a/qapi/qmp-dispatch.c +++ b/qapi/qmp-dispatch.c @@ -14,7 +14,6 @@ #include "qemu/osdep.h" #include "qapi/error.h" #include "qapi/qmp/dispatch.h" -#include "qapi/qmp/json-parser.h" #include "qapi/qmp/qdict.h" #include "qapi/qmp/qjson.h" #include "qapi/qmp/qbool.h" diff --git a/qga/main.c b/qga/main.c index 87372d40ef..2fc49d00d8 100644 --- a/qga/main.c +++ b/qga/main.c @@ -19,7 +19,6 @@ #include #endif #include "qapi/qmp/json-streamer.h" -#include "qapi/qmp/json-parser.h" #include "qapi/qmp/qdict.h" #include "qapi/qmp/qjson.h" #include "qapi/qmp/qstring.h" @@ -597,18 +596,13 @@ static void process_command(GAState *s, QDict *req) } =20 /* handle requests/control events coming in over the channel */ -static void process_event(JSONMessageParser *parser, GQueue *tokens) +static void process_event(void *opaque, QObject *obj, Error *err) { - GAState *s =3D container_of(parser, GAState, parser); - QObject *obj; + GAState *s =3D opaque; QDict *req, *rsp; - Error *err =3D NULL; int ret; =20 - g_assert(s && parser); - g_debug("process_event: called"); - obj =3D json_parser_parse_err(tokens, NULL, &err); if (err) { goto err; } @@ -1320,7 +1314,7 @@ static int run_agent(GAState *s, GAConfig *config, in= t socket_activation) s->command_state =3D ga_command_state_new(); ga_command_state_init(s, s->command_state); ga_command_state_init_all(s->command_state); - json_message_parser_init(&s->parser, process_event); + json_message_parser_init(&s->parser, process_event, s, NULL); =20 #ifndef _WIN32 if (!register_signal_handlers()) { diff --git a/qobject/json-parser.c b/qobject/json-parser.c index b14336e653..0196d511c3 100644 --- a/qobject/json-parser.c +++ b/qobject/json-parser.c @@ -536,12 +536,7 @@ static QObject *parse_value(JSONParserContext *ctxt, v= a_list *ap) } } =20 -QObject *json_parser_parse(GQueue *tokens, va_list *ap) -{ - return json_parser_parse_err(tokens, ap, NULL); -} - -QObject *json_parser_parse_err(GQueue *tokens, va_list *ap, Error **errp) +QObject *json_parser_parse(GQueue *tokens, va_list *ap, Error **errp) { JSONParserContext ctxt =3D { .buf =3D tokens }; QObject *result; diff --git a/qobject/json-streamer.c b/qobject/json-streamer.c index 9f57ebf2bd..7fd0ff8756 100644 --- a/qobject/json-streamer.c +++ b/qobject/json-streamer.c @@ -14,6 +14,7 @@ #include "qemu/osdep.h" #include "qemu-common.h" #include "qapi/qmp/json-lexer.h" +#include "qapi/qmp/json-parser.h" #include "qapi/qmp/json-streamer.h" =20 #define MAX_TOKEN_SIZE (64ULL << 20) @@ -38,8 +39,9 @@ void json_message_process_token(JSONLexer *lexer, GString= *input, JSONTokenType type, int x, int y) { JSONMessageParser *parser =3D container_of(lexer, JSONMessageParser, l= exer); + Error *err =3D NULL; JSONToken *token; - GQueue *tokens; + QObject *json; =20 switch (type) { case JSON_LCURLY: @@ -97,19 +99,20 @@ out_emit: /* send current list of tokens to parser and reset tokenizer */ parser->brace_count =3D 0; parser->bracket_count =3D 0; - /* parser->emit takes ownership of parser->tokens. Remove our own - * reference to parser->tokens before handing it out to parser->emit. - */ - tokens =3D parser->tokens; + json =3D json_parser_parse(parser->tokens, parser->ap, &err); parser->tokens =3D g_queue_new(); - parser->emit(parser, tokens); parser->token_size =3D 0; + parser->emit(parser->opaque, json, err); } =20 void json_message_parser_init(JSONMessageParser *parser, - void (*func)(JSONMessageParser *, GQueue *)) + void (*emit)(void *opaque, QObject *json, + Error *err), + void *opaque, va_list *ap) { - parser->emit =3D func; + parser->emit =3D emit; + parser->opaque =3D opaque; + parser->ap =3D ap; parser->brace_count =3D 0; parser->bracket_count =3D 0; parser->tokens =3D g_queue_new(); diff --git a/qobject/qjson.c b/qobject/qjson.c index ab4040f235..7395556069 100644 --- a/qobject/qjson.c +++ b/qobject/qjson.c @@ -13,8 +13,6 @@ =20 #include "qemu/osdep.h" #include "qapi/error.h" -#include "qapi/qmp/json-lexer.h" -#include "qapi/qmp/json-parser.h" #include "qapi/qmp/json-streamer.h" #include "qapi/qmp/qjson.h" #include "qapi/qmp/qbool.h" @@ -27,16 +25,16 @@ typedef struct JSONParsingState { JSONMessageParser parser; - va_list *ap; QObject *result; Error *err; } JSONParsingState; =20 -static void parse_json(JSONMessageParser *parser, GQueue *tokens) +static void consume_json(void *opaque, QObject *json, Error *err) { - JSONParsingState *s =3D container_of(parser, JSONParsingState, parser); + JSONParsingState *s =3D opaque; =20 - s->result =3D json_parser_parse_err(tokens, s->ap, &s->err); + s->result =3D json; + error_propagate(&s->err, err); } =20 /* @@ -54,9 +52,7 @@ static QObject *qobject_from_jsonv(const char *string, va= _list *ap, { JSONParsingState state =3D {}; =20 - state.ap =3D ap; - - json_message_parser_init(&state.parser, parse_json); + json_message_parser_init(&state.parser, consume_json, &state, ap); json_message_parser_feed(&state.parser, string, strlen(string)); json_message_parser_flush(&state.parser); json_message_parser_destroy(&state.parser); diff --git a/tests/check-qjson.c b/tests/check-qjson.c index c8c0ad95a6..4c4afcf691 100644 --- a/tests/check-qjson.c +++ b/tests/check-qjson.c @@ -1385,7 +1385,6 @@ static void multiple_values(void) qobject_unref(obj); =20 /* BUG simultaneously succeeds and fails */ - /* BUG calls json_parser_parse() with errp pointing to non-null */ obj =3D qobject_from_json("} true", &err); g_assert(qbool_get_bool(qobject_to(QBool, obj))); error_free_or_abort(&err); diff --git a/tests/libqtest.c b/tests/libqtest.c index 9c844874e4..aa451214d9 100644 --- a/tests/libqtest.c +++ b/tests/libqtest.c @@ -23,7 +23,6 @@ #include "libqtest.h" #include "qemu/cutils.h" #include "qapi/error.h" -#include "qapi/qmp/json-parser.h" #include "qapi/qmp/json-streamer.h" #include "qapi/qmp/qdict.h" #include "qapi/qmp/qjson.h" @@ -428,12 +427,10 @@ typedef struct { QDict *response; } QMPResponseParser; =20 -static void qmp_response(JSONMessageParser *parser, GQueue *tokens) +static void qmp_response(void *opaque, QObject *obj, Error *err) { - QMPResponseParser *qmp =3D container_of(parser, QMPResponseParser, par= ser); - QObject *obj; + QMPResponseParser *qmp =3D opaque; =20 - obj =3D json_parser_parse(tokens, NULL); if (!obj) { fprintf(stderr, "QMP JSON response parsing failed\n"); exit(1); @@ -450,7 +447,7 @@ QDict *qmp_fd_receive(int fd) bool log =3D getenv("QTEST_LOG") !=3D NULL; =20 qmp.response =3D NULL; - json_message_parser_init(&qmp.parser, qmp_response); + json_message_parser_init(&qmp.parser, qmp_response, &qmp, NULL); while (!qmp.response) { ssize_t len; char c; --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533732000336433.9876445341083; Wed, 8 Aug 2018 05:40:00 -0700 (PDT) Received: from localhost ([::1]:43381 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNks-0004Gf-9i for importer@patchew.org; Wed, 08 Aug 2018 08:39:58 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52495) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBo-0006hZ-4W for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:57 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBk-0008Vy-B2 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:44 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:51082 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBj-0008UU-SG for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:40 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 5AE044023476; Wed, 8 Aug 2018 12:03:39 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 317621040840; Wed, 8 Aug 2018 12:03:39 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id B5E6711386D3; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:12 +0200 Message-Id: <20180808120334.10970-35-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:39 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:39 +0000 (UTC) for IP:'10.11.54.3' DOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 34/56] json: Don't pass null @tokens to json_parser_parse() X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" json_parser_parse() normally returns the QObject on success. Except it returns null when its @tokens argument is null. Its only caller json_message_process_token() passes null @tokens when emitting a lexical error. The call is a rather opaque way to say json =3D NULL then. Simplify matters by lifting the assignment to json out of the emit path: initialize json to null, set it to the value of json_parser_parse() when there's no lexical error. Drop the special case from json_parser_parse(). Signed-off-by: Markus Armbruster --- qobject/json-parser.c | 4 ---- qobject/json-streamer.c | 25 ++++++++++++------------- 2 files changed, 12 insertions(+), 17 deletions(-) diff --git a/qobject/json-parser.c b/qobject/json-parser.c index 0196d511c3..0e4ea564ab 100644 --- a/qobject/json-parser.c +++ b/qobject/json-parser.c @@ -541,10 +541,6 @@ QObject *json_parser_parse(GQueue *tokens, va_list *ap= , Error **errp) JSONParserContext ctxt =3D { .buf =3D tokens }; QObject *result; =20 - if (!tokens) { - return NULL; - } - result =3D parse_value(&ctxt, ap); =20 error_propagate(errp, ctxt.err); diff --git a/qobject/json-streamer.c b/qobject/json-streamer.c index 7fd0ff8756..0c33186e8e 100644 --- a/qobject/json-streamer.c +++ b/qobject/json-streamer.c @@ -39,9 +39,9 @@ void json_message_process_token(JSONLexer *lexer, GString= *input, JSONTokenType type, int x, int y) { JSONMessageParser *parser =3D container_of(lexer, JSONMessageParser, l= exer); + QObject *json =3D NULL; Error *err =3D NULL; JSONToken *token; - QObject *json; =20 switch (type) { case JSON_LCURLY: @@ -72,34 +72,33 @@ void json_message_process_token(JSONLexer *lexer, GStri= ng *input, g_queue_push_tail(parser->tokens, token); =20 if (type =3D=3D JSON_ERROR) { - goto out_emit_bad; - } else if (parser->brace_count < 0 || + goto out_emit; + } + + if (parser->brace_count < 0 || parser->bracket_count < 0 || (parser->brace_count =3D=3D 0 && parser->bracket_count =3D=3D 0)) { + json =3D json_parser_parse(parser->tokens, parser->ap, &err); + parser->tokens =3D NULL; goto out_emit; - } else if (parser->token_size > MAX_TOKEN_SIZE || + } + + if (parser->token_size > MAX_TOKEN_SIZE || g_queue_get_length(parser->tokens) > MAX_TOKEN_COUNT || parser->bracket_count + parser->brace_count > MAX_NESTING) { /* Security consideration, we limit total memory allocated per obj= ect * and the maximum recursion depth that a message can force. */ - goto out_emit_bad; + goto out_emit; } =20 return; =20 -out_emit_bad: - /* - * Clear out token list and tell the parser to emit an error - * indication by passing it a NULL list - */ - json_message_free_tokens(parser); out_emit: - /* send current list of tokens to parser and reset tokenizer */ parser->brace_count =3D 0; parser->bracket_count =3D 0; - json =3D json_parser_parse(parser->tokens, parser->ap, &err); + json_message_free_tokens(parser); parser->tokens =3D g_queue_new(); parser->token_size =3D 0; parser->emit(parser->opaque, json, err); --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533730333202610.8545539717346; Wed, 8 Aug 2018 05:12:13 -0700 (PDT) Received: from localhost ([::1]:43167 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNJz-0005N8-UJ for importer@patchew.org; Wed, 08 Aug 2018 08:12:12 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52476) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBn-0006h7-Kp for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:51 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBk-0008W2-Ad for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:43 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:47722 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBj-0008US-T6 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:40 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 5A8EB72625; Wed, 8 Aug 2018 12:03:39 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 3196B20180F5; Wed, 8 Aug 2018 12:03:39 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id BBEDE11386D4; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:13 +0200 Message-Id: <20180808120334.10970-36-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 08 Aug 2018 12:03:39 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 08 Aug 2018 12:03:39 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 35/56] json: Don't create JSON_ERROR tokens that won't be used X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Signed-off-by: Markus Armbruster --- qobject/json-streamer.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/qobject/json-streamer.c b/qobject/json-streamer.c index 0c33186e8e..fa595a8761 100644 --- a/qobject/json-streamer.c +++ b/qobject/json-streamer.c @@ -56,6 +56,8 @@ void json_message_process_token(JSONLexer *lexer, GString= *input, case JSON_RSQUARE: parser->bracket_count--; break; + case JSON_ERROR: + goto out_emit; default: break; } @@ -71,10 +73,6 @@ void json_message_process_token(JSONLexer *lexer, GStrin= g *input, =20 g_queue_push_tail(parser->tokens, token); =20 - if (type =3D=3D JSON_ERROR) { - goto out_emit; - } - if (parser->brace_count < 0 || parser->bracket_count < 0 || (parser->brace_count =3D=3D 0 && --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533731832333158.11988103500516; Wed, 8 Aug 2018 05:37:12 -0700 (PDT) Received: from localhost ([::1]:43368 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNiA-0001uU-LL for importer@patchew.org; Wed, 08 Aug 2018 08:37:10 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52712) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBu-0006ox-UY for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:56 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBl-00008G-RC for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:50 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:43246 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBl-000062-9y for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:41 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 96ED77788C; Wed, 8 Aug 2018 12:03:40 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 461E21C716; Wed, 8 Aug 2018 12:03:40 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id C1FBD11386D5; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:14 +0200 Message-Id: <20180808120334.10970-37-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Wed, 08 Aug 2018 12:03:40 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Wed, 08 Aug 2018 12:03:40 +0000 (UTC) for IP:'10.11.54.5' DOMAIN:'int-mx05.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 36/56] json: Rename token JSON_ESCAPE & friends to JSON_INTERPOL X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The JSON parser optionally supports interpolation. The code calls it "escape". Awkward, because it uses the same term for escape sequences within strings. The latter usage is consistent with RFC 7159 "The JavaScript Object Notation (JSON) Data Interchange Format" and ISO C. Call the former "interpolation" instead. Signed-off-by: Markus Armbruster --- include/qapi/qmp/json-lexer.h | 2 +- qobject/json-lexer.c | 64 +++++++++++++++++------------------ qobject/json-parser.c | 8 ++--- 3 files changed, 37 insertions(+), 37 deletions(-) diff --git a/include/qapi/qmp/json-lexer.h b/include/qapi/qmp/json-lexer.h index 44bcf2ca64..ff3a6f80f0 100644 --- a/include/qapi/qmp/json-lexer.h +++ b/include/qapi/qmp/json-lexer.h @@ -27,7 +27,7 @@ typedef enum json_token_type { JSON_FLOAT, JSON_KEYWORD, JSON_STRING, - JSON_ESCAPE, + JSON_INTERPOL, JSON_SKIP, JSON_ERROR, } JSONTokenType; diff --git a/qobject/json-lexer.c b/qobject/json-lexer.c index 0b54b1af56..5b1f720331 100644 --- a/qobject/json-lexer.c +++ b/qobject/json-lexer.c @@ -115,12 +115,12 @@ enum json_lexer_state { IN_NONZERO_NUMBER, IN_NEG_NONZERO_NUMBER, IN_KEYWORD, - IN_ESCAPE, - IN_ESCAPE_L, - IN_ESCAPE_LL, - IN_ESCAPE_I, - IN_ESCAPE_I6, - IN_ESCAPE_I64, + IN_INTERPOL, + IN_INTERPOL_L, + IN_INTERPOL_LL, + IN_INTERPOL_I, + IN_INTERPOL_I6, + IN_INTERPOL_I64, IN_WHITESPACE, IN_START, }; @@ -221,40 +221,40 @@ static const uint8_t json_lexer[][256] =3D { ['\n'] =3D IN_WHITESPACE, }, =20 - /* escape */ - [IN_ESCAPE_LL] =3D { - ['d'] =3D JSON_ESCAPE, - ['u'] =3D JSON_ESCAPE, + /* interpolation */ + [IN_INTERPOL_LL] =3D { + ['d'] =3D JSON_INTERPOL, + ['u'] =3D JSON_INTERPOL, }, =20 - [IN_ESCAPE_L] =3D { - ['d'] =3D JSON_ESCAPE, - ['l'] =3D IN_ESCAPE_LL, - ['u'] =3D JSON_ESCAPE, + [IN_INTERPOL_L] =3D { + ['d'] =3D JSON_INTERPOL, + ['l'] =3D IN_INTERPOL_LL, + ['u'] =3D JSON_INTERPOL, }, =20 - [IN_ESCAPE_I64] =3D { - ['d'] =3D JSON_ESCAPE, - ['u'] =3D JSON_ESCAPE, + [IN_INTERPOL_I64] =3D { + ['d'] =3D JSON_INTERPOL, + ['u'] =3D JSON_INTERPOL, }, =20 - [IN_ESCAPE_I6] =3D { - ['4'] =3D IN_ESCAPE_I64, + [IN_INTERPOL_I6] =3D { + ['4'] =3D IN_INTERPOL_I64, }, =20 - [IN_ESCAPE_I] =3D { - ['6'] =3D IN_ESCAPE_I6, + [IN_INTERPOL_I] =3D { + ['6'] =3D IN_INTERPOL_I6, }, =20 - [IN_ESCAPE] =3D { - ['d'] =3D JSON_ESCAPE, - ['i'] =3D JSON_ESCAPE, - ['p'] =3D JSON_ESCAPE, - ['s'] =3D JSON_ESCAPE, - ['u'] =3D JSON_ESCAPE, - ['f'] =3D JSON_ESCAPE, - ['l'] =3D IN_ESCAPE_L, - ['I'] =3D IN_ESCAPE_I, + [IN_INTERPOL] =3D { + ['d'] =3D JSON_INTERPOL, + ['i'] =3D JSON_INTERPOL, + ['p'] =3D JSON_INTERPOL, + ['s'] =3D JSON_INTERPOL, + ['u'] =3D JSON_INTERPOL, + ['f'] =3D JSON_INTERPOL, + ['l'] =3D IN_INTERPOL_L, + ['I'] =3D IN_INTERPOL_I, }, =20 /* top level rule */ @@ -271,7 +271,7 @@ static const uint8_t json_lexer[][256] =3D { [','] =3D JSON_COMMA, [':'] =3D JSON_COLON, ['a' ... 'z'] =3D IN_KEYWORD, - ['%'] =3D IN_ESCAPE, + ['%'] =3D IN_INTERPOL, [' '] =3D IN_WHITESPACE, ['\t'] =3D IN_WHITESPACE, ['\r'] =3D IN_WHITESPACE, @@ -311,7 +311,7 @@ static void json_lexer_feed_char(JSONLexer *lexer, char= ch, bool flush) case JSON_RSQUARE: case JSON_COLON: case JSON_COMMA: - case JSON_ESCAPE: + case JSON_INTERPOL: case JSON_INTEGER: case JSON_FLOAT: case JSON_KEYWORD: diff --git a/qobject/json-parser.c b/qobject/json-parser.c index 0e4ea564ab..f1806ce0dc 100644 --- a/qobject/json-parser.c +++ b/qobject/json-parser.c @@ -418,7 +418,7 @@ static QObject *parse_keyword(JSONParserContext *ctxt) return NULL; } =20 -static QObject *parse_escape(JSONParserContext *ctxt, va_list *ap) +static QObject *parse_interpolation(JSONParserContext *ctxt, va_list *ap) { JSONToken *token; =20 @@ -427,7 +427,7 @@ static QObject *parse_escape(JSONParserContext *ctxt, v= a_list *ap) } =20 token =3D parser_context_pop_token(ctxt); - assert(token && token->type =3D=3D JSON_ESCAPE); + assert(token && token->type =3D=3D JSON_INTERPOL); =20 if (!strcmp(token->str, "%p")) { return va_arg(*ap, QObject *); @@ -522,8 +522,8 @@ static QObject *parse_value(JSONParserContext *ctxt, va= _list *ap) return parse_object(ctxt, ap); case JSON_LSQUARE: return parse_array(ctxt, ap); - case JSON_ESCAPE: - return parse_escape(ctxt, ap); + case JSON_INTERPOL: + return parse_interpolation(ctxt, ap); case JSON_INTEGER: case JSON_FLOAT: case JSON_STRING: --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533732942110247.137838937796; Wed, 8 Aug 2018 05:55:42 -0700 (PDT) Received: from localhost ([::1]:43529 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnO04-0001PN-VG for importer@patchew.org; Wed, 08 Aug 2018 08:55:41 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52718) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBv-0006oy-0o for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:58 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBk-00005u-Ph for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:51 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:51094 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBk-0008VF-AP for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:40 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 9E8464023827; Wed, 8 Aug 2018 12:03:39 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 53DEA20180F6; Wed, 8 Aug 2018 12:03:39 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id C806111386D8; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:15 +0200 Message-Id: <20180808120334.10970-38-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:39 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:39 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 37/56] json: Treat unwanted interpolation as lexical error X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The JSON parser optionally supports interpolation. The lexer recognizes interpolation tokens unconditionally. The parser rejects them when interpolation is disabled, in parse_interpolation(). However, it neglects to set an error then, which can make json_parser_parse() fail without setting an error. Move the check for unwanted interpolation from the parser's parse_interpolation() into the lexer's finite state machine. When interpolation is disabled, '%' is now handled like any other unexpected character. The next commit will improve how such lexical errors are handled. Signed-off-by: Markus Armbruster --- include/qapi/qmp/json-lexer.h | 4 ++-- qobject/json-lexer.c | 42 ++++++++++++++++++++++++++--------- qobject/json-parser.c | 4 ---- qobject/json-streamer.c | 2 +- tests/qmp-test.c | 4 ++++ 5 files changed, 39 insertions(+), 17 deletions(-) diff --git a/include/qapi/qmp/json-lexer.h b/include/qapi/qmp/json-lexer.h index ff3a6f80f0..5586d12f26 100644 --- a/include/qapi/qmp/json-lexer.h +++ b/include/qapi/qmp/json-lexer.h @@ -33,12 +33,12 @@ typedef enum json_token_type { } JSONTokenType; =20 typedef struct JSONLexer { - int state; + int start_state, state; GString *token; int x, y; } JSONLexer; =20 -void json_lexer_init(JSONLexer *lexer); +void json_lexer_init(JSONLexer *lexer, bool enable_interpolation); =20 void json_lexer_feed(JSONLexer *lexer, const char *buffer, size_t size); =20 diff --git a/qobject/json-lexer.c b/qobject/json-lexer.c index 5b1f720331..0ea1eae4aa 100644 --- a/qobject/json-lexer.c +++ b/qobject/json-lexer.c @@ -122,6 +122,7 @@ enum json_lexer_state { IN_INTERPOL_I6, IN_INTERPOL_I64, IN_WHITESPACE, + IN_START_INTERPOL, IN_START, }; =20 @@ -271,17 +272,38 @@ static const uint8_t json_lexer[][256] =3D { [','] =3D JSON_COMMA, [':'] =3D JSON_COLON, ['a' ... 'z'] =3D IN_KEYWORD, + [' '] =3D IN_WHITESPACE, + ['\t'] =3D IN_WHITESPACE, + ['\r'] =3D IN_WHITESPACE, + ['\n'] =3D IN_WHITESPACE, + }, + + [IN_START_INTERPOL] =3D { + ['"'] =3D IN_DQ_STRING, + ['\''] =3D IN_SQ_STRING, + ['0'] =3D IN_ZERO, + ['1' ... '9'] =3D IN_NONZERO_NUMBER, + ['-'] =3D IN_NEG_NONZERO_NUMBER, + ['{'] =3D JSON_LCURLY, + ['}'] =3D JSON_RCURLY, + ['['] =3D JSON_LSQUARE, + [']'] =3D JSON_RSQUARE, + [','] =3D JSON_COMMA, + [':'] =3D JSON_COLON, + ['a' ... 'z'] =3D IN_KEYWORD, + [' '] =3D IN_WHITESPACE, + ['\t'] =3D IN_WHITESPACE, + ['\r'] =3D IN_WHITESPACE, + ['\n'] =3D IN_WHITESPACE, + /* matches IN_START up to here */ ['%'] =3D IN_INTERPOL, - [' '] =3D IN_WHITESPACE, - ['\t'] =3D IN_WHITESPACE, - ['\r'] =3D IN_WHITESPACE, - ['\n'] =3D IN_WHITESPACE, }, }; =20 -void json_lexer_init(JSONLexer *lexer) +void json_lexer_init(JSONLexer *lexer, bool enable_interpolation) { - lexer->state =3D IN_START; + lexer->start_state =3D lexer->state =3D enable_interpolation + ? IN_START_INTERPOL : IN_START; lexer->token =3D g_string_sized_new(3); lexer->x =3D lexer->y =3D 0; } @@ -321,7 +343,7 @@ static void json_lexer_feed_char(JSONLexer *lexer, char= ch, bool flush) /* fall through */ case JSON_SKIP: g_string_truncate(lexer->token, 0); - new_state =3D IN_START; + new_state =3D lexer->start_state; break; case IN_ERROR: /* XXX: To avoid having previous bad input leaving the parser = in an @@ -340,7 +362,7 @@ static void json_lexer_feed_char(JSONLexer *lexer, char= ch, bool flush) json_message_process_token(lexer, lexer->token, JSON_ERROR, lexer->x, lexer->y); g_string_truncate(lexer->token, 0); - new_state =3D IN_START; + new_state =3D lexer->start_state; lexer->state =3D new_state; return; default: @@ -356,7 +378,7 @@ static void json_lexer_feed_char(JSONLexer *lexer, char= ch, bool flush) json_message_process_token(lexer, lexer->token, lexer->state, lexer->x, lexer->y); g_string_truncate(lexer->token, 0); - lexer->state =3D IN_START; + lexer->state =3D lexer->start_state; } } =20 @@ -371,7 +393,7 @@ void json_lexer_feed(JSONLexer *lexer, const char *buff= er, size_t size) =20 void json_lexer_flush(JSONLexer *lexer) { - if (lexer->state !=3D IN_START) { + if (lexer->state !=3D lexer->start_state) { json_lexer_feed_char(lexer, 0, true); } } diff --git a/qobject/json-parser.c b/qobject/json-parser.c index f1806ce0dc..848d469b2a 100644 --- a/qobject/json-parser.c +++ b/qobject/json-parser.c @@ -422,10 +422,6 @@ static QObject *parse_interpolation(JSONParserContext = *ctxt, va_list *ap) { JSONToken *token; =20 - if (ap =3D=3D NULL) { - return NULL; - } - token =3D parser_context_pop_token(ctxt); assert(token && token->type =3D=3D JSON_INTERPOL); =20 diff --git a/qobject/json-streamer.c b/qobject/json-streamer.c index fa595a8761..a373e0114a 100644 --- a/qobject/json-streamer.c +++ b/qobject/json-streamer.c @@ -115,7 +115,7 @@ void json_message_parser_init(JSONMessageParser *parser, parser->tokens =3D g_queue_new(); parser->token_size =3D 0; =20 - json_lexer_init(&parser->lexer); + json_lexer_init(&parser->lexer, !!ap); } =20 void json_message_parser_feed(JSONMessageParser *parser, diff --git a/tests/qmp-test.c b/tests/qmp-test.c index b77987b644..3046567819 100644 --- a/tests/qmp-test.c +++ b/tests/qmp-test.c @@ -94,6 +94,10 @@ static void test_malformed(QTestState *qts) =20 /* lexical error: interpolation */ qtest_qmp_send_raw(qts, "%%p\n"); + /* two errors, one for "%", one for "p" */ + resp =3D qtest_qmp_receive(qts); + g_assert_cmpstr(get_error_class(resp), =3D=3D, "GenericError"); + qobject_unref(resp); resp =3D qtest_qmp_receive(qts); g_assert_cmpstr(get_error_class(resp), =3D=3D, "GenericError"); qobject_unref(resp); --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533732072449628.0751242759123; Wed, 8 Aug 2018 05:41:12 -0700 (PDT) Received: from localhost ([::1]:43389 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNm3-0005W4-55 for importer@patchew.org; Wed, 08 Aug 2018 08:41:11 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52669) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBt-0006op-To for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:55 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBm-00008s-0J for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:49 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:45448 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBl-00006E-Bz for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:41 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id A9F0C8182D38; Wed, 8 Aug 2018 12:03:40 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 5CCFB2166BA0; Wed, 8 Aug 2018 12:03:40 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id CE2C911386DA; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:16 +0200 Message-Id: <20180808120334.10970-39-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 08 Aug 2018 12:03:40 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 08 Aug 2018 12:03:40 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 38/56] json: Pass lexical errors and limit violations to callback X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The callback to consume JSON values takes QObject *json, Error *err. If both are null, the callback is supposed to make up an error by itself. This sucks. qjson.c's consume_json() neglects to do so, which makes qobject_from_json() & friends return null instead of failing. I consider that a bug. The culprit is json_message_process_token(): it passes two null pointers when it runs into a lexical error or a limit violation. Fix it to pass a proper Error object then. Update the callbacks: * monitor.c's handle_qmp_command(): the code to make up an error is now dead, drop it. * qga/main.c's process_event(): lumps the "both null" case together with the "not a JSON object" case. The former is now gone. The error message "Invalid JSON syntax" is misleading for the latter. Improve it to "Input must be a JSON object". * qobject/qjson.c's consume_json(): no update; check-qjson demonstrates qobject_from_json() now sets an error on lexical errors, but still doesn't on some other errors. * tests/libqtest.c's qmp_response(): the Error object is now reliable, so use it to improve the error message. Signed-off-by: Markus Armbruster --- include/qapi/qmp/qerror.h | 3 --- monitor.c | 5 +---- qga/main.c | 3 ++- qobject/json-streamer.c | 22 ++++++++++++++++------ tests/check-qjson.c | 14 +++++++------- tests/libqtest.c | 7 +++++-- 6 files changed, 31 insertions(+), 23 deletions(-) diff --git a/include/qapi/qmp/qerror.h b/include/qapi/qmp/qerror.h index c82360f429..145571f618 100644 --- a/include/qapi/qmp/qerror.h +++ b/include/qapi/qmp/qerror.h @@ -61,9 +61,6 @@ #define QERR_IO_ERROR \ "An IO error has occurred" =20 -#define QERR_JSON_PARSING \ - "Invalid JSON syntax" - #define QERR_MIGRATION_ACTIVE \ "There's a migration process in progress" =20 diff --git a/monitor.c b/monitor.c index 71658d9905..dc0ed8df92 100644 --- a/monitor.c +++ b/monitor.c @@ -4251,10 +4251,7 @@ static void handle_qmp_command(void *opaque, QObject= *req, Error *err) QDict *qdict; QMPRequest *req_obj; =20 - if (!req && !err) { - /* json_parser_parse() sucks: can fail without setting @err */ - error_setg(&err, QERR_JSON_PARSING); - } + assert(!req !=3D !err); =20 qdict =3D qobject_to(QDict, req); if (qdict) { diff --git a/qga/main.c b/qga/main.c index 2fc49d00d8..b74e1241ef 100644 --- a/qga/main.c +++ b/qga/main.c @@ -603,12 +603,13 @@ static void process_event(void *opaque, QObject *obj,= Error *err) int ret; =20 g_debug("process_event: called"); + assert(!obj !=3D !err); if (err) { goto err; } req =3D qobject_to(QDict, obj); if (!req) { - error_setg(&err, QERR_JSON_PARSING); + error_setg(&err, "Input must be a JSON object"); goto err; } if (!qdict_haskey(req, "execute")) { diff --git a/qobject/json-streamer.c b/qobject/json-streamer.c index a373e0114a..e372ecc895 100644 --- a/qobject/json-streamer.c +++ b/qobject/json-streamer.c @@ -13,6 +13,7 @@ =20 #include "qemu/osdep.h" #include "qemu-common.h" +#include "qapi/error.h" #include "qapi/qmp/json-lexer.h" #include "qapi/qmp/json-parser.h" #include "qapi/qmp/json-streamer.h" @@ -57,6 +58,7 @@ void json_message_process_token(JSONLexer *lexer, GString= *input, parser->bracket_count--; break; case JSON_ERROR: + error_setg(&err, "JSON parse error, stray '%s'", input->str); goto out_emit; default: break; @@ -82,12 +84,20 @@ void json_message_process_token(JSONLexer *lexer, GStri= ng *input, goto out_emit; } =20 - if (parser->token_size > MAX_TOKEN_SIZE || - g_queue_get_length(parser->tokens) > MAX_TOKEN_COUNT || - parser->bracket_count + parser->brace_count > MAX_NESTING) { - /* Security consideration, we limit total memory allocated per obj= ect - * and the maximum recursion depth that a message can force. - */ + /* + * Security consideration, we limit total memory allocated per object + * and the maximum recursion depth that a message can force. + */ + if (parser->token_size > MAX_TOKEN_SIZE) { + error_setg(&err, "JSON token size limit exceeded"); + goto out_emit; + } + if (g_queue_get_length(parser->tokens) > MAX_TOKEN_COUNT) { + error_setg(&err, "JSON token count limit exceeded"); + goto out_emit; + } + if (parser->bracket_count + parser->brace_count > MAX_NESTING) { + error_setg(&err, "JSON nesting depth limit exceeded"); goto out_emit; } =20 diff --git a/tests/check-qjson.c b/tests/check-qjson.c index 4c4afcf691..895be489b3 100644 --- a/tests/check-qjson.c +++ b/tests/check-qjson.c @@ -1247,11 +1247,11 @@ static void junk_input(void) QObject *obj; =20 obj =3D qobject_from_json("@", &err); - g_assert(!err); /* BUG */ + error_free_or_abort(&err); g_assert(obj =3D=3D NULL); =20 obj =3D qobject_from_json("{\x01", &err); - g_assert(!err); /* BUG */ + error_free_or_abort(&err); g_assert(obj =3D=3D NULL); =20 obj =3D qobject_from_json("[0\xFF]", &err); @@ -1259,11 +1259,11 @@ static void junk_input(void) g_assert(obj =3D=3D NULL); =20 obj =3D qobject_from_json("00", &err); - g_assert(!err); /* BUG */ + error_free_or_abort(&err); g_assert(obj =3D=3D NULL); =20 obj =3D qobject_from_json("[1e", &err); - g_assert(!err); /* BUG */ + error_free_or_abort(&err); g_assert(obj =3D=3D NULL); } =20 @@ -1271,7 +1271,7 @@ static void unterminated_string(void) { Error *err =3D NULL; QObject *obj =3D qobject_from_json("\"abc", &err); - g_assert(!err); /* BUG */ + error_free_or_abort(&err); g_assert(obj =3D=3D NULL); } =20 @@ -1279,7 +1279,7 @@ static void unterminated_sq_string(void) { Error *err =3D NULL; QObject *obj =3D qobject_from_json("'abc", &err); - g_assert(!err); /* BUG */ + error_free_or_abort(&err); g_assert(obj =3D=3D NULL); } =20 @@ -1287,7 +1287,7 @@ static void unterminated_escape(void) { Error *err =3D NULL; QObject *obj =3D qobject_from_json("\"abc\\\"", &err); - g_assert(!err); /* BUG */ + error_free_or_abort(&err); g_assert(obj =3D=3D NULL); } =20 diff --git a/tests/libqtest.c b/tests/libqtest.c index aa451214d9..7ef8dd621f 100644 --- a/tests/libqtest.c +++ b/tests/libqtest.c @@ -431,8 +431,11 @@ static void qmp_response(void *opaque, QObject *obj, E= rror *err) { QMPResponseParser *qmp =3D opaque; =20 - if (!obj) { - fprintf(stderr, "QMP JSON response parsing failed\n"); + assert(!obj !=3D !err); + + if (err) { + error_prepend(&err, "QMP JSON response parsing failed: "); + error_report_err(err); exit(1); } =20 --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533730593977193.92764153719725; Wed, 8 Aug 2018 05:16:33 -0700 (PDT) Received: from localhost ([::1]:43226 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNOC-00013l-Nz for importer@patchew.org; Wed, 08 Aug 2018 08:16:32 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52574) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBq-0006js-2l for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBl-000070-9A for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:46 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:51102 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBk-0008Vo-Ox for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:40 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id BDB87402382B; Wed, 8 Aug 2018 12:03:39 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 770E51C716; Wed, 8 Aug 2018 12:03:39 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id D435011386DB; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:17 +0200 Message-Id: <20180808120334.10970-40-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:39 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:39 +0000 (UTC) for IP:'10.11.54.5' DOMAIN:'int-mx05.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 39/56] json: Leave rejecting invalid interpolation to parser X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Both lexer and parser reject invalid interpolation specifications. The parser's check is useless. The lexer ends the token right after the first bad character. This tends to lead to suboptimal error reporting. For instance, input [ %11d ] produces the tokens JSON_LSQUARE [ JSON_ERROR %1 JSON_INTEGER 1 JSON_KEYWORD d JSON_RSQUARE ] The parser then yields an error, an object and two more errors: error: Invalid JSON syntax object: 1 error: JSON parse error, invalid keyword error: JSON parse error, expecting value Change the lexer to accept [A-Za-z0-9]*[duipsf]. It now produces JSON_LSQUARE [ JSON_INTERPOLATION %11d JSON_RSQUARE ] and the parser reports just JSON parse error, invalid interpolation '%11d' Signed-off-by: Markus Armbruster --- qobject/json-lexer.c | 52 +++++++++---------------------------------- qobject/json-parser.c | 1 + 2 files changed, 11 insertions(+), 42 deletions(-) diff --git a/qobject/json-lexer.c b/qobject/json-lexer.c index 0ea1eae4aa..7a82aab88b 100644 --- a/qobject/json-lexer.c +++ b/qobject/json-lexer.c @@ -93,7 +93,8 @@ * (apostrophe) instead of %x22 (quotation mark), and can't contain * unescaped apostrophe, but can contain unescaped quotation mark. * - Interpolation: - * interpolation =3D %((l|ll|I64)[du]|[ipsf]) + * The lexer accepts [A-Za-z0-9]*, and leaves rejecting invalid ones + * to the parser. * * Note: * - Input must be encoded in modified UTF-8. @@ -116,11 +117,6 @@ enum json_lexer_state { IN_NEG_NONZERO_NUMBER, IN_KEYWORD, IN_INTERPOL, - IN_INTERPOL_L, - IN_INTERPOL_LL, - IN_INTERPOL_I, - IN_INTERPOL_I6, - IN_INTERPOL_I64, IN_WHITESPACE, IN_START_INTERPOL, IN_START, @@ -222,42 +218,6 @@ static const uint8_t json_lexer[][256] =3D { ['\n'] =3D IN_WHITESPACE, }, =20 - /* interpolation */ - [IN_INTERPOL_LL] =3D { - ['d'] =3D JSON_INTERPOL, - ['u'] =3D JSON_INTERPOL, - }, - - [IN_INTERPOL_L] =3D { - ['d'] =3D JSON_INTERPOL, - ['l'] =3D IN_INTERPOL_LL, - ['u'] =3D JSON_INTERPOL, - }, - - [IN_INTERPOL_I64] =3D { - ['d'] =3D JSON_INTERPOL, - ['u'] =3D JSON_INTERPOL, - }, - - [IN_INTERPOL_I6] =3D { - ['4'] =3D IN_INTERPOL_I64, - }, - - [IN_INTERPOL_I] =3D { - ['6'] =3D IN_INTERPOL_I6, - }, - - [IN_INTERPOL] =3D { - ['d'] =3D JSON_INTERPOL, - ['i'] =3D JSON_INTERPOL, - ['p'] =3D JSON_INTERPOL, - ['s'] =3D JSON_INTERPOL, - ['u'] =3D JSON_INTERPOL, - ['f'] =3D JSON_INTERPOL, - ['l'] =3D IN_INTERPOL_L, - ['I'] =3D IN_INTERPOL_I, - }, - /* top level rule */ [IN_START] =3D { ['"'] =3D IN_DQ_STRING, @@ -278,6 +238,14 @@ static const uint8_t json_lexer[][256] =3D { ['\n'] =3D IN_WHITESPACE, }, =20 + /* interpolation */ + [IN_INTERPOL] =3D { + TERMINAL(JSON_INTERPOL), + ['A' ... 'Z'] =3D IN_INTERPOL, + ['a' ... 'z'] =3D IN_INTERPOL, + ['0' ... '9'] =3D IN_INTERPOL, + }, + [IN_START_INTERPOL] =3D { ['"'] =3D IN_DQ_STRING, ['\''] =3D IN_SQ_STRING, diff --git a/qobject/json-parser.c b/qobject/json-parser.c index 848d469b2a..bd137399e5 100644 --- a/qobject/json-parser.c +++ b/qobject/json-parser.c @@ -448,6 +448,7 @@ static QObject *parse_interpolation(JSONParserContext *= ctxt, va_list *ap) } else if (!strcmp(token->str, "%f")) { return QOBJECT(qnum_from_double(va_arg(*ap, double))); } + parse_error(ctxt, token, "invalid interpolation '%s'", token->str); return NULL; } =20 --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533732615760845.5735142255758; Wed, 8 Aug 2018 05:50:15 -0700 (PDT) Received: from localhost ([::1]:43464 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNuo-0004hL-NQ for importer@patchew.org; Wed, 08 Aug 2018 08:50:14 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52667) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBt-0006on-T7 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:58 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBm-000097-3U for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:49 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:37304 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBl-00006d-Jr for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:41 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D3C2040216FA; Wed, 8 Aug 2018 12:03:40 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 7CCAB2156897; Wed, 8 Aug 2018 12:03:40 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id DA45411386DC; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:18 +0200 Message-Id: <20180808120334.10970-41-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Wed, 08 Aug 2018 12:03:40 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Wed, 08 Aug 2018 12:03:40 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 40/56] json: Replace %I64d, %I64u by %PRId64, %PRIu64 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Support for %I64d got addded in commit 2c0d4b36e7f "json: fix PRId64 on Win32". We had to hard-code I64d because we used the lexer's finite state machine to check interpolations. No more, so clean this up. Additional conversion specifications would be easy enough to implement when needed. Signed-off-by: Markus Armbruster --- qobject/json-parser.c | 10 ++++++---- tests/check-qjson.c | 10 ++++++++++ 2 files changed, 16 insertions(+), 4 deletions(-) diff --git a/qobject/json-parser.c b/qobject/json-parser.c index bd137399e5..350a9d267b 100644 --- a/qobject/json-parser.c +++ b/qobject/json-parser.c @@ -433,16 +433,18 @@ static QObject *parse_interpolation(JSONParserContext= *ctxt, va_list *ap) return QOBJECT(qnum_from_int(va_arg(*ap, int))); } else if (!strcmp(token->str, "%ld")) { return QOBJECT(qnum_from_int(va_arg(*ap, long))); - } else if (!strcmp(token->str, "%lld") || - !strcmp(token->str, "%I64d")) { + } else if (!strcmp(token->str, "%lld")) { return QOBJECT(qnum_from_int(va_arg(*ap, long long))); + } else if (!strcmp(token->str, "%" PRId64)) { + return QOBJECT(qnum_from_int(va_arg(*ap, int64_t))); } else if (!strcmp(token->str, "%u")) { return QOBJECT(qnum_from_uint(va_arg(*ap, unsigned int))); } else if (!strcmp(token->str, "%lu")) { return QOBJECT(qnum_from_uint(va_arg(*ap, unsigned long))); - } else if (!strcmp(token->str, "%llu") || - !strcmp(token->str, "%I64u")) { + } else if (!strcmp(token->str, "%llu")) { return QOBJECT(qnum_from_uint(va_arg(*ap, unsigned long long))); + } else if (!strcmp(token->str, "%" PRIu64)) { + return QOBJECT(qnum_from_uint(va_arg(*ap, uint64_t))); } else if (!strcmp(token->str, "%s")) { return QOBJECT(qstring_from_str(va_arg(*ap, const char *))); } else if (!strcmp(token->str, "%f")) { diff --git a/tests/check-qjson.c b/tests/check-qjson.c index 895be489b3..fbb607c227 100644 --- a/tests/check-qjson.c +++ b/tests/check-qjson.c @@ -919,9 +919,11 @@ static void keyword_literal(void) static void interpolation(void) { long long value_lld =3D 0x123456789abcdefLL; + int64_t value_d64 =3D value_lld; long value_ld =3D (long)value_lld; int value_d =3D (int)value_lld; unsigned long long value_llu =3D 0xfedcba9876543210ULL; + uint64_t value_u64 =3D value_llu; unsigned long value_lu =3D (unsigned long)value_llu; unsigned value_u =3D (unsigned)value_llu; double value_f =3D 2.323423423; @@ -959,6 +961,10 @@ static void interpolation(void) g_assert_cmpint(qnum_get_int(qnum), =3D=3D, value_lld); qobject_unref(qnum); =20 + qnum =3D qobject_to(QNum, qobject_from_jsonf_nofail("%" PRId64, value_= d64)); + g_assert_cmpint(qnum_get_int(qnum), =3D=3D, value_lld); + qobject_unref(qnum); + qnum =3D qobject_to(QNum, qobject_from_jsonf_nofail("%u", value_u)); g_assert_cmpuint(qnum_get_uint(qnum), =3D=3D, value_u); qobject_unref(qnum); @@ -971,6 +977,10 @@ static void interpolation(void) g_assert_cmpuint(qnum_get_uint(qnum), =3D=3D, value_llu); qobject_unref(qnum); =20 + qnum =3D qobject_to(QNum, qobject_from_jsonf_nofail("%" PRIu64, value_= u64)); + g_assert_cmpuint(qnum_get_uint(qnum), =3D=3D, value_llu); + qobject_unref(qnum); + qnum =3D qobject_to(QNum, qobject_from_jsonf_nofail("%f", value_f)); g_assert(qnum_get_double(qnum) =3D=3D value_f); qobject_unref(qnum); --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533731883573119.3682005941572; Wed, 8 Aug 2018 05:38:03 -0700 (PDT) Received: from localhost ([::1]:43371 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNj0-0002cN-Ea for importer@patchew.org; Wed, 08 Aug 2018 08:38:02 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52543) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBp-0006iy-CR for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:52 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBk-00005n-OI for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:45 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:45406 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBk-0008VK-BH for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:40 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id BC0B581663FE; Wed, 8 Aug 2018 12:03:39 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 9445210CD88E; Wed, 8 Aug 2018 12:03:39 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id E047211386DE; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:19 +0200 Message-Id: <20180808120334.10970-42-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 08 Aug 2018 12:03:39 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 08 Aug 2018 12:03:39 +0000 (UTC) for IP:'10.11.54.3' DOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 41/56] json: Nicer recovery from invalid leading zero X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" For input 0123, the lexer produces the tokens JSON_ERROR 01 JSON_INTEGER 23 Reporting an error is correct; 0123 is invalid according to RFC 7159. But the error recovery isn't nice. Make the finite state machine eat digits before going into the error state. The lexer now produces JSON_ERROR 0123 Signed-off-by: Markus Armbruster --- qobject/json-lexer.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/qobject/json-lexer.c b/qobject/json-lexer.c index 7a82aab88b..f600cc732e 100644 --- a/qobject/json-lexer.c +++ b/qobject/json-lexer.c @@ -108,6 +108,7 @@ enum json_lexer_state { IN_SQ_STRING_ESCAPE, IN_SQ_STRING, IN_ZERO, + IN_BAD_ZERO, IN_DIGITS, IN_DIGIT, IN_EXP_E, @@ -158,10 +159,14 @@ static const uint8_t json_lexer[][256] =3D { /* Zero */ [IN_ZERO] =3D { TERMINAL(JSON_INTEGER), - ['0' ... '9'] =3D IN_ERROR, + ['0' ... '9'] =3D IN_BAD_ZERO, ['.'] =3D IN_MANTISSA, }, =20 + [IN_BAD_ZERO] =3D { + ['0' ... '9'] =3D IN_BAD_ZERO, + }, + /* Float */ [IN_DIGITS] =3D { TERMINAL(JSON_FLOAT), --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 153373241562450.415920022276396; Wed, 8 Aug 2018 05:46:55 -0700 (PDT) Received: from localhost ([::1]:43419 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNra-0002AZ-CS for importer@patchew.org; Wed, 08 Aug 2018 08:46:54 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52879) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNC0-0006up-68 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:58 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBn-0000Ac-NJ for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:56 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:47790 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBn-0000AB-E9 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:43 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id DF53B72620; Wed, 8 Aug 2018 12:03:42 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 984C010CD855; Wed, 8 Aug 2018 12:03:42 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id E5EDF11386DF; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:20 +0200 Message-Id: <20180808120334.10970-43-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 08 Aug 2018 12:03:42 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 08 Aug 2018 12:03:42 +0000 (UTC) for IP:'10.11.54.3' DOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 42/56] json: Improve names of lexer states related to numbers X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Signed-off-by: Markus Armbruster --- qobject/json-lexer.c | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/qobject/json-lexer.c b/qobject/json-lexer.c index f600cc732e..733ce3f5ba 100644 --- a/qobject/json-lexer.c +++ b/qobject/json-lexer.c @@ -109,13 +109,13 @@ enum json_lexer_state { IN_SQ_STRING, IN_ZERO, IN_BAD_ZERO, - IN_DIGITS, - IN_DIGIT, + IN_EXP_DIGITS, + IN_EXP_SIGN, IN_EXP_E, IN_MANTISSA, IN_MANTISSA_DIGITS, - IN_NONZERO_NUMBER, - IN_NEG_NONZERO_NUMBER, + IN_DIGITS, + IN_SIGN, IN_KEYWORD, IN_INTERPOL, IN_WHITESPACE, @@ -168,19 +168,19 @@ static const uint8_t json_lexer[][256] =3D { }, =20 /* Float */ - [IN_DIGITS] =3D { + [IN_EXP_DIGITS] =3D { TERMINAL(JSON_FLOAT), - ['0' ... '9'] =3D IN_DIGITS, + ['0' ... '9'] =3D IN_EXP_DIGITS, }, =20 - [IN_DIGIT] =3D { - ['0' ... '9'] =3D IN_DIGITS, + [IN_EXP_SIGN] =3D { + ['0' ... '9'] =3D IN_EXP_DIGITS, }, =20 [IN_EXP_E] =3D { - ['-'] =3D IN_DIGIT, - ['+'] =3D IN_DIGIT, - ['0' ... '9'] =3D IN_DIGITS, + ['-'] =3D IN_EXP_SIGN, + ['+'] =3D IN_EXP_SIGN, + ['0' ... '9'] =3D IN_EXP_DIGITS, }, =20 [IN_MANTISSA_DIGITS] =3D { @@ -195,17 +195,17 @@ static const uint8_t json_lexer[][256] =3D { }, =20 /* Number */ - [IN_NONZERO_NUMBER] =3D { + [IN_DIGITS] =3D { TERMINAL(JSON_INTEGER), - ['0' ... '9'] =3D IN_NONZERO_NUMBER, + ['0' ... '9'] =3D IN_DIGITS, ['e'] =3D IN_EXP_E, ['E'] =3D IN_EXP_E, ['.'] =3D IN_MANTISSA, }, =20 - [IN_NEG_NONZERO_NUMBER] =3D { + [IN_SIGN] =3D { ['0'] =3D IN_ZERO, - ['1' ... '9'] =3D IN_NONZERO_NUMBER, + ['1' ... '9'] =3D IN_DIGITS, }, =20 /* keywords */ @@ -228,8 +228,8 @@ static const uint8_t json_lexer[][256] =3D { ['"'] =3D IN_DQ_STRING, ['\''] =3D IN_SQ_STRING, ['0'] =3D IN_ZERO, - ['1' ... '9'] =3D IN_NONZERO_NUMBER, - ['-'] =3D IN_NEG_NONZERO_NUMBER, + ['1' ... '9'] =3D IN_DIGITS, + ['-'] =3D IN_SIGN, ['{'] =3D JSON_LCURLY, ['}'] =3D JSON_RCURLY, ['['] =3D JSON_LSQUARE, @@ -255,8 +255,8 @@ static const uint8_t json_lexer[][256] =3D { ['"'] =3D IN_DQ_STRING, ['\''] =3D IN_SQ_STRING, ['0'] =3D IN_ZERO, - ['1' ... '9'] =3D IN_NONZERO_NUMBER, - ['-'] =3D IN_NEG_NONZERO_NUMBER, + ['1' ... '9'] =3D IN_DIGITS, + ['-'] =3D IN_SIGN, ['{'] =3D JSON_LCURLY, ['}'] =3D JSON_RCURLY, ['['] =3D JSON_LSQUARE, --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533730789961686.3986480657471; Wed, 8 Aug 2018 05:19:49 -0700 (PDT) Received: from localhost ([::1]:43268 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNRM-0003Uj-Kr for importer@patchew.org; Wed, 08 Aug 2018 08:19:48 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52617) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBq-0006lo-Qb for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:54 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBl-00007G-9P for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:46 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:45412 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBk-0008WD-QP for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:41 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 27F9C81663CF; Wed, 8 Aug 2018 12:03:40 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D59FB1C716; Wed, 8 Aug 2018 12:03:39 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id EAFDC11385D3; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:21 +0200 Message-Id: <20180808120334.10970-44-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 08 Aug 2018 12:03:40 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 08 Aug 2018 12:03:40 +0000 (UTC) for IP:'10.11.54.5' DOMAIN:'int-mx05.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 43/56] qjson: Fix qobject_from_json() & friends for multiple values X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" qobject_from_json() & friends use the consume_json() callback to receive either a value or an error from the parser. When they are fed a string that contains more than either one JSON value or one JSON syntax error, consume_json() gets called multiple times. When the last call receives a value, qobject_from_json() returns that value. Any other values are leaked. When any call receives an error, qobject_from_json() sets the first error received. Any other errors are thrown away. When values follow errors, qobject_from_json() returns both a value and sets an error. That's bad. Impact: * block.c's parse_json_protocol() ignores and leaks the value. It's used to to parse pseudo-filenames starting with "json:". The pseudo-filenames can come from the user or from image meta-data such as a QCOW2 image's backing file name. * vl.c's parse_display_qapi() ignores and leaks the error. It's used to parse the argument of command line option -display. * vl.c's main() case QEMU_OPTION_blockdev ignores the error and leaves it in @err. main() will then pass a pointer to a non-null Error * to net_init_clients(), which is forbidden. It can lead to assertion failure or other misbehavior. * check-qjson.c's multiple_values() demonstrates the badness. * The other callers are not affected since they only pass strings with exactly one JSON value or, in the case of negative tests, one error. The impact on the _nofail() functions is relatively harmless. They abort when any call receives an error. Else they return the last value, and leak the others, if any. Fix consume_json() as follows. On the first call, save value and error as before. On subsequent calls, if any, don't save them. If the first call saved a value, the next call, if any, replaces the value by an "Expecting at most one JSON value" error. Take care not to leak values or errors that aren't saved. Signed-off-by: Markus Armbruster --- qobject/qjson.c | 15 ++++++++++++++- tests/check-qjson.c | 10 +++------- 2 files changed, 17 insertions(+), 8 deletions(-) diff --git a/qobject/qjson.c b/qobject/qjson.c index 7395556069..7f69036487 100644 --- a/qobject/qjson.c +++ b/qobject/qjson.c @@ -33,8 +33,21 @@ static void consume_json(void *opaque, QObject *json, Er= ror *err) { JSONParsingState *s =3D opaque; =20 + assert(!json !=3D !err); + assert(!s->result || !s->err); + + if (s->result) { + qobject_unref(s->result); + s->result =3D NULL; + error_setg(&s->err, "Expecting at most one JSON value"); + } + if (s->err) { + qobject_unref(json); + error_free(err); + return; + } s->result =3D json; - error_propagate(&s->err, err); + s->err =3D err; } =20 /* diff --git a/tests/check-qjson.c b/tests/check-qjson.c index fbb607c227..30b1b037d3 100644 --- a/tests/check-qjson.c +++ b/tests/check-qjson.c @@ -1388,17 +1388,13 @@ static void multiple_values(void) Error *err =3D NULL; QObject *obj; =20 - /* BUG this leaks the syntax tree for "false" */ obj =3D qobject_from_json("false true", &err); - g_assert(qbool_get_bool(qobject_to(QBool, obj))); - g_assert(!err); - qobject_unref(obj); + error_free_or_abort(&err); + g_assert(obj =3D=3D NULL); =20 - /* BUG simultaneously succeeds and fails */ obj =3D qobject_from_json("} true", &err); - g_assert(qbool_get_bool(qobject_to(QBool, obj))); error_free_or_abort(&err); - qobject_unref(obj); + g_assert(obj =3D=3D NULL); } =20 int main(int argc, char **argv) --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533731491296706.700737941265; Wed, 8 Aug 2018 05:31:31 -0700 (PDT) Received: from localhost ([::1]:43338 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNca-0005fI-62 for importer@patchew.org; Wed, 08 Aug 2018 08:31:24 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52664) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBt-0006om-BI for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:54 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBl-000077-A6 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:49 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:51104 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBk-0008Vp-Ml for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:40 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id CB624402382F; Wed, 8 Aug 2018 12:03:39 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 9F65920180F8; Wed, 8 Aug 2018 12:03:39 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id F0D5111385D8; Wed, 8 Aug 2018 14:03:34 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:22 +0200 Message-Id: <20180808120334.10970-45-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:39 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:39 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 44/56] json: Fix latent parser aborts at end of input X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" json-parser.c carefully reports end of input like this: token =3D parser_context_pop_token(ctxt); if (token =3D=3D NULL) { parse_error(ctxt, NULL, "premature EOI"); goto out; } Except parser_context_pop_token() can't return null, it fails its assertion instead. Same for parser_context_peek_token(). Broken in commit 65c0f1e9558, and faithfully preserved in commit 95385fe9ace. Only a latent bug, because the streamer throws away any input that could trigger it. Drop the assertions, so we can fix the streamer in the next commit. Signed-off-by: Markus Armbruster --- qobject/json-parser.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/qobject/json-parser.c b/qobject/json-parser.c index 350a9d267b..c2974d46b3 100644 --- a/qobject/json-parser.c +++ b/qobject/json-parser.c @@ -221,14 +221,12 @@ out: static JSONToken *parser_context_pop_token(JSONParserContext *ctxt) { g_free(ctxt->current); - assert(!g_queue_is_empty(ctxt->buf)); ctxt->current =3D g_queue_pop_head(ctxt->buf); return ctxt->current; } =20 static JSONToken *parser_context_peek_token(JSONParserContext *ctxt) { - assert(!g_queue_is_empty(ctxt->buf)); return g_queue_peek_head(ctxt->buf); } =20 --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533732186097882.4095944600498; Wed, 8 Aug 2018 05:43:06 -0700 (PDT) Received: from localhost ([::1]:43399 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNns-0007HL-S0 for importer@patchew.org; Wed, 08 Aug 2018 08:43:04 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52606) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBq-0006lR-Ju for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:56 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBl-00006w-8N for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:46 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:47732 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBk-0008Vn-N3 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:40 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 0634172629; Wed, 8 Aug 2018 12:03:40 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id B4AD71C71D; Wed, 8 Aug 2018 12:03:39 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 01C3711385D9; Wed, 8 Aug 2018 14:03:35 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:23 +0200 Message-Id: <20180808120334.10970-46-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 08 Aug 2018 12:03:40 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 08 Aug 2018 12:03:40 +0000 (UTC) for IP:'10.11.54.5' DOMAIN:'int-mx05.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 45/56] json: Fix streamer not to ignore trailing unterminated structures X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" json_message_process_token() accumulates tokens until it got the sequence of tokens that comprise a single JSON value (it counts curly braces and square brackets to decide). It feeds those token sequences to json_parser_parse(). If a non-empty sequence of tokens remains at the end of the parse, it's silently ignored. check-qjson.c cases unterminated_array(), unterminated_array_comma(), unterminated_dict(), unterminated_dict_comma() demonstrate this bug. Fix as follows. Introduce a JSON_END_OF_INPUT token. When the streamer receives it, it feeds the accumulated tokens to json_parser_parse(). Signed-off-by: Markus Armbruster --- include/qapi/qmp/json-lexer.h | 1 + qobject/json-lexer.c | 2 ++ qobject/json-streamer.c | 8 ++++++++ tests/check-qjson.c | 8 ++++---- 4 files changed, 15 insertions(+), 4 deletions(-) diff --git a/include/qapi/qmp/json-lexer.h b/include/qapi/qmp/json-lexer.h index 5586d12f26..8058695e40 100644 --- a/include/qapi/qmp/json-lexer.h +++ b/include/qapi/qmp/json-lexer.h @@ -30,6 +30,7 @@ typedef enum json_token_type { JSON_INTERPOL, JSON_SKIP, JSON_ERROR, + JSON_END_OF_INPUT } JSONTokenType; =20 typedef struct JSONLexer { diff --git a/qobject/json-lexer.c b/qobject/json-lexer.c index 733ce3f5ba..823db3aef8 100644 --- a/qobject/json-lexer.c +++ b/qobject/json-lexer.c @@ -369,6 +369,8 @@ void json_lexer_flush(JSONLexer *lexer) if (lexer->state !=3D lexer->start_state) { json_lexer_feed_char(lexer, 0, true); } + json_message_process_token(lexer, lexer->token, JSON_END_OF_INPUT, + lexer->x, lexer->y); } =20 void json_lexer_destroy(JSONLexer *lexer) diff --git a/qobject/json-streamer.c b/qobject/json-streamer.c index e372ecc895..674dfe6e85 100644 --- a/qobject/json-streamer.c +++ b/qobject/json-streamer.c @@ -60,6 +60,13 @@ void json_message_process_token(JSONLexer *lexer, GStrin= g *input, case JSON_ERROR: error_setg(&err, "JSON parse error, stray '%s'", input->str); goto out_emit; + case JSON_END_OF_INPUT: + if (g_queue_is_empty(parser->tokens)) { + return; + } + json =3D json_parser_parse(parser->tokens, parser->ap, &err); + parser->tokens =3D NULL; + goto out_emit; default: break; } @@ -137,6 +144,7 @@ void json_message_parser_feed(JSONMessageParser *parser, void json_message_parser_flush(JSONMessageParser *parser) { json_lexer_flush(&parser->lexer); + assert(g_queue_is_empty(parser->tokens)); } =20 void json_message_parser_destroy(JSONMessageParser *parser) diff --git a/tests/check-qjson.c b/tests/check-qjson.c index 30b1b037d3..833d220654 100644 --- a/tests/check-qjson.c +++ b/tests/check-qjson.c @@ -1305,7 +1305,7 @@ static void unterminated_array(void) { Error *err =3D NULL; QObject *obj =3D qobject_from_json("[32", &err); - g_assert(!err); /* BUG */ + error_free_or_abort(&err); g_assert(obj =3D=3D NULL); } =20 @@ -1313,7 +1313,7 @@ static void unterminated_array_comma(void) { Error *err =3D NULL; QObject *obj =3D qobject_from_json("[32,", &err); - g_assert(!err); /* BUG */ + error_free_or_abort(&err); g_assert(obj =3D=3D NULL); } =20 @@ -1329,7 +1329,7 @@ static void unterminated_dict(void) { Error *err =3D NULL; QObject *obj =3D qobject_from_json("{'abc':32", &err); - g_assert(!err); /* BUG */ + error_free_or_abort(&err); g_assert(obj =3D=3D NULL); } =20 @@ -1337,7 +1337,7 @@ static void unterminated_dict_comma(void) { Error *err =3D NULL; QObject *obj =3D qobject_from_json("{'abc':32,", &err); - g_assert(!err); /* BUG */ + error_free_or_abort(&err); g_assert(obj =3D=3D NULL); } =20 --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533730972888803.3179976033872; Wed, 8 Aug 2018 05:22:52 -0700 (PDT) Received: from localhost ([::1]:43288 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNU8-0006Ak-VU for importer@patchew.org; Wed, 08 Aug 2018 08:22:41 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52675) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBt-0006os-Uw for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:54 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBl-00007L-Be for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:49 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:45418 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBk-00005D-U6 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:41 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 5CF8C81663CE; Wed, 8 Aug 2018 12:03:40 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 34FFF20180F7; Wed, 8 Aug 2018 12:03:40 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 06E7911385DC; Wed, 8 Aug 2018 14:03:35 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:24 +0200 Message-Id: <20180808120334.10970-47-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 08 Aug 2018 12:03:40 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 08 Aug 2018 12:03:40 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 46/56] json: Assert json_parser_parse() consumes all tokens on success X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Signed-off-by: Markus Armbruster --- qobject/json-parser.c | 1 + 1 file changed, 1 insertion(+) diff --git a/qobject/json-parser.c b/qobject/json-parser.c index c2974d46b3..208dffc96c 100644 --- a/qobject/json-parser.c +++ b/qobject/json-parser.c @@ -539,6 +539,7 @@ QObject *json_parser_parse(GQueue *tokens, va_list *ap,= Error **errp) QObject *result; =20 result =3D parse_value(&ctxt, ap); + assert(ctxt.err || g_queue_is_empty(ctxt.buf)); =20 error_propagate(errp, ctxt.err); =20 --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533733275088692.1727940854236; Wed, 8 Aug 2018 06:01:15 -0700 (PDT) Received: from localhost ([::1]:43579 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnO5R-0006Ow-VI for importer@patchew.org; Wed, 08 Aug 2018 09:01:14 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52724) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBv-0006oz-3x for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:04:00 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBl-00008e-Tj for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:51 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:51758 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBl-000064-A7 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:41 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 9971F40241CB; Wed, 8 Aug 2018 12:03:40 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 520B2104083A; Wed, 8 Aug 2018 12:03:40 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 123941138546; Wed, 8 Aug 2018 14:03:35 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:25 +0200 Message-Id: <20180808120334.10970-48-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.7]); Wed, 08 Aug 2018 12:03:40 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.7]); Wed, 08 Aug 2018 12:03:40 +0000 (UTC) for IP:'10.11.54.3' DOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 47/56] qjson: Have qobject_from_json() & friends reject empty and blank X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The last case where qobject_from_json() & friends return null without setting an error is empty or blank input. Callers: * block.c's parse_json_protocol() reports "Could not parse the JSON options". It's marked as a work-around, because it also covered actual bugs, but they got fixed in the previous few commits. * qobject_input_visitor_new_str() reports "JSON parse error". Also marked as work-around. The recent fixes have made this unreachable, because it currently gets called only for input starting with '{'. * check-qjson.c's empty_input() and blank_input() demonstrate the behavior. * The other callers are not affected since they only pass input with exactly one JSON value or, in the case of negative tests, one error. Fail with "Expecting a JSON value" instead of returning null, and simplify callers. Signed-off-by: Markus Armbruster --- block.c | 5 ----- qapi/qobject-input-visitor.c | 5 ----- qobject/qjson.c | 4 ++++ tests/check-qjson.c | 12 ++++++++++-- 4 files changed, 14 insertions(+), 12 deletions(-) diff --git a/block.c b/block.c index 39f373e035..b837684e3c 100644 --- a/block.c +++ b/block.c @@ -1478,11 +1478,6 @@ static QDict *parse_json_filename(const char *filena= me, Error **errp) =20 options_obj =3D qobject_from_json(filename, errp); if (!options_obj) { - /* Work around qobject_from_json() lossage TODO fix that */ - if (errp && !*errp) { - error_setg(errp, "Could not parse the JSON options"); - return NULL; - } error_prepend(errp, "Could not parse the JSON options: "); return NULL; } diff --git a/qapi/qobject-input-visitor.c b/qapi/qobject-input-visitor.c index da57f4cc24..3e88b27f9e 100644 --- a/qapi/qobject-input-visitor.c +++ b/qapi/qobject-input-visitor.c @@ -725,11 +725,6 @@ Visitor *qobject_input_visitor_new_str(const char *str, if (is_json) { obj =3D qobject_from_json(str, errp); if (!obj) { - /* Work around qobject_from_json() lossage TODO fix that */ - if (errp && !*errp) { - error_setg(errp, "JSON parse error"); - return NULL; - } return NULL; } args =3D qobject_to(QDict, obj); diff --git a/qobject/qjson.c b/qobject/qjson.c index 7f69036487..b9ccae2c2a 100644 --- a/qobject/qjson.c +++ b/qobject/qjson.c @@ -70,6 +70,10 @@ static QObject *qobject_from_jsonv(const char *string, v= a_list *ap, json_message_parser_flush(&state.parser); json_message_parser_destroy(&state.parser); =20 + if (!state.result && !state.err) { + error_setg(&state.err, "Expecting a JSON value"); + } + error_propagate(errp, state.err); return state.result; } diff --git a/tests/check-qjson.c b/tests/check-qjson.c index 833d220654..49490f678e 100644 --- a/tests/check-qjson.c +++ b/tests/check-qjson.c @@ -1240,13 +1240,21 @@ static void simple_interpolation(void) =20 static void empty_input(void) { - QObject *obj =3D qobject_from_json("", &error_abort); + Error *err =3D NULL; + QObject *obj; + + obj =3D qobject_from_json("", &err); + error_free_or_abort(&err); g_assert(obj =3D=3D NULL); } =20 static void blank_input(void) { - QObject *obj =3D qobject_from_json("\n ", &error_abort); + Error *err =3D NULL; + QObject *obj; + + obj =3D qobject_from_json("\n ", &err); + error_free_or_abort(&err); g_assert(obj =3D=3D NULL); } =20 --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533732356519960.1599890315997; Wed, 8 Aug 2018 05:45:56 -0700 (PDT) Received: from localhost ([::1]:43417 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNqZ-0001Jb-DA for importer@patchew.org; Wed, 08 Aug 2018 08:45:51 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52671) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBt-0006oq-Ur for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:58 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBl-00007h-LL for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:49 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:45422 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBl-00005f-77 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:41 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 7BFEE81663F5; Wed, 8 Aug 2018 12:03:40 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 526BA111AF00; Wed, 8 Aug 2018 12:03:40 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 197F1113854F; Wed, 8 Aug 2018 14:03:35 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:26 +0200 Message-Id: <20180808120334.10970-49-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 08 Aug 2018 12:03:40 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 08 Aug 2018 12:03:40 +0000 (UTC) for IP:'10.11.54.3' DOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 48/56] json: Enforce token count and size limits more tightly X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Token count and size limits exist to guard against excessive heap usage. We check them only after we created the token on the heap. That's assigning a cowboy to the barn to lasso the horse after it has bolted. Close the barn door instead: check before we create the token. Signed-off-by: Markus Armbruster --- qobject/json-streamer.c | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/qobject/json-streamer.c b/qobject/json-streamer.c index 674dfe6e85..810aae521f 100644 --- a/qobject/json-streamer.c +++ b/qobject/json-streamer.c @@ -20,7 +20,7 @@ =20 #define MAX_TOKEN_SIZE (64ULL << 20) #define MAX_TOKEN_COUNT (2ULL << 20) -#define MAX_NESTING (1ULL << 10) +#define MAX_NESTING (1 << 10) =20 static void json_message_free_token(void *token, void *opaque) { @@ -71,6 +71,23 @@ void json_message_process_token(JSONLexer *lexer, GStrin= g *input, break; } =20 + /* + * Security consideration, we limit total memory allocated per object + * and the maximum recursion depth that a message can force. + */ + if (parser->token_size + input->len + 1 > MAX_TOKEN_SIZE) { + error_setg(&err, "JSON token size limit exceeded"); + goto out_emit; + } + if (g_queue_get_length(parser->tokens) + 1 > MAX_TOKEN_COUNT) { + error_setg(&err, "JSON token count limit exceeded"); + goto out_emit; + } + if (parser->bracket_count + parser->brace_count > MAX_NESTING) { + error_setg(&err, "JSON nesting depth limit exceeded"); + goto out_emit; + } + token =3D g_malloc(sizeof(JSONToken) + input->len + 1); token->type =3D type; memcpy(token->str, input->str, input->len); @@ -91,23 +108,6 @@ void json_message_process_token(JSONLexer *lexer, GStri= ng *input, goto out_emit; } =20 - /* - * Security consideration, we limit total memory allocated per object - * and the maximum recursion depth that a message can force. - */ - if (parser->token_size > MAX_TOKEN_SIZE) { - error_setg(&err, "JSON token size limit exceeded"); - goto out_emit; - } - if (g_queue_get_length(parser->tokens) > MAX_TOKEN_COUNT) { - error_setg(&err, "JSON token count limit exceeded"); - goto out_emit; - } - if (parser->bracket_count + parser->brace_count > MAX_NESTING) { - error_setg(&err, "JSON nesting depth limit exceeded"); - goto out_emit; - } - return; =20 out_emit: --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533731906310428.09718251818197; Wed, 8 Aug 2018 05:38:26 -0700 (PDT) Received: from localhost ([::1]:43372 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNjN-0002zE-52 for importer@patchew.org; Wed, 08 Aug 2018 08:38:25 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52683) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBu-0006ov-1I for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:55 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBl-00007y-MU for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:49 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:51752 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBl-00005k-7Z for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:41 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 7FA1E40241C7; Wed, 8 Aug 2018 12:03:40 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 575A01040840; Wed, 8 Aug 2018 12:03:40 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 1FA511138555; Wed, 8 Aug 2018 14:03:35 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:27 +0200 Message-Id: <20180808120334.10970-50-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.7]); Wed, 08 Aug 2018 12:03:40 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.7]); Wed, 08 Aug 2018 12:03:40 +0000 (UTC) for IP:'10.11.54.3' DOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 49/56] json: Streamline json_message_process_token() X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Signed-off-by: Markus Armbruster --- qobject/json-streamer.c | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/qobject/json-streamer.c b/qobject/json-streamer.c index 810aae521f..954bf9d468 100644 --- a/qobject/json-streamer.c +++ b/qobject/json-streamer.c @@ -99,16 +99,13 @@ void json_message_process_token(JSONLexer *lexer, GStri= ng *input, =20 g_queue_push_tail(parser->tokens, token); =20 - if (parser->brace_count < 0 || - parser->bracket_count < 0 || - (parser->brace_count =3D=3D 0 && - parser->bracket_count =3D=3D 0)) { - json =3D json_parser_parse(parser->tokens, parser->ap, &err); - parser->tokens =3D NULL; - goto out_emit; + if ((parser->brace_count > 0 || parser->bracket_count > 0) + && parser->bracket_count >=3D 0 && parser->bracket_count >=3D 0) { + return; } =20 - return; + json =3D json_parser_parse(parser->tokens, parser->ap, &err); + parser->tokens =3D NULL; =20 out_emit: parser->brace_count =3D 0; --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533732527587292.3139144264736; Wed, 8 Aug 2018 05:48:47 -0700 (PDT) Received: from localhost ([::1]:43458 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNtO-0003ga-B1 for importer@patchew.org; Wed, 08 Aug 2018 08:48:46 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52751) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBw-0006pl-0N for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:58 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBm-00008v-0N for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:51 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:37300 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBl-00006J-Es for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:41 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id B32DB40216F6; Wed, 8 Aug 2018 12:03:40 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 5889B1C730; Wed, 8 Aug 2018 12:03:40 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 25C281138556; Wed, 8 Aug 2018 14:03:35 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:28 +0200 Message-Id: <20180808120334.10970-51-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Wed, 08 Aug 2018 12:03:40 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Wed, 08 Aug 2018 12:03:40 +0000 (UTC) for IP:'10.11.54.5' DOMAIN:'int-mx05.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 50/56] json: Unbox tokens queue in JSONMessageParser X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Signed-off-by: Markus Armbruster --- include/qapi/qmp/json-streamer.h | 2 +- qobject/json-parser.c | 1 - qobject/json-streamer.c | 30 +++++++++++------------------- 3 files changed, 12 insertions(+), 21 deletions(-) diff --git a/include/qapi/qmp/json-streamer.h b/include/qapi/qmp/json-strea= mer.h index e162fd01da..d1d7fe2595 100644 --- a/include/qapi/qmp/json-streamer.h +++ b/include/qapi/qmp/json-streamer.h @@ -31,7 +31,7 @@ typedef struct JSONMessageParser JSONLexer lexer; int brace_count; int bracket_count; - GQueue *tokens; + GQueue tokens; uint64_t token_size; } JSONMessageParser; =20 diff --git a/qobject/json-parser.c b/qobject/json-parser.c index 208dffc96c..3623567160 100644 --- a/qobject/json-parser.c +++ b/qobject/json-parser.c @@ -547,7 +547,6 @@ QObject *json_parser_parse(GQueue *tokens, va_list *ap,= Error **errp) parser_context_pop_token(&ctxt); } g_free(ctxt.current); - g_queue_free(ctxt.buf); =20 return result; } diff --git a/qobject/json-streamer.c b/qobject/json-streamer.c index 954bf9d468..9210281a65 100644 --- a/qobject/json-streamer.c +++ b/qobject/json-streamer.c @@ -22,17 +22,12 @@ #define MAX_TOKEN_COUNT (2ULL << 20) #define MAX_NESTING (1 << 10) =20 -static void json_message_free_token(void *token, void *opaque) -{ - g_free(token); -} - static void json_message_free_tokens(JSONMessageParser *parser) { - if (parser->tokens) { - g_queue_foreach(parser->tokens, json_message_free_token, NULL); - g_queue_free(parser->tokens); - parser->tokens =3D NULL; + JSONToken *token; + + while ((token =3D g_queue_pop_head(&parser->tokens))) { + g_free(token); } } =20 @@ -61,11 +56,10 @@ void json_message_process_token(JSONLexer *lexer, GStri= ng *input, error_setg(&err, "JSON parse error, stray '%s'", input->str); goto out_emit; case JSON_END_OF_INPUT: - if (g_queue_is_empty(parser->tokens)) { + if (g_queue_is_empty(&parser->tokens)) { return; } - json =3D json_parser_parse(parser->tokens, parser->ap, &err); - parser->tokens =3D NULL; + json =3D json_parser_parse(&parser->tokens, parser->ap, &err); goto out_emit; default: break; @@ -79,7 +73,7 @@ void json_message_process_token(JSONLexer *lexer, GString= *input, error_setg(&err, "JSON token size limit exceeded"); goto out_emit; } - if (g_queue_get_length(parser->tokens) + 1 > MAX_TOKEN_COUNT) { + if (g_queue_get_length(&parser->tokens) + 1 > MAX_TOKEN_COUNT) { error_setg(&err, "JSON token count limit exceeded"); goto out_emit; } @@ -97,21 +91,19 @@ void json_message_process_token(JSONLexer *lexer, GStri= ng *input, =20 parser->token_size +=3D input->len; =20 - g_queue_push_tail(parser->tokens, token); + g_queue_push_tail(&parser->tokens, token); =20 if ((parser->brace_count > 0 || parser->bracket_count > 0) && parser->bracket_count >=3D 0 && parser->bracket_count >=3D 0) { return; } =20 - json =3D json_parser_parse(parser->tokens, parser->ap, &err); - parser->tokens =3D NULL; + json =3D json_parser_parse(&parser->tokens, parser->ap, &err); =20 out_emit: parser->brace_count =3D 0; parser->bracket_count =3D 0; json_message_free_tokens(parser); - parser->tokens =3D g_queue_new(); parser->token_size =3D 0; parser->emit(parser->opaque, json, err); } @@ -126,7 +118,7 @@ void json_message_parser_init(JSONMessageParser *parser, parser->ap =3D ap; parser->brace_count =3D 0; parser->bracket_count =3D 0; - parser->tokens =3D g_queue_new(); + g_queue_init(&parser->tokens); parser->token_size =3D 0; =20 json_lexer_init(&parser->lexer, !!ap); @@ -141,7 +133,7 @@ void json_message_parser_feed(JSONMessageParser *parser, void json_message_parser_flush(JSONMessageParser *parser) { json_lexer_flush(&parser->lexer); - assert(g_queue_is_empty(parser->tokens)); + assert(g_queue_is_empty(&parser->tokens)); } =20 void json_message_parser_destroy(JSONMessageParser *parser) --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533731659471350.0099506893399; Wed, 8 Aug 2018 05:34:19 -0700 (PDT) Received: from localhost ([::1]:43349 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNfJ-0007np-Lb for importer@patchew.org; Wed, 08 Aug 2018 08:34:13 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52672) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBt-0006or-Up for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:55 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBl-00008O-Ql for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:49 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:43254 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBl-00006C-CN for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:41 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id A01C977742; Wed, 8 Aug 2018 12:03:40 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 6D4E62166BA8; Wed, 8 Aug 2018 12:03:40 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 2E9871138557; Wed, 8 Aug 2018 14:03:35 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:29 +0200 Message-Id: <20180808120334.10970-52-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Wed, 08 Aug 2018 12:03:40 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Wed, 08 Aug 2018 12:03:40 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 51/56] json: Eliminate lexer state IN_ERROR and pseudo-token JSON_MIN X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Signed-off-by: Markus Armbruster --- include/qapi/qmp/json-lexer.h | 10 ++++------ qobject/json-lexer.c | 18 ++++++++---------- 2 files changed, 12 insertions(+), 16 deletions(-) diff --git a/include/qapi/qmp/json-lexer.h b/include/qapi/qmp/json-lexer.h index 8058695e40..f3524de07a 100644 --- a/include/qapi/qmp/json-lexer.h +++ b/include/qapi/qmp/json-lexer.h @@ -14,10 +14,9 @@ #ifndef QEMU_JSON_LEXER_H #define QEMU_JSON_LEXER_H =20 - -typedef enum json_token_type { - JSON_MIN =3D 100, - JSON_LCURLY =3D JSON_MIN, +typedef enum { + JSON_ERROR =3D 0, /* must be zero */ + JSON_LCURLY, JSON_RCURLY, JSON_LSQUARE, JSON_RSQUARE, @@ -29,8 +28,7 @@ typedef enum json_token_type { JSON_STRING, JSON_INTERPOL, JSON_SKIP, - JSON_ERROR, - JSON_END_OF_INPUT + JSON_END_OF_INPUT /* must be last */ } JSONTokenType; =20 typedef struct JSONLexer { diff --git a/qobject/json-lexer.c b/qobject/json-lexer.c index 823db3aef8..0332f9dbe1 100644 --- a/qobject/json-lexer.c +++ b/qobject/json-lexer.c @@ -101,8 +101,9 @@ * - Decoding and validating is left to the parser. */ =20 -enum json_lexer_state { - IN_ERROR =3D 0, /* must really be 0, see json_lexer[] */ +enum { + IN_START =3D JSON_END_OF_INPUT + 1, + IN_START_INTERPOL, IN_DQ_STRING_ESCAPE, IN_DQ_STRING, IN_SQ_STRING_ESCAPE, @@ -119,11 +120,9 @@ enum json_lexer_state { IN_KEYWORD, IN_INTERPOL, IN_WHITESPACE, - IN_START_INTERPOL, - IN_START, }; =20 -QEMU_BUILD_BUG_ON((int)JSON_MIN <=3D (int)IN_START); +QEMU_BUILD_BUG_ON(JSON_ERROR !=3D 0); /* json_lexer[] relies on this */ =20 #define TERMINAL(state) [0 ... 0x7F] =3D (state) =20 @@ -131,10 +130,10 @@ QEMU_BUILD_BUG_ON((int)JSON_MIN <=3D (int)IN_START); from OLD_STATE required lookahead. This happens whenever the table below uses the TERMINAL macro. */ #define TERMINAL_NEEDED_LOOKAHEAD(old_state, terminal) \ - (terminal !=3D IN_ERROR && json_lexer[(old_state)][0] =3D=3D (terminal= )) + (terminal !=3D JSON_ERROR && json_lexer[(old_state)][0] =3D=3D (termin= al)) =20 static const uint8_t json_lexer[][256] =3D { - /* Relies on default initialization to IN_ERROR! */ + /* Relies on default initialization to JSON_ERROR */ =20 /* double quote string */ [IN_DQ_STRING_ESCAPE] =3D { @@ -318,7 +317,7 @@ static void json_lexer_feed_char(JSONLexer *lexer, char= ch, bool flush) g_string_truncate(lexer->token, 0); new_state =3D lexer->start_state; break; - case IN_ERROR: + case JSON_ERROR: /* XXX: To avoid having previous bad input leaving the parser = in an * unresponsive state where we consume unpredictable amounts of * subsequent "good" input, percolate this error state up to t= he @@ -335,8 +334,7 @@ static void json_lexer_feed_char(JSONLexer *lexer, char= ch, bool flush) json_message_process_token(lexer, lexer->token, JSON_ERROR, lexer->x, lexer->y); g_string_truncate(lexer->token, 0); - new_state =3D lexer->start_state; - lexer->state =3D new_state; + lexer->state =3D lexer->start_state; return; default: break; --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533732236928665.8704744104826; Wed, 8 Aug 2018 05:43:56 -0700 (PDT) Received: from localhost ([::1]:43401 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNoh-00081i-R4 for importer@patchew.org; Wed, 08 Aug 2018 08:43:55 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52668) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBt-0006oo-Ta for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:57 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBl-000086-Qt for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:49 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:51120 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBl-000068-AA for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:41 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id A269640201BF; Wed, 8 Aug 2018 12:03:40 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 766B720180F7; Wed, 8 Aug 2018 12:03:40 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 31D22113855A; Wed, 8 Aug 2018 14:03:35 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:30 +0200 Message-Id: <20180808120334.10970-53-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:40 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 08 Aug 2018 12:03:40 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 52/56] json: Eliminate lexer state IN_WHITESPACE, pseudo-token JSON_SKIP X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Bonus: static json_lexer[] loses its unused elements. It shrinks from 8KiB to 4.75KiB for me. Signed-off-by: Markus Armbruster --- include/qapi/qmp/json-lexer.h | 1 - qobject/json-lexer.c | 30 +++++++++--------------------- 2 files changed, 9 insertions(+), 22 deletions(-) diff --git a/include/qapi/qmp/json-lexer.h b/include/qapi/qmp/json-lexer.h index f3524de07a..1a2dbbb717 100644 --- a/include/qapi/qmp/json-lexer.h +++ b/include/qapi/qmp/json-lexer.h @@ -27,7 +27,6 @@ typedef enum { JSON_KEYWORD, JSON_STRING, JSON_INTERPOL, - JSON_SKIP, JSON_END_OF_INPUT /* must be last */ } JSONTokenType; =20 diff --git a/qobject/json-lexer.c b/qobject/json-lexer.c index 0332f9dbe1..26ba45956d 100644 --- a/qobject/json-lexer.c +++ b/qobject/json-lexer.c @@ -119,7 +119,6 @@ enum { IN_SIGN, IN_KEYWORD, IN_INTERPOL, - IN_WHITESPACE, }; =20 QEMU_BUILD_BUG_ON(JSON_ERROR !=3D 0); /* json_lexer[] relies on this */ @@ -213,15 +212,6 @@ static const uint8_t json_lexer[][256] =3D { ['a' ... 'z'] =3D IN_KEYWORD, }, =20 - /* whitespace */ - [IN_WHITESPACE] =3D { - TERMINAL(JSON_SKIP), - [' '] =3D IN_WHITESPACE, - ['\t'] =3D IN_WHITESPACE, - ['\r'] =3D IN_WHITESPACE, - ['\n'] =3D IN_WHITESPACE, - }, - /* top level rule */ [IN_START] =3D { ['"'] =3D IN_DQ_STRING, @@ -236,10 +226,10 @@ static const uint8_t json_lexer[][256] =3D { [','] =3D JSON_COMMA, [':'] =3D JSON_COLON, ['a' ... 'z'] =3D IN_KEYWORD, - [' '] =3D IN_WHITESPACE, - ['\t'] =3D IN_WHITESPACE, - ['\r'] =3D IN_WHITESPACE, - ['\n'] =3D IN_WHITESPACE, + [' '] =3D IN_START, + ['\t'] =3D IN_START, + ['\r'] =3D IN_START, + ['\n'] =3D IN_START, }, =20 /* interpolation */ @@ -263,11 +253,11 @@ static const uint8_t json_lexer[][256] =3D { [','] =3D JSON_COMMA, [':'] =3D JSON_COLON, ['a' ... 'z'] =3D IN_KEYWORD, - [' '] =3D IN_WHITESPACE, - ['\t'] =3D IN_WHITESPACE, - ['\r'] =3D IN_WHITESPACE, - ['\n'] =3D IN_WHITESPACE, /* matches IN_START up to here */ + [' '] =3D IN_START_INTERPOL, + ['\t'] =3D IN_START_INTERPOL, + ['\r'] =3D IN_START_INTERPOL, + ['\n'] =3D IN_START_INTERPOL, ['%'] =3D IN_INTERPOL, }, }; @@ -294,7 +284,7 @@ static void json_lexer_feed_char(JSONLexer *lexer, char= ch, bool flush) assert(lexer->state <=3D ARRAY_SIZE(json_lexer)); new_state =3D json_lexer[lexer->state][(uint8_t)ch]; char_consumed =3D !TERMINAL_NEEDED_LOOKAHEAD(lexer->state, new_sta= te); - if (char_consumed && !flush) { + if (char_consumed && new_state !=3D lexer->start_state && !flush) { g_string_append_c(lexer->token, ch); } =20 @@ -312,8 +302,6 @@ static void json_lexer_feed_char(JSONLexer *lexer, char= ch, bool flush) case JSON_STRING: json_message_process_token(lexer, lexer->token, new_state, lexer->x, lexer->y); - /* fall through */ - case JSON_SKIP: g_string_truncate(lexer->token, 0); new_state =3D lexer->start_state; break; --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533731463303437.2626977935769; Wed, 8 Aug 2018 05:31:03 -0700 (PDT) Received: from localhost ([::1]:43337 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNcE-0005MU-42 for importer@patchew.org; Wed, 08 Aug 2018 08:31:02 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52678) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBt-0006ou-W6 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:55 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBm-00009F-5H for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:49 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:43268 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBl-00006c-K9 for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:41 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D15B6738F1; Wed, 8 Aug 2018 12:03:40 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 91C7721568A1; Wed, 8 Aug 2018 12:03:40 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 3658D113855F; Wed, 8 Aug 2018 14:03:35 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:31 +0200 Message-Id: <20180808120334.10970-54-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Wed, 08 Aug 2018 12:03:40 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Wed, 08 Aug 2018 12:03:40 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 53/56] json: Make JSONToken opaque outside json-parser.c X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Signed-off-by: Markus Armbruster --- include/qapi/qmp/json-parser.h | 4 ++++ include/qapi/qmp/json-streamer.h | 7 ------- qobject/json-parser.c | 19 +++++++++++++++++++ qobject/json-streamer.c | 8 +------- 4 files changed, 24 insertions(+), 14 deletions(-) diff --git a/include/qapi/qmp/json-parser.h b/include/qapi/qmp/json-parser.h index a34209db7a..21b23d7bec 100644 --- a/include/qapi/qmp/json-parser.h +++ b/include/qapi/qmp/json-parser.h @@ -15,7 +15,11 @@ #define QEMU_JSON_PARSER_H =20 #include "qemu-common.h" +#include "qapi/qmp/json-lexer.h" =20 +typedef struct JSONToken JSONToken; + +JSONToken *json_token(JSONTokenType type, int x, int y, GString *tokstr); QObject *json_parser_parse(GQueue *tokens, va_list *ap, Error **errp); =20 #endif diff --git a/include/qapi/qmp/json-streamer.h b/include/qapi/qmp/json-strea= mer.h index d1d7fe2595..29950ac37c 100644 --- a/include/qapi/qmp/json-streamer.h +++ b/include/qapi/qmp/json-streamer.h @@ -16,13 +16,6 @@ =20 #include "qapi/qmp/json-lexer.h" =20 -typedef struct JSONToken { - int type; - int x; - int y; - char str[]; -} JSONToken; - typedef struct JSONMessageParser { void (*emit)(void *opaque, QObject *json, Error *err); diff --git a/qobject/json-parser.c b/qobject/json-parser.c index 3623567160..d8f9df2fd3 100644 --- a/qobject/json-parser.c +++ b/qobject/json-parser.c @@ -26,6 +26,13 @@ #include "qapi/qmp/json-lexer.h" #include "qapi/qmp/json-streamer.h" =20 +struct JSONToken { + JSONTokenType type; + int x; + int y; + char str[]; +}; + typedef struct JSONParserContext { Error *err; @@ -533,6 +540,18 @@ static QObject *parse_value(JSONParserContext *ctxt, v= a_list *ap) } } =20 +JSONToken *json_token(JSONTokenType type, int x, int y, GString *tokstr) +{ + JSONToken *token =3D g_malloc(sizeof(JSONToken) + tokstr->len + 1); + + token->type =3D type; + memcpy(token->str, tokstr->str, tokstr->len); + token->str[tokstr->len] =3D 0; + token->x =3D x; + token->y =3D y; + return token; +} + QObject *json_parser_parse(GQueue *tokens, va_list *ap, Error **errp) { JSONParserContext ctxt =3D { .buf =3D tokens }; diff --git a/qobject/json-streamer.c b/qobject/json-streamer.c index 9210281a65..467bc29413 100644 --- a/qobject/json-streamer.c +++ b/qobject/json-streamer.c @@ -82,13 +82,7 @@ void json_message_process_token(JSONLexer *lexer, GStrin= g *input, goto out_emit; } =20 - token =3D g_malloc(sizeof(JSONToken) + input->len + 1); - token->type =3D type; - memcpy(token->str, input->str, input->len); - token->str[input->len] =3D 0; - token->x =3D x; - token->y =3D y; - + token =3D json_token(type, x, y, input); parser->token_size +=3D input->len; =20 g_queue_push_tail(&parser->tokens, token); --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533731306979830.3605987872705; Wed, 8 Aug 2018 05:28:26 -0700 (PDT) Received: from localhost ([::1]:43319 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNZY-0002hl-My for importer@patchew.org; Wed, 08 Aug 2018 08:28:16 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52676) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBt-0006ot-VD for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:55 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBm-000090-1k for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:49 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:43264 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBl-00006W-Gn for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:41 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id C55B4738F0; Wed, 8 Aug 2018 12:03:40 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 9AC45101B733; Wed, 8 Aug 2018 12:03:40 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 3A29711384AF; Wed, 8 Aug 2018 14:03:35 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:32 +0200 Message-Id: <20180808120334.10970-55-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Wed, 08 Aug 2018 12:03:40 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Wed, 08 Aug 2018 12:03:40 +0000 (UTC) for IP:'10.11.54.3' DOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 54/56] qobject: Drop superfluous includes of qemu-common.h X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Signed-off-by: Markus Armbruster --- include/qapi/qmp/json-parser.h | 1 - qobject/json-lexer.c | 1 - qobject/json-streamer.c | 1 - qobject/qbool.c | 1 - qobject/qlist.c | 1 - qobject/qnull.c | 1 - qobject/qnum.c | 1 - qobject/qobject.c | 1 - qobject/qstring.c | 1 - 9 files changed, 9 deletions(-) diff --git a/include/qapi/qmp/json-parser.h b/include/qapi/qmp/json-parser.h index 21b23d7bec..55f75954c3 100644 --- a/include/qapi/qmp/json-parser.h +++ b/include/qapi/qmp/json-parser.h @@ -14,7 +14,6 @@ #ifndef QEMU_JSON_PARSER_H #define QEMU_JSON_PARSER_H =20 -#include "qemu-common.h" #include "qapi/qmp/json-lexer.h" =20 typedef struct JSONToken JSONToken; diff --git a/qobject/json-lexer.c b/qobject/json-lexer.c index 26ba45956d..dc21eb52cf 100644 --- a/qobject/json-lexer.c +++ b/qobject/json-lexer.c @@ -12,7 +12,6 @@ */ =20 #include "qemu/osdep.h" -#include "qemu-common.h" #include "qapi/qmp/json-lexer.h" #include "qapi/qmp/json-streamer.h" =20 diff --git a/qobject/json-streamer.c b/qobject/json-streamer.c index 467bc29413..da53e770e9 100644 --- a/qobject/json-streamer.c +++ b/qobject/json-streamer.c @@ -12,7 +12,6 @@ */ =20 #include "qemu/osdep.h" -#include "qemu-common.h" #include "qapi/error.h" #include "qapi/qmp/json-lexer.h" #include "qapi/qmp/json-parser.h" diff --git a/qobject/qbool.c b/qobject/qbool.c index b58249925c..06dfc43498 100644 --- a/qobject/qbool.c +++ b/qobject/qbool.c @@ -13,7 +13,6 @@ =20 #include "qemu/osdep.h" #include "qapi/qmp/qbool.h" -#include "qemu-common.h" =20 /** * qbool_from_bool(): Create a new QBool from a bool diff --git a/qobject/qlist.c b/qobject/qlist.c index 37c1c167f1..b3274af88b 100644 --- a/qobject/qlist.c +++ b/qobject/qlist.c @@ -17,7 +17,6 @@ #include "qapi/qmp/qnum.h" #include "qapi/qmp/qstring.h" #include "qemu/queue.h" -#include "qemu-common.h" =20 /** * qlist_new(): Create a new QList diff --git a/qobject/qnull.c b/qobject/qnull.c index f6f55f11ea..00870a1824 100644 --- a/qobject/qnull.c +++ b/qobject/qnull.c @@ -11,7 +11,6 @@ */ =20 #include "qemu/osdep.h" -#include "qemu-common.h" #include "qapi/qmp/qnull.h" =20 QNull qnull_ =3D { diff --git a/qobject/qnum.c b/qobject/qnum.c index 1501c82832..7012fc57f2 100644 --- a/qobject/qnum.c +++ b/qobject/qnum.c @@ -14,7 +14,6 @@ =20 #include "qemu/osdep.h" #include "qapi/qmp/qnum.h" -#include "qemu-common.h" =20 /** * qnum_from_int(): Create a new QNum from an int64_t diff --git a/qobject/qobject.c b/qobject/qobject.c index cf4b7e229e..878dd76e79 100644 --- a/qobject/qobject.c +++ b/qobject/qobject.c @@ -8,7 +8,6 @@ */ =20 #include "qemu/osdep.h" -#include "qemu-common.h" #include "qapi/qmp/qbool.h" #include "qapi/qmp/qnull.h" #include "qapi/qmp/qnum.h" diff --git a/qobject/qstring.c b/qobject/qstring.c index 0f1510e792..1c6897df00 100644 --- a/qobject/qstring.c +++ b/qobject/qstring.c @@ -12,7 +12,6 @@ =20 #include "qemu/osdep.h" #include "qapi/qmp/qstring.h" -#include "qemu-common.h" =20 /** * qstring_new(): Create a new empty QString --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533733110435187.97354752177557; Wed, 8 Aug 2018 05:58:30 -0700 (PDT) Received: from localhost ([::1]:43552 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnO2e-0003q4-GK for importer@patchew.org; Wed, 08 Aug 2018 08:58:20 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52734) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBv-0006p0-Ek for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:04:00 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBm-00009U-9V for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:51 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:43276 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBl-00007D-RL for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:42 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 0BCC56883E; Wed, 8 Aug 2018 12:03:41 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id B68DF1040827; Wed, 8 Aug 2018 12:03:40 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 3E29711384BE; Wed, 8 Aug 2018 14:03:35 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:33 +0200 Message-Id: <20180808120334.10970-56-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Wed, 08 Aug 2018 12:03:41 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Wed, 08 Aug 2018 12:03:41 +0000 (UTC) for IP:'10.11.54.3' DOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 55/56] json: Clean up headers X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The JSON parser has three public headers, json-lexer.h, json-parser.h, json-streamer.h. They all contain stuff that is of no interest outside qobject/json-*.c. Collect the public interface in include/qapi/qmp/json-parser.h, and everything else in qobject/json-parser-int.h. Signed-off-by: Markus Armbruster --- include/qapi/qmp/json-parser.h | 36 ++++++++++++--- include/qapi/qmp/json-streamer.h | 46 ------------------- monitor.c | 2 +- qga/main.c | 2 +- qobject/json-lexer.c | 3 +- .../json-lexer.h =3D> qobject/json-parser-int.h | 26 ++++++----- qobject/json-parser.c | 4 +- qobject/json-streamer.c | 4 +- qobject/qjson.c | 2 +- tests/libqtest.c | 2 +- 10 files changed, 51 insertions(+), 76 deletions(-) delete mode 100644 include/qapi/qmp/json-streamer.h rename include/qapi/qmp/json-lexer.h =3D> qobject/json-parser-int.h (62%) diff --git a/include/qapi/qmp/json-parser.h b/include/qapi/qmp/json-parser.h index 55f75954c3..7345a9bd5c 100644 --- a/include/qapi/qmp/json-parser.h +++ b/include/qapi/qmp/json-parser.h @@ -1,5 +1,5 @@ /* - * JSON Parser=20 + * JSON Parser * * Copyright IBM, Corp. 2009 * @@ -11,14 +11,36 @@ * */ =20 -#ifndef QEMU_JSON_PARSER_H -#define QEMU_JSON_PARSER_H +#ifndef QAPI_QMP_JSON_PARSER_H +#define QAPI_QMP_JSON_PARSER_H =20 -#include "qapi/qmp/json-lexer.h" +typedef struct JSONLexer { + int start_state, state; + GString *token; + int x, y; +} JSONLexer; =20 -typedef struct JSONToken JSONToken; +typedef struct JSONMessageParser { + void (*emit)(void *opaque, QObject *json, Error *err); + void *opaque; + va_list *ap; + JSONLexer lexer; + int brace_count; + int bracket_count; + GQueue tokens; + uint64_t token_size; +} JSONMessageParser; =20 -JSONToken *json_token(JSONTokenType type, int x, int y, GString *tokstr); -QObject *json_parser_parse(GQueue *tokens, va_list *ap, Error **errp); +void json_message_parser_init(JSONMessageParser *parser, + void (*emit)(void *opaque, QObject *json, + Error *err), + void *opaque, va_list *ap); + +void json_message_parser_feed(JSONMessageParser *parser, + const char *buffer, size_t size); + +void json_message_parser_flush(JSONMessageParser *parser); + +void json_message_parser_destroy(JSONMessageParser *parser); =20 #endif diff --git a/include/qapi/qmp/json-streamer.h b/include/qapi/qmp/json-strea= mer.h deleted file mode 100644 index 29950ac37c..0000000000 --- a/include/qapi/qmp/json-streamer.h +++ /dev/null @@ -1,46 +0,0 @@ -/* - * JSON streaming support - * - * Copyright IBM, Corp. 2009 - * - * Authors: - * Anthony Liguori - * - * This work is licensed under the terms of the GNU LGPL, version 2.1 or l= ater. - * See the COPYING.LIB file in the top-level directory. - * - */ - -#ifndef QEMU_JSON_STREAMER_H -#define QEMU_JSON_STREAMER_H - -#include "qapi/qmp/json-lexer.h" - -typedef struct JSONMessageParser -{ - void (*emit)(void *opaque, QObject *json, Error *err); - void *opaque; - va_list *ap; - JSONLexer lexer; - int brace_count; - int bracket_count; - GQueue tokens; - uint64_t token_size; -} JSONMessageParser; - -void json_message_process_token(JSONLexer *lexer, GString *input, - JSONTokenType type, int x, int y); - -void json_message_parser_init(JSONMessageParser *parser, - void (*emit)(void *opaque, QObject *json, - Error *err), - void *opaque, va_list *ap); - -void json_message_parser_feed(JSONMessageParser *parser, - const char *buffer, size_t size); - -void json_message_parser_flush(JSONMessageParser *parser); - -void json_message_parser_destroy(JSONMessageParser *parser); - -#endif diff --git a/monitor.c b/monitor.c index dc0ed8df92..ff8960f857 100644 --- a/monitor.c +++ b/monitor.c @@ -58,7 +58,7 @@ #include "qapi/qmp/qnum.h" #include "qapi/qmp/qstring.h" #include "qapi/qmp/qjson.h" -#include "qapi/qmp/json-streamer.h" +#include "qapi/qmp/json-parser.h" #include "qapi/qmp/qlist.h" #include "qom/object_interfaces.h" #include "trace-root.h" diff --git a/qga/main.c b/qga/main.c index b74e1241ef..6d70242d05 100644 --- a/qga/main.c +++ b/qga/main.c @@ -18,7 +18,7 @@ #include #include #endif -#include "qapi/qmp/json-streamer.h" +#include "qapi/qmp/json-parser.h" #include "qapi/qmp/qdict.h" #include "qapi/qmp/qjson.h" #include "qapi/qmp/qstring.h" diff --git a/qobject/json-lexer.c b/qobject/json-lexer.c index dc21eb52cf..4d2b840929 100644 --- a/qobject/json-lexer.c +++ b/qobject/json-lexer.c @@ -12,8 +12,7 @@ */ =20 #include "qemu/osdep.h" -#include "qapi/qmp/json-lexer.h" -#include "qapi/qmp/json-streamer.h" +#include "json-parser-int.h" =20 #define MAX_TOKEN_SIZE (64ULL << 20) =20 diff --git a/include/qapi/qmp/json-lexer.h b/qobject/json-parser-int.h similarity index 62% rename from include/qapi/qmp/json-lexer.h rename to qobject/json-parser-int.h index 1a2dbbb717..442d17996a 100644 --- a/include/qapi/qmp/json-lexer.h +++ b/qobject/json-parser-int.h @@ -1,5 +1,5 @@ /* - * JSON lexer + * JSON Parser * * Copyright IBM, Corp. 2009 * @@ -11,8 +11,10 @@ * */ =20 -#ifndef QEMU_JSON_LEXER_H -#define QEMU_JSON_LEXER_H +#ifndef JSON_PARSER_INT_H +#define JSON_PARSER_INT_H + +#include "qapi/qmp/json-parser.h" =20 typedef enum { JSON_ERROR =3D 0, /* must be zero */ @@ -30,18 +32,20 @@ typedef enum { JSON_END_OF_INPUT /* must be last */ } JSONTokenType; =20 -typedef struct JSONLexer { - int start_state, state; - GString *token; - int x, y; -} JSONLexer; +typedef struct JSONToken JSONToken; =20 +/* json-lexer.c */ void json_lexer_init(JSONLexer *lexer, bool enable_interpolation); - void json_lexer_feed(JSONLexer *lexer, const char *buffer, size_t size); - void json_lexer_flush(JSONLexer *lexer); - void json_lexer_destroy(JSONLexer *lexer); =20 +/* json-streamer.c */ +void json_message_process_token(JSONLexer *lexer, GString *input, + JSONTokenType type, int x, int y); + +/* json-parser.c */ +JSONToken *json_token(JSONTokenType type, int x, int y, GString *tokstr); +QObject *json_parser_parse(GQueue *tokens, va_list *ap, Error **errp); + #endif diff --git a/qobject/json-parser.c b/qobject/json-parser.c index d8f9df2fd3..179b8a45d7 100644 --- a/qobject/json-parser.c +++ b/qobject/json-parser.c @@ -22,9 +22,7 @@ #include "qapi/qmp/qnull.h" #include "qapi/qmp/qnum.h" #include "qapi/qmp/qstring.h" -#include "qapi/qmp/json-parser.h" -#include "qapi/qmp/json-lexer.h" -#include "qapi/qmp/json-streamer.h" +#include "json-parser-int.h" =20 struct JSONToken { JSONTokenType type; diff --git a/qobject/json-streamer.c b/qobject/json-streamer.c index da53e770e9..47dd7ea576 100644 --- a/qobject/json-streamer.c +++ b/qobject/json-streamer.c @@ -13,9 +13,7 @@ =20 #include "qemu/osdep.h" #include "qapi/error.h" -#include "qapi/qmp/json-lexer.h" -#include "qapi/qmp/json-parser.h" -#include "qapi/qmp/json-streamer.h" +#include "json-parser-int.h" =20 #define MAX_TOKEN_SIZE (64ULL << 20) #define MAX_TOKEN_COUNT (2ULL << 20) diff --git a/qobject/qjson.c b/qobject/qjson.c index b9ccae2c2a..db36101f3b 100644 --- a/qobject/qjson.c +++ b/qobject/qjson.c @@ -13,7 +13,7 @@ =20 #include "qemu/osdep.h" #include "qapi/error.h" -#include "qapi/qmp/json-streamer.h" +#include "qapi/qmp/json-parser.h" #include "qapi/qmp/qjson.h" #include "qapi/qmp/qbool.h" #include "qapi/qmp/qdict.h" diff --git a/tests/libqtest.c b/tests/libqtest.c index 7ef8dd621f..19a39091a1 100644 --- a/tests/libqtest.c +++ b/tests/libqtest.c @@ -23,7 +23,7 @@ #include "libqtest.h" #include "qemu/cutils.h" #include "qapi/error.h" -#include "qapi/qmp/json-streamer.h" +#include "qapi/qmp/json-parser.h" #include "qapi/qmp/qdict.h" #include "qapi/qmp/qjson.h" #include "qapi/qmp/qlist.h" --=20 2.17.1 From nobody Wed Nov 5 07:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1533732169904875.0041735357864; Wed, 8 Aug 2018 05:42:49 -0700 (PDT) Received: from localhost ([::1]:43398 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNnZ-0006xh-NF for importer@patchew.org; Wed, 08 Aug 2018 08:42:45 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52772) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnNBx-0006qq-2c for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:58 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fnNBm-00009M-8H for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:53 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:37328 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fnNBl-00007T-Qw for qemu-devel@nongnu.org; Wed, 08 Aug 2018 08:03:42 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 2E6A440216E6; Wed, 8 Aug 2018 12:03:41 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-62.ams2.redhat.com [10.36.117.62]) by smtp.corp.redhat.com (Postfix) with ESMTPS id BBA0A2156711; Wed, 8 Aug 2018 12:03:40 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 41F7211384BF; Wed, 8 Aug 2018 14:03:35 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 14:03:34 +0200 Message-Id: <20180808120334.10970-57-armbru@redhat.com> In-Reply-To: <20180808120334.10970-1-armbru@redhat.com> References: <20180808120334.10970-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Wed, 08 Aug 2018 12:03:41 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Wed, 08 Aug 2018 12:03:41 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 56/56] docs/interop/qmp-spec: How to force known good parser state X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, mdroth@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Section "QGA Synchronization" specifies that sending "a raw 0xFF sentinel byte" makes the server "reset its state and discard all pending data prior to the sentinel." What actually happens there is a lexical error, which will produce one ore more error responses. Moreover, it's not specific to QGA. Create new section "Forcing the JSON parser into known-good state" to document the technique properly. Rewrite section "QGA Synchronization" to document just the other direction, i.e. command guest-sync-delimited. Section "Protocol Specification" mentions "synchronization bytes (documented below)". Delete that. While there, fix it not to claim '"Server" is QEMU itself', but '"Server" is either QEMU or the QEMU Guest Agent'. Signed-off-by: Markus Armbruster --- docs/interop/qmp-spec.txt | 37 ++++++++++++++++++++++++------------- 1 file changed, 24 insertions(+), 13 deletions(-) diff --git a/docs/interop/qmp-spec.txt b/docs/interop/qmp-spec.txt index 1566b8ae5e..d4a42fe2cc 100644 --- a/docs/interop/qmp-spec.txt +++ b/docs/interop/qmp-spec.txt @@ -20,9 +20,9 @@ operating system. 2. Protocol Specification =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D =20 -This section details the protocol format. For the purpose of this document -"Client" is any application which is using QMP to communicate with QEMU and -"Server" is QEMU itself. +This section details the protocol format. For the purpose of this +document, "Server" is either QEMU or the QEMU Guest Agent, and +"Client" is any application communicating with it via QMP. =20 JSON data structures, when mentioned in this document, are always in the following format: @@ -34,9 +34,8 @@ by the JSON standard: =20 http://www.ietf.org/rfc/rfc7159.txt =20 -The protocol is always encoded in UTF-8 except for synchronization -bytes (documented below); although thanks to json-string escape -sequences, the server will reply using only the strict ASCII subset. +The sever expects its input to be encoded in UTF-8, and sends its +output encoded in ASCII. =20 For convenience, json-object members mentioned in this document will be in a certain order. However, in real protocol usage they can be in @@ -215,16 +214,28 @@ Some events are rate-limited to at most one per secon= d. If additional dropped, and the last one is delayed. "Similar" normally means same event type. See qmp-events.txt for details. =20 -2.6 QGA Synchronization +2.6 Forcing the JSON parser into known-good state +------------------------------------------------- + +Incomplete or invalid input can leave the server's JSON parser in a +state where it can't parse additional commands. To get it back into +known-good state, the client should provoke a lexical error. + +The cleanest way to do that is sending an ASCII control character +other than '\t' (horizontal tab), '\r' (carriage return), and '\n' +(new line). + +Sadly, older versions of QEMU can fail to flag this as an error. If a +client needs to deal with them, it should send a 0xFF byte. + +2.7 QGA Synchronization ----------------------- =20 When using QGA, an additional synchronization feature is built into -the protocol. If the Client sends a raw 0xFF sentinel byte (not valid -JSON), then the Server will reset its state and discard all pending -data prior to the sentinel. Conversely, if the Client makes use of -the 'guest-sync-delimited' command, the Server will send a raw 0xFF -sentinel byte prior to its response, to aid the Client in discarding -any data prior to the sentinel. +the protocol. If the Client makes use of the 'guest-sync-delimited' +command, the Server will send a raw 0xFF sentinel byte prior to its +response, to aid the Client in discarding any data prior to the +sentinel. =20 =20 3. QMP Examples --=20 2.17.1