From nobody Tue Feb 10 20:48:03 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1697470113; cv=none; d=zohomail.com; s=zohoarc; b=crUzLnloit+Mud/7rPhvzyRZh/coaP2o3opHcDATgnMxbJKqQb1O7YvrB5SvLSRAuRvzhW4O9VNAtH1ZaghBPLi3WAdYF7TkBuZFpngzFH5dHX7xJInAM/qKKqRXQtxRpCu4AbiWl2HAEbmaGI7XqEODGALXzzlpdwSOW34CyNE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1697470113; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=4UGeHM+wuJlFvsAuVCpcyMS7Unx0q539CcUwF56Xqec=; b=fel0sY2WECkJbIy0Ic85tTodlKoRx/Q7vXOBeiokD8tSDOKQlc85jXan9iOvQqARkOsYewjTu8/v1ywOEpB32z7AICWurtTcyivKb3NSVRhqJRgmjB7OX4rtjOVq3LyB3bfWkEtWajmnqE41ggK0gTkKce/Pyu2iD+aNn913LxA= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1697470113343988.420252623303; Mon, 16 Oct 2023 08:28:33 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qsPVM-0001X9-Ak; Mon, 16 Oct 2023 11:27:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qsPVK-0001Wa-4o for qemu-devel@nongnu.org; Mon, 16 Oct 2023 11:27:38 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qsPV4-0003OL-Mk for qemu-devel@nongnu.org; Mon, 16 Oct 2023 11:27:37 -0400 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-173-3Jp_3TsrNiewVpUPY31ydw-1; Mon, 16 Oct 2023 11:27:18 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id F1B3C81B15E for ; Mon, 16 Oct 2023 15:27:17 +0000 (UTC) Received: from tapioca.redhat.com (unknown [10.45.225.170]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8B12B202B17E; Mon, 16 Oct 2023 15:27:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1697470042; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4UGeHM+wuJlFvsAuVCpcyMS7Unx0q539CcUwF56Xqec=; b=fp1PagbT3WS6zEBhyNQa72WD78isnsq4ZvuYCChgBqcuSiIUKVRrWsaEJcf2L52veD6RsT ZGu2gDD3jed5+aZVq4B4mL7Bzx9qH1fTQhVKRDlpZORIyDDyLAL9kJawoCE6OsdTwpxVle yFT4fIgS0aWTWLpEA2vdJIYGeujy+FQ= X-MC-Unique: 3Jp_3TsrNiewVpUPY31ydw-1 From: Victor Toso To: qemu-devel@nongnu.org Cc: Markus Armbruster , John Snow , =?UTF-8?q?Daniel=20P=20=2E=20Berrang=C3=A9?= , Andrea Bolognani Subject: [PATCH v2 06/11] qapi: golang: structs: Address 'null' members Date: Mon, 16 Oct 2023 17:26:59 +0200 Message-ID: <20231016152704.221611-7-victortoso@redhat.com> In-Reply-To: <20231016152704.221611-1-victortoso@redhat.com> References: <20231016152704.221611-1-victortoso@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=victortoso@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1697470115519100004 Content-Type: text/plain; charset="utf-8" Explaining why this is needed needs some context, so taking the example of StrOrNull alternate type and considering a simplified struct that has two fields: qapi: | { 'struct': 'MigrationExample', | 'data': { '*label': 'StrOrNull', | 'target': 'StrOrNull' } } We have a optional member 'label' which can have three JSON values: 1. A string: { "target": "a.host.com", "label": "happy" } 2. A null : { "target": "a.host.com", "label": null } 3. Absent : { "target": null} The member 'target' is not optional, hence it can't be absent. A Go struct that contains a optional type that can be JSON Null like 'label' in the example above, will need extra care when Marshaling and Unmarshaling from JSON. This patch handles this very specific case: - It implements the Marshaler interface for these structs to properly handle these values. - It adds the interface AbsentAlternate() and implement it for any Alternate that can be JSON Null Signed-off-by: Victor Toso --- scripts/qapi/golang.py | 243 ++++++++++++++++++++++++++++++++++++++--- 1 file changed, 228 insertions(+), 15 deletions(-) diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py index 73d0804d0a..6a7e37dd90 100644 --- a/scripts/qapi/golang.py +++ b/scripts/qapi/golang.py @@ -110,6 +110,26 @@ """ =20 =20 +TEMPLATE_STRUCT_WITH_NULLABLE_MARSHAL =3D """ +func (s {type_name}) MarshalJSON() ([]byte, error) {{ +\tm :=3D make(map[string]any) +{map_members}{map_special} +\treturn json.Marshal(&m) +}} + +func (s *{type_name}) UnmarshalJSON(data []byte) error {{ +\ttmp :=3D {struct}{{}} + +\tif err :=3D json.Unmarshal(data, &tmp); err !=3D nil {{ +\t\treturn err +\t}} + +{set_members}{set_special} +\treturn nil +}} +""" + + def gen_golang(schema: QAPISchema, output_dir: str, prefix: str) -> None: vis =3D QAPISchemaGenGolangVisitor(prefix) schema.visit(vis) @@ -256,21 +276,31 @@ def get_struct_field( self: QAPISchemaGenGolangVisitor, qapi_name: str, qapi_type_name: str, + within_nullable_struct: bool, is_optional: bool, is_variant: bool, -) -> dict[str:str]: +) -> Tuple[dict[str:str], bool]: =20 field =3D qapi_to_field_name(qapi_name) member_type =3D qapi_schema_type_to_go_type(qapi_type_name) + is_nullable =3D False =20 optional =3D "" if is_optional: - if member_type not in self.accept_null_types: + if member_type in self.accept_null_types: + is_nullable =3D True + else: optional =3D ",omitempty" =20 # Use pointer to type when field is optional isptr =3D "*" if is_optional and member_type[0] not in "*[" else "" =20 + if within_nullable_struct: + # Within a struct which has a field of type that can hold JSON NUL= L, + # we have to _not_ use a pointer, otherwise the Marshal methods are + # not called. + isptr =3D "" if member_type in self.accept_null_types else isptr + fieldtag =3D ( '`json:"-"`' if is_variant else f'`json:"{qapi_name}{optional}"`' ) @@ -279,32 +309,202 @@ def get_struct_field( "type": f"{isptr}{member_type}", "tag": f"{fieldtag}", } - return arg + return arg, is_nullable + + +# This helper is used whithin a struct that has members that accept JSON N= ULL. +def map_and_set( + is_nullable: bool, field: str, field_is_optional: bool, name: str +) -> Tuple[str, str]: + + mapstr =3D "" + setstr =3D "" + if is_nullable: + mapstr =3D f""" +\tif val, absent :=3D s.{field}.ToAnyOrAbsent(); !absent {{ +\t\tm["{name}"] =3D val +\t}} +""" + setstr +=3D f""" +\tif _, absent :=3D (&tmp.{field}).ToAnyOrAbsent(); !absent {{ +\t\ts.{field} =3D &tmp.{field} +\t}} +""" + elif field_is_optional: + mapstr =3D f""" +\tif s.{field} !=3D nil {{ +\t\tm["{name}"] =3D s.{field} +\t}} +""" + setstr =3D f"""\ts.{field} =3D tmp.{field}\n""" + else: + mapstr =3D f"""\tm["{name}"] =3D s.{field}\n""" + setstr =3D f"""\ts.{field} =3D tmp.{field}\n""" + + return mapstr, setstr + + +def recursive_base_nullable( + self: QAPISchemaGenGolangVisitor, base: Optional[QAPISchemaObjectType] +) -> Tuple[List[dict[str:str]], str, str, str, str]: + fields: List[dict[str:str]] =3D [] + map_members =3D "" + set_members =3D "" + map_special =3D "" + set_special =3D "" + + if not base: + return fields, map_members, set_members, map_special, set_special + + if base.base is not None: + embed_base =3D self.schema.lookup_entity(base.base.name) + ( + fields, + map_members, + set_members, + map_special, + set_special, + ) =3D recursive_base_nullable(self, embed_base) + + for member in base.local_members: + field, _ =3D get_struct_field( + self, member.name, member.type.name, True, member.optional, Fa= lse + ) + fields.append(field) + + member_type =3D qapi_schema_type_to_go_type(member.type.name) + nullable =3D member_type in self.accept_null_types + field_name =3D qapi_to_field_name(member.name) + tomap, toset =3D map_and_set( + nullable, field_name, member.optional, member.name + ) + if nullable: + map_special +=3D tomap + set_special +=3D toset + else: + map_members +=3D tomap + set_members +=3D toset + + return fields, map_members, set_members, map_special, set_special + + +# Helper function. This is executed when the QAPI schema has members +# that could accept JSON NULL (e.g: StrOrNull in QEMU"s QAPI schema). +# This struct will need to be extended with Marshal/Unmarshal methods to +# properly handle such atypical members. +# +# Only the Marshallaing methods are generated but we do need to iterate ov= er +# all the members to properly set/check them in those methods. +def struct_with_nullable_generate_marshal( + self: QAPISchemaGenGolangVisitor, + name: str, + base: Optional[QAPISchemaObjectType], + members: List[QAPISchemaObjectTypeMember], + variants: Optional[QAPISchemaVariants], +) -> str: + + ( + fields, + map_members, + set_members, + map_special, + set_special, + ) =3D recursive_base_nullable(self, base) + + if members: + for member in members: + field, _ =3D get_struct_field( + self, + member.name, + member.type.name, + True, + member.optional, + False, + ) + fields.append(field) + + member_type =3D qapi_schema_type_to_go_type(member.type.name) + nullable =3D member_type in self.accept_null_types + tomap, toset =3D map_and_set( + nullable, + qapi_to_field_name(member.name), + member.optional, + member.name, + ) + if nullable: + map_special +=3D tomap + set_special +=3D toset + else: + map_members +=3D tomap + set_members +=3D toset + + if variants: + for variant in variants.variants: + if variant.type.is_implicit(): + continue + + field, _ =3D get_struct_field( + self, + variant.name, + variant.type.name, + True, + variant.optional, + True, + ) + fields.append(field) + + member_type =3D qapi_schema_type_to_go_type(variant.type.name) + nullable =3D member_type in self.accept_null_types + tomap, toset =3D map_and_set( + nullable, + qapi_to_field_name(variant.name), + variant.optional, + variant.name, + ) + if nullable: + map_special +=3D tomap + set_special +=3D toset + else: + map_members +=3D tomap + set_members +=3D toset + + type_name =3D qapi_to_go_type_name(name) + struct =3D generate_struct_type("", fields, ident=3D1)[:-1] + return TEMPLATE_STRUCT_WITH_NULLABLE_MARSHAL.format( + struct=3Dstruct[1:], + type_name=3Dtype_name, + map_members=3Dmap_members, + map_special=3Dmap_special, + set_members=3Dset_members, + set_special=3Dset_special, + ) =20 =20 def recursive_base( self: QAPISchemaGenGolangVisitor, base: Optional[QAPISchemaObjectType], discriminator: Optional[str] =3D None, -) -> List[dict[str:str]]: +) -> Tuple[List[dict[str:str]], bool]: fields: List[dict[str:str]] =3D [] + with_nullable =3D False =20 if not base: - return fields + return fields, with_nullable =20 if base.base is not None: embed_base =3D self.schema.lookup_entity(base.base.name) - fields =3D recursive_base(self, embed_base, discriminator) + fields, with_nullable =3D recursive_base(self, embed_base, discrim= inator) =20 for member in base.local_members: if discriminator and member.name =3D=3D discriminator: continue - field =3D get_struct_field( - self, member.name, member.type.name, member.optional, False + field, nullable =3D get_struct_field( + self, member.name, member.type.name, False, member.optional, F= alse ) fields.append(field) + with_nullable =3D True if nullable else with_nullable =20 - return fields + return fields, with_nullable =20 =20 # Helper function that is used for most of QAPI types @@ -319,14 +519,20 @@ def qapi_to_golang_struct( variants: Optional[QAPISchemaVariants], ) -> str: =20 - fields =3D recursive_base(self, base) + fields, with_nullable =3D recursive_base(self, base) =20 if members: for member in members: - field =3D get_struct_field( - self, member.name, member.type.name, member.optional, False + field, nullable =3D get_struct_field( + self, + member.name, + member.type.name, + False, + member.optional, + False, ) fields.append(field) + with_nullable =3D True if nullable else with_nullable =20 if variants: fields.append({"comment": "Variants fields"}) @@ -334,13 +540,18 @@ def qapi_to_golang_struct( if variant.type.is_implicit(): continue =20 - field =3D get_struct_field( - self, variant.name, variant.type.name, True, True + field, nullable =3D get_struct_field( + self, variant.name, variant.type.name, False, True, True ) fields.append(field) + with_nullable =3D True if nullable else with_nullable =20 type_name =3D qapi_to_go_type_name(name) content =3D generate_struct_type(type_name, fields) + if with_nullable: + content +=3D struct_with_nullable_generate_marshal( + self, name, base, members, variants + ) return content =20 =20 @@ -465,7 +676,9 @@ def visit_begin(self, schema: QAPISchema) -> None: for target in self.target: self.target[target] =3D f"package {self.golang_package_name}\n" =20 - if target =3D=3D "helper": + if target =3D=3D "struct": + imports =3D '\nimport "encoding/json"\n' + elif target =3D=3D "helper": imports =3D """\nimport ( \t"encoding/json" \t"strings" --=20 2.41.0