From nobody Mon Apr 7 00:45:29 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1736506304; cv=none; d=zohomail.com; s=zohoarc; b=WkfwkYdKeIw3WnVwiPhVtqCQserDjyJKOKdW1HqUMJ122hBjNNh6ic/g6pQwFPECDZSYFlnbrf1ea+FBr65BMaALuM4cUwBSlXx8B24Y5tWhqaAGAlRjDlBml0bDPSMyL8u8CPFg48geS+H/19y6jQzseo8SZGuSeTkFD//AxB8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1736506304; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=/ZJT7zdBJIOBEpYSg4HBbHB2q/01X5+9t6cnHeqNSho=; b=I7mpRRKaANLWDAa57DrUbl0zc+M/ciNpdeqfp4c7LyTpjXNTsYkUmDbB1PH6oCYRVymoZjzk2nTAPMsSTGV+ggvddI5vZ+VW0sPyRSQPengzd2wFjlf5pGeuMWCmTP2hZGOe/FI4sKl1RNW9O0vDHUIkTyHVewEMNzQP44OXlC8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1736506304716540.3616689909773; Fri, 10 Jan 2025 02:51:44 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tWCah-000546-3W; Fri, 10 Jan 2025 05:50:11 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tWCad-00051y-9i for qemu-devel@nongnu.org; Fri, 10 Jan 2025 05:50:07 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tWCaa-0001G0-U5 for qemu-devel@nongnu.org; Fri, 10 Jan 2025 05:50:07 -0500 Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-456-FL8cK1EROdu0OLu82dbJ9w-1; Fri, 10 Jan 2025 05:50:02 -0500 Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id D9B0F19560B7 for ; Fri, 10 Jan 2025 10:50:01 +0000 (UTC) Received: from tapioca.redhat.com (unknown [10.45.225.126]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 0C2561955BE3; Fri, 10 Jan 2025 10:49:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1736506204; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/ZJT7zdBJIOBEpYSg4HBbHB2q/01X5+9t6cnHeqNSho=; b=OgHOxNlK5Y+tTeQ4/F1sqmQ3oFBw2Xhn2n0hjriQVMiBdmm4Lm0izjeij1JJwv8SkJptn/ sbnUZjdO9kF2WpS1eHYZBwef0bshy1QgJUZKEjMl6QC6ixAFXNbSH8Wq4LqkdX7k2yMca9 KJQh01roon9ordup9ggDfgZEIgPHiCo= X-MC-Unique: FL8cK1EROdu0OLu82dbJ9w-1 X-Mimecast-MFC-AGG-ID: FL8cK1EROdu0OLu82dbJ9w From: Victor Toso To: qemu-devel@nongnu.org Cc: Markus Armbruster , John Snow , =?UTF-8?q?Daniel=20P=20=2E=20Berrang=C3=A9?= , Andrea Bolognani Subject: [PATCH v3 4/8] qapi: golang: structs: Address nullable members Date: Fri, 10 Jan 2025 11:49:42 +0100 Message-ID: <20250110104946.74960-5-victortoso@redhat.com> In-Reply-To: <20250110104946.74960-1-victortoso@redhat.com> References: <20250110104946.74960-1-victortoso@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=victortoso@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -8 X-Spam_score: -0.9 X-Spam_bar: / X-Spam_report: (-0.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.436, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_SBL=1.623 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1736506306023019000 Content-Type: text/plain; charset="utf-8" Explaining why this is needed needs some context, so taking the example of StrOrNull alternate type and considering a simplified struct that has two fields: qapi: | { 'struct': 'MigrationExample', | 'data': { '*label': 'StrOrNull', | 'target': 'StrOrNull' } } We have an optional member 'label' which can have three JSON values: 1. A string: { "target": "a.host.com", "label": "happy" } 2. A null : { "target": "a.host.com", "label": null } 3. Absent : { "target": null} The member 'target' is not optional, hence it can't be absent. A Go struct that contains an optional type that can be JSON Null like 'label' in the example above, will need extra care when Marshaling and Unmarshaling from JSON. This patch handles this very specific case: - It implements the Marshaler interface for these structs to properly handle these values. - It adds the interface AbsentAlternate() and implement it for any Alternate that can be JSON Null. See its uses in map_and_set() Signed-off-by: Victor Toso --- scripts/qapi/golang.py | 300 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 289 insertions(+), 11 deletions(-) diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py index df40bd89f2..ada89f0ce8 100644 --- a/scripts/qapi/golang.py +++ b/scripts/qapi/golang.py @@ -55,6 +55,28 @@ } return nil } + +// This helper is used to move struct's fields into a map. +// This function is useful to merge JSON objects. +func unwrapToMap(m map[string]any, data any) error { + if bytes, err :=3D json.Marshal(&data); err !=3D nil { + return fmt.Errorf("unwrapToMap: %s", err) + } else if err :=3D json.Unmarshal(bytes, &m); err !=3D nil { + return fmt.Errorf("unwrapToMap: %s, data=3D%s", err, string(bytes)) + } + return nil +} +""" + +TEMPLATE_ALTERNATE =3D """ +// Only implemented on Alternate types that can take JSON NULL as value. +// +// This is a helper for the marshalling code. It should return true only w= hen +// the Alternate is empty (no members are set), otherwise it returns false= and +// the member set to be Marshalled. +type AbsentAlternate interface { + ToAnyOrAbsent() (any, bool) +} """ =20 TEMPLATE_ALTERNATE_CHECK_INVALID_JSON_NULL =3D """ @@ -96,6 +118,19 @@ return nil }""" =20 +TEMPLATE_ALTERNATE_NULLABLE =3D """ +func (s *{name}) ToAnyOrAbsent() (any, bool) {{ + if s !=3D nil {{ + if s.IsNull {{ + return nil, false +{absent_check_fields} + }} + }} + + return nil, true +}} +""" + TEMPLATE_ALTERNATE_METHODS =3D """ func (s {name}) MarshalJSON() ([]byte, error) {{ {marshal_check_fields} @@ -109,6 +144,26 @@ """ =20 =20 +TEMPLATE_STRUCT_WITH_NULLABLE_MARSHAL =3D """ +func (s {type_name}) MarshalJSON() ([]byte, error) {{ + m :=3D make(map[string]any) +{map_members}{map_special} + return json.Marshal(&m) +}} + +func (s *{type_name}) UnmarshalJSON(data []byte) error {{ + tmp :=3D {struct}{{}} + + if err :=3D json.Unmarshal(data, &tmp); err !=3D nil {{ + return err + }} + +{set_members}{set_special} + return nil +}} +""" + + # Takes the documentation object of a specific type and returns # that type's documentation and its member's docs. def qapi_to_golang_struct_docs(doc: QAPIDoc) -> (str, Dict[str, str]): @@ -357,20 +412,30 @@ def get_struct_field( qapi_name: str, qapi_type_name: str, field_doc: str, + within_nullable_struct: bool, is_optional: bool, is_variant: bool, -) -> dict[str:str]: +) -> Tuple[dict[str:str], bool]: field =3D qapi_to_field_name(qapi_name) member_type =3D qapi_schema_type_to_go_type(qapi_type_name) + is_nullable =3D False =20 optional =3D "" if is_optional: - if member_type not in self.accept_null_types: + if member_type in self.accept_null_types: + is_nullable =3D True + else: optional =3D ",omitempty" =20 # Use pointer to type when field is optional isptr =3D "*" if is_optional and member_type[0] not in "*[" else "" =20 + if within_nullable_struct: + # Within a struct which has a field of type that can hold JSON NUL= L, + # we have to _not_ use a pointer, otherwise the Marshal methods are + # not called. + isptr =3D "" if member_type in self.accept_null_types else isptr + fieldtag =3D ( '`json:"-"`' if is_variant else f'`json:"{qapi_name}{optional}"`' ) @@ -382,38 +447,228 @@ def get_struct_field( if field_doc !=3D "": arg["doc"] =3D field_doc =20 - return arg + return arg, is_nullable + + +# This helper is used whithin a struct that has members that accept JSON N= ULL. +def map_and_set( + is_nullable: bool, field: str, field_is_optional: bool, name: str +) -> Tuple[str, str]: + mapstr =3D "" + setstr =3D "" + if is_nullable: + mapstr =3D f""" + if val, absent :=3D s.{field}.ToAnyOrAbsent(); !absent {{ + m["{name}"] =3D val + }} +""" + setstr +=3D f""" + if _, absent :=3D (&tmp.{field}).ToAnyOrAbsent(); !absent {{ + s.{field} =3D &tmp.{field} + }} +""" + elif field_is_optional: + mapstr =3D f""" + if s.{field} !=3D nil {{ + m["{name}"] =3D s.{field} + }} +""" + setstr =3D f""" s.{field} =3D tmp.{field}\n""" + else: + mapstr =3D f""" m["{name}"] =3D s.{field}\n""" + setstr =3D f""" s.{field} =3D tmp.{field}\n""" + + return mapstr, setstr + + +def recursive_base_nullable( + self: QAPISchemaGenGolangVisitor, base: Optional[QAPISchemaObjectType] +) -> Tuple[List[dict[str:str]], str, str, str, str]: + fields: List[dict[str:str]] =3D [] + map_members =3D "" + set_members =3D "" + map_special =3D "" + set_special =3D "" + + if not base: + return fields, map_members, set_members, map_special, set_special + + doc =3D self.docmap.get(base.name, None) + _, docfields =3D qapi_to_golang_struct_docs(doc) + + if base.base is not None: + embed_base =3D self.schema.lookup_entity(base.base.name) + ( + fields, + map_members, + set_members, + map_special, + set_special, + ) =3D recursive_base_nullable(self, embed_base) + + for member in base.local_members: + field_doc =3D docfields.get(member.name, "") + field, _ =3D get_struct_field( + self, + member.name, + member.type.name, + field_doc, + True, + member.optional, + False, + ) + fields.append(field) + + member_type =3D qapi_schema_type_to_go_type(member.type.name) + nullable =3D member_type in self.accept_null_types + field_name =3D qapi_to_field_name(member.name) + tomap, toset =3D map_and_set( + nullable, field_name, member.optional, member.name + ) + if nullable: + map_special +=3D tomap + set_special +=3D toset + else: + map_members +=3D tomap + set_members +=3D toset + + return fields, map_members, set_members, map_special, set_special + + +# Helper function. This is executed when the QAPI schema has members +# that could accept JSON NULL (e.g: StrOrNull in QEMU"s QAPI schema). +# This struct will need to be extended with Marshal/Unmarshal methods to +# properly handle such atypical members. +# +# Only the Marshallaing methods are generated but we do need to iterate ov= er +# all the members to properly set/check them in those methods. +def struct_with_nullable_generate_marshal( + self: QAPISchemaGenGolangVisitor, + name: str, + base: Optional[QAPISchemaObjectType], + members: List[QAPISchemaObjectTypeMember], + variants: Optional[QAPISchemaVariants], +) -> str: + ( + fields, + map_members, + set_members, + map_special, + set_special, + ) =3D recursive_base_nullable(self, base) + + doc =3D self.docmap.get(name, None) + _, docfields =3D qapi_to_golang_struct_docs(doc) + + if members: + for member in members: + field_doc =3D docfields.get(member.name, "") + field, _ =3D get_struct_field( + self, + member.name, + member.type.name, + field_doc, + True, + member.optional, + False, + ) + fields.append(field) + + member_type =3D qapi_schema_type_to_go_type(member.type.name) + nullable =3D member_type in self.accept_null_types + tomap, toset =3D map_and_set( + nullable, + qapi_to_field_name(member.name), + member.optional, + member.name, + ) + if nullable: + map_special +=3D tomap + set_special +=3D toset + else: + map_members +=3D tomap + set_members +=3D toset + + if variants: + for variant in variants.variants: + if variant.type.is_implicit(): + continue + + field, _ =3D get_struct_field( + self, + variant.name, + variant.type.name, + True, + variant.optional, + True, + ) + fields.append(field) + + member_type =3D qapi_schema_type_to_go_type(variant.type.name) + nullable =3D member_type in self.accept_null_types + tomap, toset =3D map_and_set( + nullable, + qapi_to_field_name(variant.name), + variant.optional, + variant.name, + ) + if nullable: + map_special +=3D tomap + set_special +=3D toset + else: + map_members +=3D tomap + set_members +=3D toset + + type_name =3D qapi_to_go_type_name(name) + struct =3D generate_struct_type("", args=3Dfields, indent=3D1) + return string_to_code( + TEMPLATE_STRUCT_WITH_NULLABLE_MARSHAL.format( + struct=3Dstruct[1:-1], + type_name=3Dtype_name, + map_members=3Dmap_members, + map_special=3Dmap_special, + set_members=3Dset_members, + set_special=3Dset_special, + ) + ) =20 =20 def recursive_base( self: QAPISchemaGenGolangVisitor, base: Optional[QAPISchemaObjectType], -) -> List[dict[str:str]]: + discriminator: Optional[str] =3D None, +) -> Tuple[List[dict[str:str]], bool]: fields: List[dict[str:str]] =3D [] + with_nullable =3D False =20 if not base: - return fields + return fields, with_nullable =20 if base.base is not None: embed_base =3D self.schema.lookup_entity(base.base.name) - fields =3D recursive_base(self, embed_base) + fields, with_nullable =3D recursive_base(self, embed_base, discrim= inator) =20 doc =3D self.docmap.get(base.name, None) _, docfields =3D qapi_to_golang_struct_docs(doc) =20 for member in base.local_members: + if discriminator and member.name =3D=3D discriminator: + continue + field_doc =3D docfields.get(member.name, "") - field =3D get_struct_field( + field, nullable =3D get_struct_field( self, member.name, member.type.name, field_doc, + False, member.optional, False, ) fields.append(field) + with_nullable =3D True if nullable else with_nullable =20 - return fields + return fields, with_nullable =20 =20 # Helper function that is used for most of QAPI types @@ -429,7 +684,8 @@ def qapi_to_golang_struct( indent: int =3D 0, doc_enabled: bool =3D True, ) -> str: - fields =3D recursive_base(self, base) + discriminator =3D None if not variants else variants.tag_member.name + fields, with_nullable =3D recursive_base(self, base, discriminator) =20 doc =3D self.docmap.get(name, None) type_doc, docfields =3D qapi_to_golang_struct_docs(doc) @@ -439,15 +695,17 @@ def qapi_to_golang_struct( if members: for member in members: field_doc =3D docfields.get(member.name, "") if doc_enabled el= se "" - field =3D get_struct_field( + field, nullable =3D get_struct_field( self, member.name, member.type.name, field_doc, + False, member.optional, False, ) fields.append(field) + with_nullable =3D True if nullable else with_nullable =20 exists =3D {} if variants: @@ -458,15 +716,17 @@ def qapi_to_golang_struct( =20 exists[variant.name] =3D True field_doc =3D docfields.get(variant.name, "") if doc_enabled e= lse "" - field =3D get_struct_field( + field, nullable =3D get_struct_field( self, variant.name, variant.type.name, field_doc, + False, True, True, ) fields.append(field) + with_nullable =3D True if nullable else with_nullable =20 type_name =3D qapi_to_go_type_name(name) content =3D string_to_code( @@ -474,6 +734,10 @@ def qapi_to_golang_struct( type_name, type_doc=3Dtype_doc, args=3Dfields, indent=3Dindent ) ) + if with_nullable: + content +=3D struct_with_nullable_generate_marshal( + self, name, base, members, variants + ) return content =20 =20 @@ -482,6 +746,7 @@ def generate_template_alternate( name: str, variants: Optional[QAPISchemaVariants], ) -> str: + absent_check_fields =3D "" args: List[dict[str:str]] =3D [] nullable =3D name in self.accept_null_types if nullable: @@ -515,6 +780,12 @@ def generate_template_alternate( assert nullable continue =20 + if nullable: + absent_check_fields +=3D string_to_code( + TEMPLATE_ALTERNATE_NULLABLE_CHECK[1:].format( + var_name=3Dvar_name + ) + ) skip_indent =3D 1 + len(FOUR_SPACES) if marshal_check_fields =3D=3D "": skip_indent =3D 1 @@ -526,6 +797,12 @@ def generate_template_alternate( ].format(var_name=3Dvar_name, var_type=3Dvar_type) =20 content +=3D string_to_code(generate_struct_type(name, args=3Dargs)) + if nullable: + content +=3D string_to_code( + TEMPLATE_ALTERNATE_NULLABLE.format( + name=3Dname, absent_check_fields=3Dabsent_check_fields[:-1] + ) + ) content +=3D string_to_code( TEMPLATE_ALTERNATE_METHODS.format( name=3Dname, @@ -634,6 +911,7 @@ def visit_begin(self, schema: QAPISchema) -> None: self.target[target] +=3D string_to_code(imports) =20 self.target["helper"] +=3D string_to_code(TEMPLATE_HELPER) + self.target["alternate"] +=3D string_to_code(TEMPLATE_ALTERNATE) =20 def visit_end(self) -> None: del self.schema --=20 2.47.1