From nobody Tue Nov 26 13:36:34 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 207.211.31.81 as permitted sender) client-ip=207.211.31.81; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-1.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zoho.com: domain of redhat.com designates 207.211.31.81 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1573483566; cv=none; d=zoho.com; s=zohoarc; b=J4f7jtAST0LmAxwX+U6gdMU00hNycVsDp7HJdEefluMmshG6yr+wq6cRqmsPeogXVmtvJOg/PZi5fFPMRniEd1zRiF/SIMhC7Fw4Rq6MLBqUJlM/2JtS95msoBwVBC6BaxyO5MLI9LaTxGMVN1+xPolspM6L2HeBxDnSit76NYc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1573483566; h=Content-Type:Content-Transfer-Encoding:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=8n2cF/Ge9owStxSXUtmBcHnGEcjmQcXvOZNW2oSXvek=; b=SCvjxw+lKd0CZYDmrmBHIwhgJ1GsnuCJV2dKAkmMN5N4JhSIDgvX2zw5Lp2+OLCUgEJX5hyxi7L1HTq92F0SlOBdbqK8AJa+bSAiVNLqnGo/o+K+UNIU0p6h+tTU3lhaVcGVnLL/8Z0vDRzzd+h4Z0myRkOZ6JGbaQLBz6nsHGI= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass; spf=pass (zoho.com: domain of redhat.com designates 207.211.31.81 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from us-smtp-delivery-1.mimecast.com (us-smtp-2.mimecast.com [207.211.31.81]) by mx.zohomail.com with SMTPS id 1573483566905115.03546009661636; Mon, 11 Nov 2019 06:46:06 -0800 (PST) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-65-myI5Gw5mMq2R3inB-nmHqg-1; Mon, 11 Nov 2019 09:45:24 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id B0CBD1852E21; Mon, 11 Nov 2019 14:45:18 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 7FC6084F7; Mon, 11 Nov 2019 14:45:18 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 30F904E57D; Mon, 11 Nov 2019 14:45:18 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id xABEj155020486 for ; Mon, 11 Nov 2019 09:45:02 -0500 Received: by smtp.corp.redhat.com (Postfix) id F14695DD73; Mon, 11 Nov 2019 14:45:01 +0000 (UTC) Received: from catbus.gsslab.fab.redhat.com (dhcp-32.gsslab.fab.redhat.com [10.33.9.32]) by smtp.corp.redhat.com (Postfix) with ESMTP id 60EC55D9C9; Mon, 11 Nov 2019 14:45:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1573483565; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=8n2cF/Ge9owStxSXUtmBcHnGEcjmQcXvOZNW2oSXvek=; b=fIFLcCXllAqcOaWnbtW584TCAbXBFoNfkd7tNwMVAt/6zcTS/Pjs5Cyg1GFpArUPfbSsRY IKbe1DVrT+DRf6oJ4XESeODjcBKCPMjwdVhjRPm3lI9gY6pMFtCNRdvsgtP2wHAwcnSYa/ HMRmPH0LJPrXYYhGm0du64X0EWTdLh8= From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= To: libvir-list@redhat.com Date: Mon, 11 Nov 2019 14:38:17 +0000 Message-Id: <20191111143826.16050-15-berrange@redhat.com> In-Reply-To: <20191111143826.16050-1-berrange@redhat.com> References: <20191111143826.16050-1-berrange@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-loop: libvir-list@redhat.com Subject: [libvirt] [PATCH v5 14/23] src: rewrite remote protocol checker in Python X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-MC-Unique: myI5Gw5mMq2R3inB-nmHqg-1 X-Mimecast-Spam-Score: 0 Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) Content-Type: text/plain; charset="utf-8" As part of an goal to eliminate Perl from libvirt build tools, rewrite the pdwtags processing script in Python. The original inline shell and perl code was completely unintelligible. The new python code is a manual conversion that attempts todo basically the same thing. Signed-off-by: Daniel P. Berrang=C3=A9 Reviewed-by: J=C3=A1n Tomko Tested-by: Cole Robinson --- Makefile.am | 1 + build-aux/syntax-check.mk | 3 +- scripts/check-remote-protocol.py | 136 +++++++++++++++++++++++++++++++ src/Makefile.am | 98 ++++------------------ 4 files changed, 155 insertions(+), 83 deletions(-) create mode 100644 scripts/check-remote-protocol.py diff --git a/Makefile.am b/Makefile.am index e7ebe7281a..8c9c73c715 100644 --- a/Makefile.am +++ b/Makefile.am @@ -50,6 +50,7 @@ EXTRA_DIST =3D \ scripts/check-aclrules.py \ scripts/check-drivername.py \ scripts/check-driverimpls.py \ + scripts/check-remote-protocol.py \ scripts/check-spacing.py \ scripts/check-symfile.py \ scripts/check-symsorting.py \ diff --git a/build-aux/syntax-check.mk b/build-aux/syntax-check.mk index 26eb5b94d9..a61818855c 100644 --- a/build-aux/syntax-check.mk +++ b/build-aux/syntax-check.mk @@ -412,6 +412,7 @@ sc_prohibit_mkstemp: # access with F_OK or R_OK is okay, though. sc_prohibit_access_xok: @prohibit=3D'access(at)? *\(.*X_OK' \ + in_vc_files=3D'\.[ch]$$' \ halt=3D'use virFileIsExecutable instead of access(,X_OK)' \ $(_sc_search_regexp) =20 @@ -2216,7 +2217,7 @@ exclude_file_name_regexp--sc_prohibit_PATH_MAX =3D \ ^build-aux/syntax-check\.mk$$ =20 exclude_file_name_regexp--sc_prohibit_access_xok =3D \ - ^(build-aux/syntax-check\.mk|src/util/virutil\.c)$$ + ^(src/util/virutil\.c)$$ =20 exclude_file_name_regexp--sc_prohibit_asprintf =3D \ ^(build-aux/syntax-check\.mk|bootstrap.conf$$|examples/|src/util/virstri= ng\.[ch]$$|tests/vircgroupmock\.c|tools/virt-login-shell\.c|tools/nss/libvi= rt_nss\.c$$) diff --git a/scripts/check-remote-protocol.py b/scripts/check-remote-protoc= ol.py new file mode 100644 index 0000000000..074f8c0ae1 --- /dev/null +++ b/scripts/check-remote-protocol.py @@ -0,0 +1,136 @@ +#!/usr/bin/env python +# +# Copyright (C) 2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# . +# +# This uses pdwtags to check remote protocol defs +# +# * the "split" splits on the /* DD */ comments, so that $p iterates +# through the struct definitions. +# * process only "struct remote_..." entries +# * remove comments and preceding TAB throughout +# * remove empty lines throughout +# * remove white space at end of buffer + +from __future__ import print_function + +import os +import os.path +import re +import subprocess +import sys + +cc =3D sys.argv[1] +objext =3D sys.argv[2] +proto_lo =3D sys.argv[3] +expected =3D sys.argv[4] + +proto_lo =3D proto_lo.replace("/", "/.libs/") + +ccargv =3D cc.split(" ") +ccargv.append("-v") +ccproc =3D subprocess.Popen(ccargv, stdout=3Dsubprocess.PIPE, + stderr=3Dsubprocess.STDOUT) +out, err =3D ccproc.communicate() +out =3D out.decode("utf-8") +if out.find("clang") !=3D -1: + print("WARNING: skipping pdwtags test with Clang", file=3Dsys.stderr) + sys.exit(0) + + +def which(program): + def is_exe(fpath): + return (os.path.isfile(fpath) and + os.access(fpath, os.X_OK)) + + fpath, fname =3D os.path.split(program) + if fpath: + if is_exe(program): + return program + else: + for path in os.environ["PATH"].split(os.pathsep): + exe_file =3D os.path.join(path, program) + if is_exe(exe_file): + return exe_file + + return None + + +pdwtags =3D which("pdwtags") +if pdwtags is None: + print("WARNING: you lack pdwtags; skipping the protocol test", + file=3Dsys.stderr) + print("WARNING: install the dwarves package to get pdwtags", + file=3Dsys.stderr) + sys.exit(0) + +proto_o =3D proto_lo.replace(".lo", ".o") + +if not os.path.exists(proto_o): + raise Exception("Missing %s", proto_o) + +pdwtagsproc =3D subprocess.Popen(["pdwtags", "--verbose", proto_o], + stdout=3Dsubprocess.PIPE, stderr=3Dsubproce= ss.PIPE) +out, err =3D pdwtagsproc.communicate() +out =3D out.decode("utf-8") +err =3D err.decode("utf-8") + +if out =3D=3D "" and err !=3D "": + print("WARNING: no output, pdwtags appears broken:", file=3Dsys.stderr) + for l in err.strip().split("\n"): + print("WARNING: %s" % l, file=3Dsys.stderr) + print("WARNING: skipping the remote protocol test", file=3Dsys.stderr) + sys.exit(0) + +# With pdwtags 1.8, --verbose output includes separators like these: +# /* 93 */ +# /* <0> (null):0 */ +# with the second line omitted for intrinsic types. +# Whereas with pdwtags 1.3, they look like this: +# /* <2d2> /usr/include/libio.h:180 */ +# The alternation of the following regexps matches both cases. +r1 =3D r'''/\* \d+ \*/''' +r2 =3D r'''/\* <[0-9a-fA-F]+> \S+:\d+ \*/''' + +libs_prefix =3D "remote_|qemu_|lxc_|admin_" +other_prefix =3D "keepalive|vir(Net|LockSpace|LXCMonitor)" +struct_prefix =3D "(" + libs_prefix + "|" + other_prefix + ")" + +n =3D 0 +bits =3D re.split(r'''\n*(?:%s|%s)\n''' % (r1, r2), out) +actual =3D ["/* -*- c -*- */"] + +for bit in bits: + if re.search(r'''^(struct|enum)\s+''' + struct_prefix, bit): + bit =3D re.sub(r'''\t*/\*.*?\*/''', "", bit) + bit =3D re.sub(r'''\s+\n''', '''\n''', bit) + bit =3D re.sub(r'''\s+$''', "", bit) + bit =3D re.sub(r'''\t''', " ", bit) + actual.append(bit) + n =3D n + 1 + +if n < 1: + print("WARNING: No structs/enums matched. Your", file=3Dsys.stderr) + print("WARNING: pdwtags program is probably too old", file=3Dsys.stder= r) + print("WARNING: skipping the remote protocol test", file=3Dsys.stderr) + print("WARNING: install dwarves-1.3 or newer", file=3Dsys.stderr) + sys.exit(8) + +diff =3D subprocess.Popen(["diff", "-u", expected, "-"], stdin=3Dsubproces= s.PIPE) +actualstr =3D "\n".join(actual) + "\n" +diff.communicate(input=3Dactualstr.encode("utf-8")) + +sys.exit(diff.returncode) diff --git a/src/Makefile.am b/src/Makefile.am index bb63e2486c..c40e61e7ee 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -196,83 +196,6 @@ DRIVER_SOURCES +=3D \ =20 =20 =20 -# Ensure that we don't change the struct or member names or member ordering -# in remote_protocol.x The embedded perl below needs a few comments, and -# presumes you know what pdwtags output looks like: -# * use -0777 -n to slurp the entire file into $_. -# * the "split" splits on the /* DD */ comments, so that $p iterates -# through the struct definitions. -# * process only "struct remote_..." entries -# * remove comments and preceding TAB throughout -# * remove empty lines throughout -# * remove white space at end of buffer - -# With pdwtags 1.8, --verbose output includes separators like these: -# /* 93 */ -# /* <0> (null):0 */ -# with the second line omitted for intrinsic types. -# Whereas with pdwtags 1.3, they look like this: -# /* <2d2> /usr/include/libio.h:180 */ -# The alternation of the following regexps matches both cases. -r1 =3D /\* \d+ \*/ -r2 =3D /\* <[[:xdigit:]]+> \S+:\d+ \*/ -libs_prefix =3D remote_|qemu_|lxc_|admin_ -other_prefix =3D keepalive|vir(Net|LockSpace|LXCMonitor) -struct_prefix =3D ($(libs_prefix)|$(other_prefix)) - -# Depending on configure options, libtool creates one or both of -# remote/{,.libs/}libvirt_driver_remote_la-remote_protocol.o. We want -# the newest of the two, in case configure options changed and a stale -# file is left around from an earlier build. -# The pdwtags output is completely different when building with clang -# which causes the comparison against expected output to fail, so skip -# if using clang as CC. -PDWTAGS =3D \ - $(AM_V_GEN)if $(CC) -v 2>&1 | grep -q clang; then \ - echo 'WARNING: skipping pdwtags test with Clang' >&2; \ - exit 0; \ - fi; \ - if (pdwtags --help) > /dev/null 2>&1; then \ - o=3D`ls -t $(<:.lo=3D.$(OBJEXT)) \ - $(subst /,/.libs/,$(<:.lo=3D.$(OBJEXT))) \ - 2>/dev/null | sed -n 1p`; \ - test -f "$$o" || { echo ".o for $< not found" >&2; exit 1; }; \ - pdwtags --verbose $$o > $(@F)-t1 2> $(@F)-t2; \ - if test ! -s $(@F)-t1 && test -s $(@F)-t2; then \ - rm -rf $(@F)-t?; \ - echo 'WARNING: pdwtags appears broken; skipping the $@ test' >&2;\ - else \ - $(PERL) -0777 -n \ - -e 'foreach my $$p (split m!\n*(?:$(r1)|$(r2))\n!) {' \ - -e ' if ($$p =3D~ /^(struct|enum) $(struct_prefix)/) {' \ - -e ' $$p =3D~ s!\t*/\*.*?\*/!!sg;' \ - -e ' $$p =3D~ s!\s+\n!\n!sg;' \ - -e ' $$p =3D~ s!\s+$$!!;' \ - -e ' $$p =3D~ s!\t! !g;' \ - -e ' print "$$p\n";' \ - -e ' $$n++;' \ - -e ' }' \ - -e '}' \ - -e 'BEGIN {' \ - -e ' print "/* -*- c -*- */\n";' \ - -e '}' \ - -e 'END {' \ - -e ' if ($$n < 1) {' \ - -e ' warn "WARNING: your pdwtags program is too old\n";' \ - -e ' warn "WARNING: skipping the $@ test\n";' \ - -e ' warn "WARNING: install dwarves-1.3 or newer\n";' \ - -e ' exit 8;' \ - -e ' }' \ - -e '}' \ - < $(@F)-t1 > $(@F)-t3; \ - case $$? in 8) rm -f $(@F)-t?; exit 0;; 0) ;; *) exit 1;; esac;\ - diff -u $(@)s $(@F)-t3; st=3D$$?; rm -f $(@F)-t?; exit $$st; \ - fi; \ - else \ - echo 'WARNING: you lack pdwtags; skipping the $@ test' >&2; \ - echo 'WARNING: install the dwarves package to get pdwtags' >&2; \ - fi - # .libs/libvirt.so is built by libtool as a side-effect of the Makefile # rule for libvirt.la. However, checking symbols relies on Linux ELF layo= ut if WITH_LINUX @@ -301,27 +224,38 @@ PROTOCOL_STRUCTS =3D \ if WITH_REMOTE check-protocol: $(PROTOCOL_STRUCTS) $(PROTOCOL_STRUCTS:structs=3Dstruct) =20 +# Ensure that we don't change the struct or member names or member ordering +# in remote_protocol.x The process-pdwtags.py post-processes output to +# extract the bits we want. + +CHECK_REMOTE_PROTOCOL =3D $(top_srcdir)/scripts/check-remote-protocol.py + # The .o file that pdwtags parses is created as a side effect of running # libtool; but from make's perspective we depend on the .lo file. $(srcdir)/remote_protocol-struct \ $(srcdir)/qemu_protocol-struct \ $(srcdir)/lxc_protocol-struct: \ $(srcdir)/%-struct: remote/libvirt_driver_remote_la-%.lo - $(PDWTAGS) + $(AM_V_GEN)$(RUNUTF8) $(PYTHON) $(CHECK_REMOTE_PROTOCOL) \ + "$(CC)" "$(OBJEXT)" $< $(@)s $(srcdir)/virnetprotocol-struct $(srcdir)/virkeepaliveprotocol-struct: \ $(srcdir)/%-struct: rpc/libvirt_net_rpc_la-%.lo - $(PDWTAGS) + $(AM_V_GEN)$(RUNUTF8) $(PYTHON) $(CHECK_REMOTE_PROTOCOL) \ + "$(CC)" "$(OBJEXT)" $< $(@)s if WITH_LXC $(srcdir)/lxc_monitor_protocol-struct: \ $(srcdir)/%-struct: lxc/libvirt_driver_lxc_impl_la-%.lo - $(PDWTAGS) + $(AM_V_GEN)$(RUNUTF8) $(PYTHON) $(CHECK_REMOTE_PROTOCOL) \ + "$(CC)" "$(OBJEXT)" $< $(@)s endif WITH_LXC $(srcdir)/lock_protocol-struct: \ $(srcdir)/%-struct: locking/lockd_la-%.lo - $(PDWTAGS) + $(AM_V_GEN)$(RUNUTF8) $(PYTHON) $(CHECK_REMOTE_PROTOCOL) \ + "$(CC)" "$(OBJEXT)" $< $(@)s $(srcdir)/admin_protocol-struct: \ $(srcdir)/%-struct: admin/libvirt_admin_la-%.lo - $(PDWTAGS) + $(AM_V_GEN)$(RUNUTF8) $(PYTHON) $(CHECK_REMOTE_PROTOCOL) \ + "$(CC)" "$(OBJEXT)" $< $(@)s =20 else !WITH_REMOTE # The $(PROTOCOL_STRUCTS) files must live in git, because they cannot be --=20 2.21.0 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list