From nobody Sun Feb 8 20:45:30 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.libvirt.org designates 8.43.85.245 as permitted sender) client-ip=8.43.85.245; envelope-from=devel-bounces@lists.libvirt.org; helo=lists.libvirt.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of lists.libvirt.org designates 8.43.85.245 as permitted sender) smtp.mailfrom=devel-bounces@lists.libvirt.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.libvirt.org (lists.libvirt.org [8.43.85.245]) by mx.zohomail.com with SMTPS id 1739183184483205.55258504768562; Mon, 10 Feb 2025 02:26:24 -0800 (PST) Received: by lists.libvirt.org (Postfix, from userid 996) id 8D0B613E0; Mon, 10 Feb 2025 05:26:23 -0500 (EST) Received: from lists.libvirt.org (localhost [IPv6:::1]) by lists.libvirt.org (Postfix) with ESMTP id 496D315AF; Mon, 10 Feb 2025 05:26:05 -0500 (EST) Received: by lists.libvirt.org (Postfix, from userid 996) id 16E3A1588; Mon, 10 Feb 2025 05:26:02 -0500 (EST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.libvirt.org (Postfix) with ESMTPS id 33B3313E0 for ; Mon, 10 Feb 2025 05:26:01 -0500 (EST) Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-382-yhQlMHRJNUyHYxXwyeF7LA-1; Mon, 10 Feb 2025 05:25:59 -0500 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 717061800874 for ; Mon, 10 Feb 2025 10:25:58 +0000 (UTC) Received: from work.fritz.box (unknown [10.44.33.7]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 7048319560A3; Mon, 10 Feb 2025 10:25:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on lists.libvirt.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,RCVD_IN_VALIDITY_RPBL_BLOCKED, RCVD_IN_VALIDITY_SAFE_BLOCKED,SPF_HELO_NONE autolearn=unavailable autolearn_force=no version=3.4.4 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739183160; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=93H7X8zDOvyME+e7Az0dJ0mWYiO102lQJgUN9QFWftc=; b=U8WQHAIZMeqXxdTiAcHK1obXWfNSP8UDneX2qBWsYVVvpiyqgLytPysoVGXm2k70lQxkc+ IcSddp079q7RZ4ts5tEb1GvekbeOpWtRi+5OR2sVm0W1P+Ons/FQfLkdXDrRnF/gSbsuAd Oq+sGAmbjdT5V+vTvlAxieXp8KPxHL0= X-MC-Unique: yhQlMHRJNUyHYxXwyeF7LA-1 X-Mimecast-MFC-AGG-ID: yhQlMHRJNUyHYxXwyeF7LA From: Tim Wiederhake To: devel@lists.libvirt.org Subject: [PATCH] tests: Record negative tests as pass instead of expected-fail Date: Mon, 10 Feb 2025 11:25:55 +0100 Message-ID: <20250210102555.40190-1-twiederh@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: h6ew3rH-hgcKOZj8vVe4p987ha5USFVSCezB7QwG5DA_1739183158 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Message-ID-Hash: MTSIYSSWXHVT46GPZR7BWV3ECEOFEZ6G X-Message-ID-Hash: MTSIYSSWXHVT46GPZR7BWV3ECEOFEZ6G X-MailFrom: twiederh@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-config-1; header-match-config-2; header-match-config-3; header-match-devel.lists.libvirt.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; suspicious-header CC: Tim Wiederhake X-Mailman-Version: 3.2.2 Precedence: list List-Id: Development discussions about the libvirt library & tools Archived-At: List-Archive: List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1739183186594019000 Content-Type: text/plain; charset="utf-8"; x-default="true" meson's "test()" function provides a "should_fail: bool" argument that checks for a command to exit with a non-zero exit code instead of the usual zero exit code to signal success. If the program under test does so, it is recorded as "EXPECTEDFAIL" instead of "OK". While there is an argument to be made that the program under test failed as expected, the test in itself is successful and should be recorded as such. Before: $ meson test ... 151/300 libvirt:bin / libvirtd fail with missing config = EXPECTEDFAIL 0.03s exit status 1 ... Ok: 299 Expected Fail: 1 Fail: 0 Unexpected Pass: 0 Skipped: 0 Timeout: 0 After: $ meson test ... 151/300 libvirt:bin / libvirtd fail with missing config = OK 0.03s ... Ok: 300 Expected Fail: 0 Fail: 0 Unexpected Pass: 0 Skipped: 0 Timeout: 0 Signed-off-by: Tim Wiederhake --- tests/expected-fail | 13 +++++++++++++ tests/meson.build | 5 ++--- 2 files changed, 15 insertions(+), 3 deletions(-) create mode 100755 tests/expected-fail diff --git a/tests/expected-fail b/tests/expected-fail new file mode 100755 index 0000000000..85738c95dc --- /dev/null +++ b/tests/expected-fail @@ -0,0 +1,13 @@ +#!/bin/sh + +# meson's "test()" function provides a "should_fail: bool" argument that +# checks for a command to exit with a non-zero exit code instead of the us= ual +# zero exit code to signal success. If the program under test does so, it = is +# recorded as "EXPECTEDFAIL" instead of "OK". While there is an argument t= o be +# made that the program under test failed as expected, the test in itself = is +# successful and should be recorded as such. + +if "$@" +then + exit 1 +fi diff --git a/tests/meson.build b/tests/meson.build index 0d76d37959..0ebcdc1496 100644 --- a/tests/meson.build +++ b/tests/meson.build @@ -725,9 +725,8 @@ endif =20 if conf.has('WITH_LIBVIRTD') test('libvirtd fail with missing config', - libvirtd_prog, - args: [ '--config=3Dno-such-conf', '--timeout=3D5' ], - should_fail: true, + find_program('expected-fail'), + args: [ libvirtd_prog, '--config=3Dno-such-conf', '--timeout=3D5' ], suite: 'bin', ) =20 --=20 2.48.1