From nobody Fri Sep 25 04:43:45 2026 Received: from out-2uec-a113.jellyfish.systems (out-2uec-a113.jellyfish.systems [63.250.43.113]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6CD404B405A for ; Wed, 16 Sep 2026 16:16:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=63.250.43.113 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789575413; cv=none; b=WfbQ7l19jJGHmCcrDahJ7F05NSRTUhNbfSk07ktplLAo8nXC+ZVEcOIPN9e8dogm4pYDsDBybYMPfhR8rb+3yIX30rHWCIV2CxqpFRBlf+UmiyyQTIULNfpcvFvGi0RI0Z7sjV4P4S8EdMxYcHWZSZvGFatuvs62Hmaerl7MjuQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789575413; c=relaxed/simple; bh=m85Ei046pGaHQ1bb2sq9uqBWK3qTOxvlC9tEX2125B4=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=i8EmeFTg7vgQNxAxiFO5DEA+rl7rvAIV4VxswkT8s1nbeqrRE7d/cioH1SvKpf5VwGYWkkM51BPzpCM4m/KVEluTau1QxbDeF6oHJJDTxg9abxnb9NnDhAA6iultn9c05Yy4x1zLO7fAR6z1NpoJVUFOUaTXdlKiHXLLcLT3nH8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=tychen.cc; spf=pass smtp.mailfrom=tychen.cc; dkim=pass (2048-bit key) header.d=tychen.cc header.i=@tychen.cc header.b=POi3uZcn; arc=none smtp.client-ip=63.250.43.113 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=tychen.cc Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tychen.cc Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=tychen.cc header.i=@tychen.cc header.b="POi3uZcn" Received: from fedora (unknown [117.147.120.98]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mail.spacemail.com (Postfix) with ESMTPSA id 4hlP5g4RXYz8sWN; Wed, 16 Sep 2026 16:11:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tychen.cc; s=spacemail; t=1789575098; bh=wdv0Ct5RL7y/bWSPN0HuVAGS4/ZmWxUYIOG19TOM51o=; h=From:To:Cc:Subject:Date:From; b=POi3uZcnfcCafo/CX0jNjHwMbabcDk7bGh+wVVavY/47RpHGu9G14EZpsIgUfgJfl A95k3z1O9lYGzLHhNbx2C02LW+Cc2s0neFYBKF8kJSogvFErUPDw5mcAZAO2CGFlTI ivTc9Us7EgnyYUh1ZciTAs2f6kpQT2CX2jiY40YCs2OaW690a5Dx3iq37nQPYgNTbg 1QIVrh+lcN4vzJvRdLKF4w55q8zGELXF1Xuo+6B5UNn/OugVoWPBk1dKbyWEPqnPsz qx0Jx6cO6YVnKiJw+S3qbDjXero8KyG9VrO/EdSqja1icT/dA6FUivWrWC/PGk5Jml l2q/btvVoR3dA== From: Tianyi Chen To: Namhyung Kim , Arnaldo Carvalho de Melo Cc: irogers@google.com, swapnil.sapkal@amd.com, ravi.bangoria@amd.com, linux-perf-users@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH] perf sched stats: Reject mismatched or incomplete snapshots Date: Thu, 17 Sep 2026 00:11:25 +0800 Message-ID: <20260916161125.2548499-1-hi@tychen.cc> X-Mailer: git-send-email 2.55.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Envelope-From: hi@tychen.cc Content-Type: text/plain; charset="utf-8" The before and after records are paired by list position. If a CPU or domain disappears, counters can be subtracted from a different record or left as absolute values. An extra record can also advance the cursor past the list. Identify the second snapshot by its timestamp or CPU ordering, and require matching CPU/domain IDs and versions before subtracting. Check that every record has a counterpart before printing, and propagate errors from either input of diff. Add a shell test with synthetic snapshots, including equal timestamps, CPU filtering, and missing or reordered CPU/domain records. Fixes: 5a357ae6ad63 ("perf sched stats: Add support for report subcommand") Assisted-by: LLM Signed-off-by: Tianyi Chen --- The new shell test passes with GCC and Clang ASan/UBSan and fails on unpatched perf. Live and record/report/diff checks with temporary /proc/schedstat fixtures also pass. tools/perf/Documentation/perf-sched.txt | 6 +- tools/perf/builtin-sched.c | 189 ++++++++++++------ tools/perf/tests/shell/schedstat_snapshots.sh | 155 ++++++++++++++ 3 files changed, 283 insertions(+), 67 deletions(-) create mode 100755 tools/perf/tests/shell/schedstat_snapshots.sh diff --git a/tools/perf/Documentation/perf-sched.txt b/tools/perf/Documenta= tion/perf-sched.txt index 4da06215163a..fe0abef2bd13 100644 --- a/tools/perf/Documentation/perf-sched.txt +++ b/tools/perf/Documentation/perf-sched.txt @@ -95,8 +95,10 @@ There are several variants of 'perf sched': events, ``try_to_wakeup()`` call among others. This is useful in unders= tanding the scheduler behavior for the workload. =20 - Note: The tool will not give correct results if there is topological re= ordering or - online/offline of cpus in between capturing snapshots of `/proc/s= chedstat`. + Note: Reports reject incomplete snapshots and mismatched CPU or domain = IDs. + Topology changes that retain the same IDs, including CPUs going o= ffline + and returning online between snapshots, cannot be detected and ma= y still + produce incorrect results. =20 Example usage: perf sched stats record -- sleep 1 diff --git a/tools/perf/builtin-sched.c b/tools/perf/builtin-sched.c index dd39a4fb6c7a..9e7d170b30ad 100644 --- a/tools/perf/builtin-sched.c +++ b/tools/perf/builtin-sched.c @@ -4281,18 +4281,18 @@ struct schedstat_cpu { }; =20 static struct list_head cpu_head =3D LIST_HEAD_INIT(cpu_head); -static struct schedstat_cpu *cpu_second_pass; -static struct schedstat_domain *domain_second_pass; +static struct list_head *cpu_second_pass; +static struct list_head *domain_second_pass; +static u64 schedstat_timestamp; static bool after_workload_flag; static bool verbose_field; =20 static void free_schedstat(struct list_head *head); =20 -static void store_schedstat_cpu_diff(struct schedstat_cpu *after_workload) +static void store_schedstat_cpu_diff(struct perf_record_schedstat_cpu *bef= ore, + struct perf_record_schedstat_cpu *after) { - struct perf_record_schedstat_cpu *before =3D cpu_second_pass->cpu_data; - struct perf_record_schedstat_cpu *after =3D after_workload->cpu_data; - __u16 version =3D after_workload->cpu_data->version; + __u16 version =3D after->version; =20 #define CPU_FIELD(_type, _name, _desc, _format, _is_pct, _pct_of, _ver) \ (before->_ver._name =3D after->_ver._name - before->_ver._name) @@ -4308,11 +4308,10 @@ static void store_schedstat_cpu_diff(struct schedst= at_cpu *after_workload) #undef CPU_FIELD } =20 -static void store_schedstat_domain_diff(struct schedstat_domain *after_wor= kload) +static void store_schedstat_domain_diff(struct perf_record_schedstat_domai= n *before, + struct perf_record_schedstat_domain *after) { - struct perf_record_schedstat_domain *before =3D domain_second_pass->domai= n_data; - struct perf_record_schedstat_domain *after =3D after_workload->domain_dat= a; - __u16 version =3D after_workload->domain_data->version; + __u16 version =3D after->version; =20 #define DOMAIN_FIELD(_type, _name, _desc, _format, _is_jiffies, _ver) \ (before->_ver._name =3D after->_ver._name - before->_ver._name) @@ -4814,12 +4813,35 @@ static int show_schedstat_data(struct list_head *he= ad1, struct cpu_domain_map ** * other after completion of the workload. The above linked list stores th= e diff of the cpu and * domain statistics. */ +static int schedstat_snapshot_error(void) +{ + pr_err("Incompatible or incomplete schedstat snapshots\n"); + return -EINVAL; +} + +static bool schedstat_domains_complete(void) +{ + struct schedstat_cpu *cpu; + + if (!domain_second_pass) + return true; + cpu =3D list_entry(cpu_second_pass, struct schedstat_cpu, cpu_list); + return domain_second_pass =3D=3D &cpu->domain_head; +} + +static int schedstat_snapshots_complete(void) +{ + if (!after_workload_flag || !cpu_second_pass || + cpu_second_pass->next !=3D &cpu_head || !schedstat_domains_complete()) + return schedstat_snapshot_error(); + return 0; +} + static int perf_sched__process_schedstat(const struct perf_tool *tool __ma= ybe_unused, struct perf_session *session __maybe_unused, union perf_event *event) { struct perf_cpu this_cpu; - static __u32 initial_cpu; =20 switch (event->header.type) { case PERF_RECORD_SCHEDSTAT_CPU: @@ -4836,63 +4858,91 @@ static int perf_sched__process_schedstat(const stru= ct perf_tool *tool __maybe_un return 0; =20 if (event->header.type =3D=3D PERF_RECORD_SCHEDSTAT_CPU) { - struct schedstat_cpu *temp =3D zalloc(sizeof(*temp)); - - if (!temp) - return -ENOMEM; - - temp->cpu_data =3D zalloc(sizeof(*temp->cpu_data)); - if (!temp->cpu_data) - return -ENOMEM; + struct perf_record_schedstat_cpu *data =3D &event->schedstat_cpu; + struct schedstat_cpu *cpu; =20 - memcpy(temp->cpu_data, &event->schedstat_cpu, sizeof(*temp->cpu_data)); - - if (!list_empty(&cpu_head) && temp->cpu_data->cpu =3D=3D initial_cpu) - after_workload_flag =3D true; - - if (!after_workload_flag) { - if (list_empty(&cpu_head)) - initial_cpu =3D temp->cpu_data->cpu; - - list_add_tail(&temp->cpu_list, &cpu_head); - INIT_LIST_HEAD(&temp->domain_head); - } else { - if (temp->cpu_data->cpu =3D=3D initial_cpu) { - cpu_second_pass =3D list_first_entry(&cpu_head, struct schedstat_cpu, - cpu_list); - cpu_second_pass->cpu_data->timestamp =3D - temp->cpu_data->timestamp - cpu_second_pass->cpu_data->timestamp; - } else { - cpu_second_pass =3D list_next_entry(cpu_second_pass, cpu_list); + if (list_empty(&cpu_head)) { + after_workload_flag =3D false; + cpu_second_pass =3D &cpu_head; + domain_second_pass =3D NULL; + schedstat_timestamp =3D data->timestamp; + } else if (!after_workload_flag) { + cpu =3D list_last_entry(&cpu_head, struct schedstat_cpu, cpu_list); + /* Snapshots share a timestamp and list CPUs in increasing order. */ + if (data->timestamp !=3D schedstat_timestamp || + data->cpu <=3D cpu->cpu_data->cpu) { + after_workload_flag =3D true; + schedstat_timestamp =3D data->timestamp; } - domain_second_pass =3D list_first_entry(&cpu_second_pass->domain_head, - struct schedstat_domain, domain_list); - store_schedstat_cpu_diff(temp); - free(temp->cpu_data); - free(temp); } - } else if (event->header.type =3D=3D PERF_RECORD_SCHEDSTAT_DOMAIN) { - struct schedstat_cpu *cpu_tail; - struct schedstat_domain *temp =3D zalloc(sizeof(*temp)); =20 - if (!temp) - return -ENOMEM; + if (after_workload_flag) { + if (data->timestamp !=3D schedstat_timestamp || !schedstat_domains_comp= lete()) + return schedstat_snapshot_error(); + cpu_second_pass =3D cpu_second_pass->next; + if (cpu_second_pass =3D=3D &cpu_head) + return schedstat_snapshot_error(); + cpu =3D list_entry(cpu_second_pass, struct schedstat_cpu, cpu_list); + if (data->cpu !=3D cpu->cpu_data->cpu || + data->version !=3D cpu->cpu_data->version || + data->timestamp < cpu->cpu_data->timestamp) + return schedstat_snapshot_error(); + cpu->cpu_data->timestamp =3D data->timestamp - cpu->cpu_data->timestamp; + store_schedstat_cpu_diff(cpu->cpu_data, data); + domain_second_pass =3D cpu->domain_head.next; + return 0; + } =20 - temp->domain_data =3D zalloc(sizeof(*temp->domain_data)); - if (!temp->domain_data) + cpu =3D zalloc(sizeof(*cpu)); + if (!cpu) return -ENOMEM; + cpu->cpu_data =3D memdup(data, sizeof(*data)); + if (!cpu->cpu_data) { + free(cpu); + return -ENOMEM; + } + INIT_LIST_HEAD(&cpu->domain_head); + list_add_tail(&cpu->cpu_list, &cpu_head); + } else { + struct perf_record_schedstat_domain *data =3D &event->schedstat_domain; + struct schedstat_domain *domain; + struct schedstat_cpu *cpu; + + if (list_empty(&cpu_head) || data->timestamp !=3D schedstat_timestamp) + return schedstat_snapshot_error(); + if (after_workload_flag) { + cpu =3D list_entry(cpu_second_pass, struct schedstat_cpu, cpu_list); + if (domain_second_pass =3D=3D &cpu->domain_head) + return schedstat_snapshot_error(); + domain =3D list_entry(domain_second_pass, struct schedstat_domain, + domain_list); + if (data->cpu !=3D domain->domain_data->cpu || + data->domain !=3D domain->domain_data->domain || + data->version !=3D domain->domain_data->version) + return schedstat_snapshot_error(); + store_schedstat_domain_diff(domain->domain_data, data); + domain_second_pass =3D domain_second_pass->next; + return 0; + } =20 - memcpy(temp->domain_data, &event->schedstat_domain, sizeof(*temp->domain= _data)); - - if (!after_workload_flag) { - cpu_tail =3D list_last_entry(&cpu_head, struct schedstat_cpu, cpu_list); - list_add_tail(&temp->domain_list, &cpu_tail->domain_head); - } else { - store_schedstat_domain_diff(temp); - domain_second_pass =3D list_next_entry(domain_second_pass, domain_list); - free(temp->domain_data); - free(temp); + cpu =3D list_last_entry(&cpu_head, struct schedstat_cpu, cpu_list); + if (data->cpu !=3D cpu->cpu_data->cpu || data->version !=3D cpu->cpu_dat= a->version) + return schedstat_snapshot_error(); + if (!list_empty(&cpu->domain_head)) { + domain =3D list_last_entry(&cpu->domain_head, struct schedstat_domain, + domain_list); + if (data->domain <=3D domain->domain_data->domain) + return schedstat_snapshot_error(); } + domain =3D zalloc(sizeof(*domain)); + if (!domain) + return -ENOMEM; + domain->domain_data =3D memdup(data, sizeof(*data)); + if (!domain->domain_data) { + free(domain); + return -ENOMEM; + } + list_add_tail(&domain->domain_list, &cpu->domain_head); } =20 return 0; @@ -4947,6 +4997,8 @@ static int perf_sched__schedstat_report(struct perf_s= ched *sched) user_requested_cpus =3D evlist__core(session->evlist)->user_requested_cpu= s; =20 err =3D perf_session__process_events(session); + if (!err) + err =3D schedstat_snapshots_complete(); =20 if (!err) { setup_pager(); @@ -4976,7 +5028,7 @@ static int perf_sched__schedstat_diff(struct perf_sch= ed *sched, struct list_head cpu_head_ses0, cpu_head_ses1; struct perf_session *session[2]; struct perf_data data[2] =3D {0}; - int ret =3D 0, err =3D 0; + int ret =3D 0; static const char *defaults[] =3D { "perf.data.old", "perf.data", @@ -5009,8 +5061,10 @@ static int perf_sched__schedstat_diff(struct perf_sc= hed *sched, goto out_delete_ses0; } =20 - err =3D perf_session__process_events(session[0]); - if (err) { + ret =3D perf_session__process_events(session[0]); + if (!ret) + ret =3D schedstat_snapshots_complete(); + if (ret) { free_schedstat(&cpu_head); goto out_delete_ses0; } @@ -5028,8 +5082,10 @@ static int perf_sched__schedstat_diff(struct perf_sc= hed *sched, goto out_delete_ses1; } =20 - err =3D perf_session__process_events(session[1]); - if (err) { + ret =3D perf_session__process_events(session[1]); + if (!ret) + ret =3D schedstat_snapshots_complete(); + if (ret) { free_schedstat(&cpu_head); goto out_delete_ses1; } @@ -5152,6 +5208,9 @@ static int perf_sched__schedstat_live(struct perf_sch= ed *sched, user_requested_cpus); if (err) goto out; + err =3D schedstat_snapshots_complete(); + if (err) + goto out; =20 setup_pager(); =20 diff --git a/tools/perf/tests/shell/schedstat_snapshots.sh b/tools/perf/tes= ts/shell/schedstat_snapshots.sh new file mode 100755 index 000000000000..a4bdf2673d43 --- /dev/null +++ b/tools/perf/tests/shell/schedstat_snapshots.sh @@ -0,0 +1,155 @@ +#!/bin/sh +# Validate CPU and domain pairing in perf sched stats snapshots +# SPDX-License-Identifier: GPL-2.0 + +set -e + +# shellcheck source=3Dlib/setup_python.sh +. "$(dirname "$0")/lib/setup_python.sh" + +if ! perf version --build-options | grep -q 'libtraceevent:.*on'; then + echo "[Skip] perf sched requires libtraceevent" + exit 2 +fi + +$PYTHON - <<'PY' +import os +import re +import struct +import subprocess +import sys +import tempfile + +# Native-endian perf.data with only NRCPUS and CPU_DOMAIN_INFO features. +endian =3D '<' if sys.byteorder =3D=3D 'little' else '>' + + +def pack(fmt, *values): + return struct.pack(endian + fmt, *values) + + +def string(value): + data =3D value.encode() + b'\0' + return pack('I', len(data)) + data + + +def cpu(cpu_id, timestamp, value, version): + return pack('IHHQIHH6I3Q', 85, 0, 72, timestamp, cpu_id, version, 0, + *([value] * 9)) + + +def domain(cpu_id, domain_id, timestamp, value, version): + # All supported versions use the largest union member's record size. + return pack('IHHQIHH45I4x', 86, 0, 208, timestamp, cpu_id, version, + domain_id, *([value] * 45)) + + +def snapshot(timestamp, value, version=3D17, cpus=3D(0, 1, 2), domains=3D(= 0, 1)): + records =3D [] + for cpu_id in cpus: + records.append(cpu(cpu_id, timestamp, value + cpu_id * 100, versio= n)) + for domain_id in domains: + records.append(domain(cpu_id, domain_id, timestamp, + value + cpu_id * 100, version)) + return records + + +def write_file(path, records, version=3D17): + metadata =3D pack('II', version, 2) + for cpu_id in range(3): + metadata +=3D pack('II', cpu_id, 2) + for domain_id in range(2): + metadata +=3D pack('I', domain_id) + if version >=3D 17: + metadata +=3D string('SMT' if domain_id =3D=3D 0 else 'MC') + metadata +=3D string('7') + string('0-2') + features =3D [pack('II', 3, 3), metadata] + data =3D b''.join(records) + offset =3D 104 + len(data) + 16 * len(features) + sections =3D b'' + for feature in features: + sections +=3D pack('QQ', offset, len(feature)) + offset +=3D len(feature) + header =3D pack('13Q', 0x32454c4946524550, 104, 144, 104, 0, + 104, len(data), 0, 0, (1 << 7) | (1 << 32), 0, 0, 0) + with open(path, 'wb') as output: + output.write(header + data + sections + b''.join(features)) + + +def run(args, valid, domains=3DTrue): + result =3D subprocess.run(['perf', 'sched', 'stats'] + args, + stdout=3Dsubprocess.PIPE, stderr=3Dsubprocess.= PIPE, + text=3DTrue, timeout=3D10) + if valid: + assert result.returncode =3D=3D 0, result.stderr + assert re.search(r'^yld_count\s+:\s+10\b', result.stdout, re.M), r= esult.stdout + if domains: + assert re.search(r'^busy_lb_count\s+:\s+10\b', result.stdout, = re.M), result.stdout + else: + assert 'busy_lb_count' not in result.stdout, result.stdout + else: + assert result.returncode > 0, (args, result.returncode, result.std= out) + assert 'Incompatible or incomplete schedstat snapshots' in result.= stderr + assert not result.stdout, result.stdout + assert 'Sanitizer' not in result.stderr, result.stderr + + +with tempfile.TemporaryDirectory(prefix=3D'perf-schedstat-') as directory: + good =3D os.path.join(directory, 'good.data') + test =3D os.path.join(directory, 'test.data') + before =3D snapshot(100, 1000) + after =3D snapshot(200, 1010) + write_file(good, before + after) + + for version in (15, 16, 17): + for timestamp in (100, 200): + write_file(test, snapshot(100, 1000, version) + + snapshot(timestamp, 1010, version), version) + run(['report', '-C', '0,1,2', '-i', test], True) + run(['diff', test, test], True) + print('Matching snapshots, including equal timestamps: [Success]') + + write_file(test, snapshot(100, 1000, domains=3D()) + + snapshot(200, 1010, domains=3D())) + run(['report', '-C', '0,1,2', '-i', test], True, domains=3DFalse) + run(['diff', test, test], True, domains=3DFalse) + print('CPUs without domains: [Success]') + + write_file(test, before + snapshot(200, 1010, cpus=3D(0, 1))) + run(['report', '-C', '0,1', '-i', test], True) + run(['report', '-C', '1', '-i', good], True) + write_file(test, snapshot(100, 1000, cpus=3D(0, 1)) + + snapshot(200, 1010, cpus=3D(0, 1))) + run(['diff', good, test], True) + print('CPU filtering and different CPU sets across files: [Success]') + + write_file(test, snapshot(100, 0xfffffffa, cpus=3D(0,)) + + snapshot(200, 4, cpus=3D(0,))) + run(['report', '-C', '0', '-i', test], True) + print('Wrapping 32-bit counters: [Success]') + + cases =3D { + 'missing first CPU': before + snapshot(200, 1010, cpus=3D(1, 2)), + 'missing middle CPU': before + snapshot(200, 1010, cpus=3D(0, 2)), + 'missing last CPU': before + snapshot(200, 1010, cpus=3D(0, 1)), + 'added CPU': snapshot(100, 1000, cpus=3D(0, 1)) + after, + 'reordered CPUs': before + snapshot(200, 1010, cpus=3D(0, 2, 1)), + 'equal timestamp, missing first CPU': before + snapshot(100, 1010,= cpus=3D(1, 2)), + 'missing first domain': before + snapshot(200, 1010, domains=3D(1,= )), + 'missing last domain': before + after[:-1], + 'added domain': snapshot(100, 1000, domains=3D(0,)) + after, + 'reordered domains': before + snapshot(200, 1010, domains=3D(1, 0)= ), + 'domain without CPU': before[1:] + after, + 'wrong domain CPU': before + [after[0], domain(1, 0, 200, 1010, 17= )] + after[2:], + 'changed version': before + snapshot(200, 1010, version=3D16), + 'backwards timestamp': before + snapshot(50, 1010), + 'third snapshot': before + after + snapshot(300, 1020), + 'missing second snapshot': before, + } + for name, records in cases.items(): + write_file(test, records) + run(['report', '-C', '0,1,2', '-i', test], False) + run(['diff', test, good], False) + run(['diff', good, test], False) + print(name + ': [Success]') +PY base-commit: 91b0782fc9e9d2f0a40b5256146e014802fdbb36 --=20 2.55.0