From nobody Thu Dec 18 00:45:41 2025 Received: from madrid.collaboradmins.com (madrid.collaboradmins.com [46.235.227.194]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8431B134DE; Wed, 24 Jul 2024 21:40:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=46.235.227.194 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721857249; cv=none; b=E2w22v1p2FTCtqk0cbO6VUTsmEMwy6FEQedV4rkGd0lhOU8Bs5owEm7cc7T2aVhhlEC7R/Lx8eLsM7gEFFno3Jfra+4mpDceu1AMhIxFvGe209lLL2jxXlSoaJ8k1GPGCR8hp50Ge5UGxTgNVRyVBoYds8GmzxrVzQiM54L43as= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721857249; c=relaxed/simple; bh=KLkBXKWN3Xo1+VCP4LbzIF5m/qy0hXW32uXWVmTNOzU=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:To:Cc; b=hJJzXGn9G+V5spTfWs0h8DHHCaDCkB60pmD2KjMjMawNEHs79Rh7CUr1TyazvhKsHRLshiw97KtYiI8hRCZYfUVvAkuiV+7B/dYHHh+Ysw+laiiR2oHwbyFyAs3TZwpeD+4vDmTx2KC8PmrhuLudcHt7SGuE8+midcWuT7l79KQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=collabora.com; spf=pass smtp.mailfrom=collabora.com; dkim=pass (2048-bit key) header.d=collabora.com header.i=@collabora.com header.b=r5J+uoIP; arc=none smtp.client-ip=46.235.227.194 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=collabora.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=collabora.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=collabora.com header.i=@collabora.com header.b="r5J+uoIP" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=collabora.com; s=mail; t=1721857245; bh=KLkBXKWN3Xo1+VCP4LbzIF5m/qy0hXW32uXWVmTNOzU=; h=From:Date:Subject:To:Cc:From; b=r5J+uoIPWdJQ2kfk+IUSRhy0/1Zfky+jo56/S2Ti3y+4csBlV3yO5bliZF/v864ka DklXwTf03RUPoxwW0QQoffbpd3XWCyMrkt6HvUic3QrN9NuJhrp5aDGSB0RVLuzQ5y gurb/KQHxIvKwDdZWB8g8RXxdxpF1wFMpn5q/4fwcxTq3GPSs1QaW/ZPF7FQUMk8of 0AVPRn35uNjUUXzbhoAoffT0egTruafMMqBmq5JbB6v09yCXhD6M0p1gmZG6rax6Mu 2jlgR8UK290mShkqpQoqpUUPf/mc+H3J3vVG26O6ETdx/df/mrektR8f/sI4lCOW4X PhNJrEtwHEPZQ== Received: from [192.168.1.251] (zone.collabora.co.uk [167.235.23.81]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: nfraprado) by madrid.collaboradmins.com (Postfix) with ESMTPSA id EDD4E3780520; Wed, 24 Jul 2024 21:40:42 +0000 (UTC) From: =?utf-8?q?N=C3=ADcolas_F=2E_R=2E_A=2E_Prado?= Date: Wed, 24 Jul 2024 17:40:00 -0400 Subject: [PATCH RFC] kselftest: devices: Add test to detect missing devices Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20240724-kselftest-dev-exist-v1-1-9bc21aa761b5@collabora.com> X-B4-Tracking: v=1; b=H4sIAK90oWYC/6tWKk4tykwtVrJSqFYqSi3LLM7MzwNyDHUUlJIzE vPSU3UzU4B8JSMDIxMDcyMT3ezi1Jy0ktTiEt2U1DLd1IpMICspyTApOc3CwsTM1EQJqLOgKDU tswJsarRSkJuzUmxtLQDLQt8LagAAAA== To: Shuah Khan , Greg Kroah-Hartman Cc: "Bird, Tim" , Laura Nao , Saravana Kannan , "Rafael J. Wysocki" , AngeloGioacchino Del Regno , kernel@collabora.com, kernelci@lists.linux.dev, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, =?utf-8?q?N=C3=ADcolas_F=2E_R=2E_A=2E_Prado?= X-Mailer: b4 0.14.0 Introduce a new test to identify regressions causing devices to go missing on the system. For each bus and class on the system the test checks the number of devices present against a reference file, which needs to have been generated by the program at a previous point on a known-good kernel, and if there are missing devices they are reported. Signed-off-by: N=C3=ADcolas F. R. A. Prado --- Hi, Key points about this test: * Goal: Identify regressions causing devices to go missing on the system * Focus: * Ease of maintenance: the reference file is generated programatically * Minimum of false-positives: the script makes as few assumptions as poss= ible about the stability of device identifiers to ensure renames/refactors d= on't trigger false-positives * How it works: For each bus and class on the system the test checks the nu= mber of devices present against a reference file, which needs to have been generated by the program at a previous point on a known-good kernel, and = if there are missing devices they are reported. * Comparison to other tests: It might be possible(*) to replace the discove= rable devices test [1] with this. The benefits of this test is that it's easier to setup and maintain and has wider coverage of devices. Additional detail: * Having more devices on the running system than the reference does not cau= se a failure, but a warning is printed in that case to suggest that the refere= nce be updated. * Missing devices are detected per bus/class based on the number of devices. When the test fails, the known metadata for each of the expected and dete= cted devices is printed and some simple similitarity comparison is done to sug= gest the devices that are the most likely to be missing. * The proposed place to store the generated reference files is the 'platform-test-parameters' repository in KernelCI [2]. Example output: This is an example of a failing test case when one of the t= wo devices in the nvmem bus went missing: # Missing devices for subsystem 'nvmem': 1 (Expected 2, found 1) # =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D # Devices expected: # # uevent: # OF_NAME=3Defuse # OF_FULLNAME=3D/soc/efuse@11c10000 # OF_COMPATIBLE_0=3Dmediatek,mt8195-efuse # OF_COMPATIBLE_1=3Dmediatek,efuse # OF_COMPATIBLE_N=3D2 # # uevent: # OF_NAME=3Dflash # OF_FULLNAME=3D/soc/spi@1132c000/flash@0 # OF_COMPATIBLE_0=3Djedec,spi-nor # OF_COMPATIBLE_N=3D1 # # ----------------- # Devices found: # # uevent: # OF_NAME=3Defuse # OF_FULLNAME=3D/soc/efuse@11c10000 # OF_COMPATIBLE_0=3Dmediatek,mt8195-efuse # OF_COMPATIBLE_1=3Dmediatek,efuse # OF_COMPATIBLE_N=3D2 # # ----------------- # Devices missing (best guess): # # uevent: # OF_NAME=3Dflash # OF_FULLNAME=3D/soc/spi@1132c000/flash@0 # OF_COMPATIBLE_0=3Djedec,spi-nor # OF_COMPATIBLE_N=3D1 # # =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D not ok 19 bus.nvmem Example of how the data for these devices is encoded in the reference file: bus: ... nvmem: count: 2 devices: - info: uevent: 'OF_NAME=3Defuse =20 OF_FULLNAME=3D/soc/efuse@11c10000 =20 OF_COMPATIBLE_0=3Dmediatek,mt8195-efuse =20 OF_COMPATIBLE_1=3Dmediatek,efuse =20 OF_COMPATIBLE_N=3D2 =20 ' - info: uevent: 'OF_NAME=3Dflash =20 OF_FULLNAME=3D/soc/spi@1132c000/flash@0 =20 OF_COMPATIBLE_0=3Djedec,spi-nor =20 OF_COMPATIBLE_N=3D1 =20 ' (Full reference file: http://0x0.st/Xp60.yaml;) Caveat: Relying only on the count of devices in a subsystem makes the test susceptible to false-negatives eg. if a device goes missing and another in = the same subsystem is added the count will be the same so this regression won't= be reported. In order to avoid this we may include properties that must match individual devices, but we must be very careful (and it's why I haven't don= e it) since matching against properties that aren't guaranteed to be stable will introduce false-positives (ie. detecting false regressions) due to eventual renames. Some things to improve in the near future / gather feedback on: * (*): Currently this test only checks for the existence of devices. We cou= ld extend it to also encode into the reference which devices are bound to dr= ivers to be able to completely replace the discoverable devices probe kselftest= [1]. * Expanding identifying properties: Currently the properties that are stored (when present) in the reference for each device to be used for identifica= tion in the result output are uevent, device/uevent, firmware_node/uevent and = name. Suggestions of others properties to add are welcome. * Adding more filtering to reduce noise: * Ignoring buses/classes: Currently the devlink class is ignored by the t= est since it seems like a kernel internal detail that userspace doesn't act= ually care about. We should add others that are similar. * Ignoring non-devices: There can be entries in /sys/class/ that aren't devices. For now we're filtering down to only symlinks, but there might= be a better way. * As mentioned in the caveat section above we may want to add actual matchi= ng of devices based on properties to avoid false-negatives if we identify suitable properties. * It would be nice to have an option in the program to compare a newer refe= rence to an older one to make it easier for the user to see the differences and decide if the new reference is ok. * Since the reference file is not supposed to be manually edited, JSON migh= t be a better choice than YAML since it is included in the python standard lib= rary. Let me know your thoughts. Thanks, N=C3=ADcolas [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree= /tools/testing/selftests/devices/probe/test_discoverable_devices.py?id=3D0d= ebb20c5c812f8750c20c3406bc94a1e8ea4742 [2] https://github.com/kernelci/platform-test-parameters --- tools/testing/selftests/Makefile | 1 + tools/testing/selftests/devices/exist/Makefile | 3 + tools/testing/selftests/devices/exist/exist.py | 268 +++++++++++++++++++++= ++++ 3 files changed, 272 insertions(+) diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Mak= efile index bc8fe9e8f7f2..9c49b5ec5bef 100644 --- a/tools/testing/selftests/Makefile +++ b/tools/testing/selftests/Makefile @@ -14,6 +14,7 @@ TARGETS +=3D cpufreq TARGETS +=3D cpu-hotplug TARGETS +=3D damon TARGETS +=3D devices/error_logs +TARGETS +=3D devices/exist TARGETS +=3D devices/probe TARGETS +=3D dmabuf-heaps TARGETS +=3D drivers/dma-buf diff --git a/tools/testing/selftests/devices/exist/Makefile b/tools/testing= /selftests/devices/exist/Makefile new file mode 100644 index 000000000000..3075cac32092 --- /dev/null +++ b/tools/testing/selftests/devices/exist/Makefile @@ -0,0 +1,3 @@ +TEST_PROGS :=3D exist.py + +include ../../lib.mk diff --git a/tools/testing/selftests/devices/exist/exist.py b/tools/testing= /selftests/devices/exist/exist.py new file mode 100755 index 000000000000..8241b2fabc8e --- /dev/null +++ b/tools/testing/selftests/devices/exist/exist.py @@ -0,0 +1,268 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2024 Collabora Ltd + +# * Goal: Identify regressions causing devices to go missing on the system +# * Focus: +# * Ease of maintenance: the reference file is generated programatically +# * Minimum of false-positives: the script makes as few assumptions as +# possible about the stability of device identifiers to ensure +# renames/refactors don't trigger false-positives +# * How it works: For each bus and class on the system the test checks the +# number of devices present against a reference file, which needs to hav= e been +# generated by the program at a previous point on a known-good kernel, a= nd if +# there are missing devices they are reported. + +import os +import sys +import argparse + +import yaml + +# Allow ksft module to be imported from different directory +this_dir =3D os.path.dirname(os.path.realpath(__file__)) +sys.path.append(os.path.join(this_dir, "../../kselftest/")) + +import ksft + + +def generate_devs_obj(): + obj =3D {} + + device_sources =3D [ + { + "base_dir": "/sys/class", + "add_path": "", + "key_name": "class", + "ignored": ["devlink"], + }, + { + "base_dir": "/sys/bus", + "add_path": "devices", + "key_name": "bus", + "ignored": [], + }, + ] + + properties =3D sorted(["uevent", "device/uevent", "firmware_node/ueven= t", "name"]) + + for source in device_sources: + source_subsystems =3D {} + for subsystem in sorted(os.listdir(source["base_dir"])): + if subsystem in source["ignored"]: + continue + + devs_path =3D os.path.join(source["base_dir"], subsystem, sour= ce["add_path"]) + dev_dirs =3D [dev for dev in os.scandir(devs_path) if dev.is_s= ymlink()] + devs_data =3D [] + for dev_dir in dev_dirs: + dev_path =3D os.path.join(devs_path, dev_dir) + dev_data =3D {"info": {}} + for prop in properties: + if os.path.isfile(os.path.join(dev_path, prop)): + with open(os.path.join(dev_path, prop)) as f: + dev_data["info"][prop] =3D f.read() + devs_data.append(dev_data) + if len(dev_dirs): + source_subsystems[subsystem] =3D { + "count": len(dev_dirs), + "devices": devs_data, + } + obj[source["key_name"]] =3D source_subsystems + + return obj + + +def commented(s): + return s.replace("\n", "\n# ") + + +def indented(s, n): + return " " * n + s.replace("\n", "\n" + " " * n) + + +def stripped(s): + return s.strip("\n") + + +def devices_difference(dev1, dev2): + difference =3D 0 + + for prop in dev1["info"].keys(): + for l1, l2 in zip( + dev1["info"].get(prop, "").split("\n"), + dev2["info"].get(prop, "").split("\n"), + ): + if l1 !=3D l2: + difference +=3D 1 + return difference + + +def guess_missing_devices(cur_devs_subsystem, ref_devs_subsystem): + # Detect what devices on the current system are the most similar to de= vices + # on the reference one by one until the leftovers are the most dissimi= lar + # devices and therefore most likely the missing ones. + found_count =3D cur_devs_subsystem["count"] + expected_count =3D ref_devs_subsystem["count"] + missing_count =3D found_count - expected_count + + diffs =3D [] + for cur_d in cur_devs_subsystem["devices"]: + for ref_d in ref_devs_subsystem["devices"]: + diffs.append((devices_difference(cur_d, ref_d), cur_d, ref_d)) + + diffs.sort(key=3Dlambda x: x[0]) + + assigned_ref_devs =3D [] + assigned_cur_devs =3D [] + for diff in diffs: + if len(assigned_ref_devs) >=3D expected_count - missing_count: + break + if diff[1] in assigned_cur_devs or diff[2] in assigned_ref_devs: + continue + assigned_cur_devs.append(diff[1]) + assigned_ref_devs.append(diff[2]) + + missing_devices =3D [] + for d in ref_devs_subsystem["devices"]: + if d not in assigned_ref_devs: + missing_devices.append(d) + + return missing_devices + + +def dump_devices_info(cur_devs_subsystem, ref_devs_subsystem): + def dump_device_info(dev): + for name, val in dev["info"].items(): + ksft.print_msg(indented(name + ":", 2)) + val =3D stripped(val) + if val: + ksft.print_msg(commented(indented(val, 4))) + ksft.print_msg("") + + ksft.print_msg("=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D") + ksft.print_msg("Devices expected:") + ksft.print_msg("") + for d in ref_devs_subsystem["devices"]: + dump_device_info(d) + ksft.print_msg("-----------------") + ksft.print_msg("Devices found:") + ksft.print_msg("") + for d in cur_devs_subsystem["devices"]: + dump_device_info(d) + ksft.print_msg("-----------------") + ksft.print_msg("Devices missing (best guess):") + ksft.print_msg("") + missing_devices =3D guess_missing_devices(cur_devs_subsystem, ref_devs= _subsystem) + for d in missing_devices: + dump_device_info(d) + ksft.print_msg("=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D") + + +def run_test(ref_filename): + ksft.print_msg(f"Using reference file: {ref_filename}") + + with open(ref_filename) as f: + ref_devs_obj =3D yaml.safe_load(f) + + num_tests =3D 0 + for dev_source in ref_devs_obj.values(): + num_tests +=3D len(dev_source) + ksft.set_plan(num_tests) + + cur_devs_obj =3D generate_devs_obj() + + reference_outdated =3D False + + for source, ref_devs_source_obj in ref_devs_obj.items(): + for subsystem, ref_devs_subsystem_obj in ref_devs_source_obj.items= (): + test_name =3D f"{source}.{subsystem}" + if not ( + cur_devs_obj.get(source) and cur_devs_obj.get(source).get(= subsystem) + ): + ksft.print_msg(f"Device subsystem '{subsystem}' missing") + ksft.test_result_fail(test_name) + continue + cur_devs_subsystem_obj =3D cur_devs_obj[source][subsystem] + + found_count =3D cur_devs_subsystem_obj["count"] + expected_count =3D ref_devs_subsystem_obj["count"] + if found_count < expected_count: + ksft.print_msg( + f"Missing devices for subsystem '{subsystem}': {expect= ed_count - found_count} (Expected {expected_count}, found {found_count})" + ) + dump_devices_info(cur_devs_subsystem_obj, ref_devs_subsyst= em_obj) + ksft.test_result_fail(test_name) + else: + ksft.test_result_pass(test_name) + if found_count > expected_count: + reference_outdated =3D True + + if len(cur_devs_obj[source]) > len(ref_devs_source_obj): + reference_outdated =3D True + + if reference_outdated: + ksft.print_msg( + "Warning: The current system contains more devices and/or subs= ystems than the reference. Updating the reference is recommended." + ) + + +def get_possible_ref_filenames(): + filenames =3D [] + + dt_board_compatible_file =3D "/proc/device-tree/compatible" + if os.path.exists(dt_board_compatible_file): + with open(dt_board_compatible_file) as f: + for line in f: + compatibles =3D [compat for compat in line.split("\0") if = compat] + filenames.extend(compatibles) + else: + dmi_id_dir =3D "/sys/devices/virtual/dmi/id" + vendor_dmi_file =3D os.path.join(dmi_id_dir, "sys_vendor") + product_dmi_file =3D os.path.join(dmi_id_dir, "product_name") + + with open(vendor_dmi_file) as f: + vendor =3D f.read().replace("\n", "") + with open(product_dmi_file) as f: + product =3D f.read().replace("\n", "") + + filenames =3D [vendor + "," + product] + + return filenames + + +def get_ref_filename(ref_dir): + chosen_ref_filename =3D "" + full_ref_paths =3D [os.path.join(ref_dir, f + ".yaml") for f in get_po= ssible_ref_filenames()] + for path in full_ref_paths: + if os.path.exists(path): + chosen_ref_filename =3D path + break + + if not chosen_ref_filename: + tried_paths =3D ",".join(["'" + p + "'" for p in full_ref_paths]) + ksft.print_msg(f"No matching reference file found (tried {tried_pa= ths})") + ksft.exit_fail() + + return chosen_ref_filename + + +parser =3D argparse.ArgumentParser() +parser.add_argument( + "--reference-dir", default=3D".", help=3D"Directory containing the ref= erence files" +) +parser.add_argument("--generate-reference", action=3D"store_true", help=3D= "Generate a reference file with the devices on the running system") +args =3D parser.parse_args() + +if args.generate_reference: + print(f"# Kernel version: {os.uname().release}") + print(yaml.dump(generate_devs_obj())) + sys.exit(0) + +ksft.print_header() + +ref_filename =3D get_ref_filename(args.reference_dir) + +run_test(ref_filename) + +ksft.finished() --- base-commit: 73399b58e5e5a1b28a04baf42e321cfcfc663c2f change-id: 20240724-kselftest-dev-exist-bb1bcf884654 Best regards, --=20 N=C3=ADcolas F. R. A. Prado