From nobody Sun Feb 8 22:06:33 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70E6DEB64DC for ; Tue, 27 Jun 2023 00:59:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230238AbjF0A7d (ORCPT ); Mon, 26 Jun 2023 20:59:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55614 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230079AbjF0A7I (ORCPT ); Mon, 26 Jun 2023 20:59:08 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 960E41FC4 for ; Mon, 26 Jun 2023 17:56:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687827370; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=E6YHVyNv9jllDW5EgfHaZ/VxsLpfT00bCuAF1yM8w0Q=; b=CKBJPC5FW7tJgyUOIrrg5ZE6bQrNBpzNr/i8s6M4/Uruhksu6+mrDbQK3luGtnXQ7uv3xM 4P11clny9CZdAMIKnM/o7YRNbywdmLTPq92QhD0lYGJ8MZ8jFhq3FvIVJsvkBQE+diTAe8 /7x9bnZBblrChZ3YFhPw2lIc2ETsHxU= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-354-PUu6twrkOO6E8zMAzzUyTg-1; Mon, 26 Jun 2023 20:56:07 -0400 X-MC-Unique: PUu6twrkOO6E8zMAzzUyTg-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C1D85891F21; Tue, 27 Jun 2023 00:56:06 +0000 (UTC) Received: from llong.com (unknown [10.22.33.159]) by smtp.corp.redhat.com (Postfix) with ESMTP id DC4F32166B25; Tue, 27 Jun 2023 00:56:05 +0000 (UTC) From: Waiman Long To: Tejun Heo , Zefan Li , Johannes Weiner , Jonathan Corbet , Shuah Khan Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Juri Lelli , Valentin Schneider , Frederic Weisbecker , Mrunal Patel , Ryan Phillips , Brent Rowsell , Peter Hunt , Phil Auld , Waiman Long Subject: [PATCH v3 9/9] cgroup/cpuset: Extend test_cpuset_prs.sh to test remote partition Date: Mon, 26 Jun 2023 20:55:29 -0400 Message-Id: <20230627005529.1564984-10-longman@redhat.com> In-Reply-To: <20230627005529.1564984-1-longman@redhat.com> References: <20230627005529.1564984-1-longman@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This patch extends the test_cpuset_prs.sh test script to support testing the new remote partition and use the new "cpuset.cpus.exclusive" file by adding new tests for them. In addition, the following changes are also made: 1) Run the state transition tests directly under root to ease testing of remote partition and remove the unneeded test column. 2) Add a column to for the list of expected isolated CPUs and compare it with the actual value by looking at the state of /sys/kernel/debug/sched/domains which will be available if the verbose flag is set. Signed-off-by: Waiman Long --- .../selftests/cgroup/test_cpuset_prs.sh | 398 ++++++++++++------ 1 file changed, 259 insertions(+), 139 deletions(-) diff --git a/tools/testing/selftests/cgroup/test_cpuset_prs.sh b/tools/test= ing/selftests/cgroup/test_cpuset_prs.sh index 2b5215cc599f..ace992d1ed9e 100755 --- a/tools/testing/selftests/cgroup/test_cpuset_prs.sh +++ b/tools/testing/selftests/cgroup/test_cpuset_prs.sh @@ -3,7 +3,7 @@ # # Test for cpuset v2 partition root state (PRS) # -# The sched verbose flag is set, if available, so that the console log +# The sched verbose flag can be optionally set so that the console log # can be examined for the correct setting of scheduling domain. # =20 @@ -22,27 +22,27 @@ WAIT_INOTIFY=3D$(cd $(dirname $0); pwd)/wait_inotify # Find cgroup v2 mount point CGROUP2=3D$(mount -t cgroup2 | head -1 | awk -e '{print $3}') [[ -n "$CGROUP2" ]] || skip_test "Cgroup v2 mount point not found!" +SUBPARTS_CPUS=3D$CGROUP2/.__DEBUG__.cpuset.cpus.subpartitions +CPULIST=3D$(cat $CGROUP2/cpuset.cpus.effective) =20 -CPUS=3D$(lscpu | grep "^CPU(s):" | sed -e "s/.*:[[:space:]]*//") -[[ $CPUS -lt 8 ]] && skip_test "Test needs at least 8 cpus available!" +NR_CPUS=3D$(lscpu | grep "^CPU(s):" | sed -e "s/.*:[[:space:]]*//") +[[ $NR_CPUS -lt 8 ]] && skip_test "Test needs at least 8 cpus available!" =20 # Set verbose flag and delay factor PROG=3D$1 -VERBOSE=3D +VERBOSE=3D0 DELAY_FACTOR=3D1 SCHED_DEBUG=3D while [[ "$1" =3D -* ]] do case "$1" in - -v) VERBOSE=3D1 + -v) ((VERBOSE++)) # Enable sched/verbose can slow thing down [[ $DELAY_FACTOR -eq 1 ]] && DELAY_FACTOR=3D2 - break ;; -d) DELAY_FACTOR=3D$2 shift - break ;; *) echo "Usage: $PROG [-v] [-d " exit @@ -52,7 +52,7 @@ do done =20 # Set sched verbose flag if available when "-v" option is specified -if [[ -n "$VERBOSE" && -d /sys/kernel/debug/sched ]] +if [[ $VERBOSE -gt 0 && -d /sys/kernel/debug/sched ]] then # Used to restore the original setting during cleanup SCHED_DEBUG=3D$(cat /sys/kernel/debug/sched/verbose) @@ -61,14 +61,26 @@ fi =20 cd $CGROUP2 echo +cpuset > cgroup.subtree_control + +# +# If cpuset has been set up and used in child cgroups, we may not be able = to +# create partition under root cgroup because of the CPU exclusivity rule. +# So we are going to skip the test if this is the case. +# [[ -d test ]] || mkdir test -cd test +echo 0-6 > test/cpuset.cpus +echo root > test/cpuset.cpus.partition +cat test/cpuset.cpus.partition | grep -q invalid +RESULT=3D$? +echo member > test/cpuset.cpus.partition +echo "" > test/cpuset.cpus +[[ $RESULT -eq 0 ]] && skip_test "Child cgroups are using cpuset!" =20 cleanup() { online_cpus + cd $CGROUP2 rmdir A1/A2/A3 A1/A2 A1 B1 > /dev/null 2>&1 - cd .. rmdir test > /dev/null 2>&1 [[ -n "$SCHED_DEBUG" ]] && echo "$SCHED_DEBUG" > /sys/kernel/debug/sched/verbose @@ -103,7 +115,7 @@ test_partition() [[ $? -eq 0 ]] || exit 1 ACTUAL_VAL=3D$(cat cpuset.cpus.partition) [[ $ACTUAL_VAL !=3D $EXPECTED_VAL ]] && { - echo "cpuset.cpus.partition: expect $EXPECTED_VAL, found $EXPECTED_VAL" + echo "cpuset.cpus.partition: expect $EXPECTED_VAL, found $ACTUAL_VAL" echo "Test FAILED" exit 1 } @@ -114,7 +126,7 @@ test_effective_cpus() EXPECTED_VAL=3D$1 ACTUAL_VAL=3D$(cat cpuset.cpus.effective) [[ "$ACTUAL_VAL" !=3D "$EXPECTED_VAL" ]] && { - echo "cpuset.cpus.effective: expect '$EXPECTED_VAL', found '$EXPECTED_VA= L'" + echo "cpuset.cpus.effective: expect '$EXPECTED_VAL', found '$ACTUAL_VAL'" echo "Test FAILED" exit 1 } @@ -139,6 +151,7 @@ test_add_proc() # test_isolated() { + cd $CGROUP2/test echo 2-3 > cpuset.cpus TYPE=3D$(cat cpuset.cpus.partition) [[ $TYPE =3D member ]] || echo member > cpuset.cpus.partition @@ -203,125 +216,163 @@ test_isolated() # # Cgroup test hierarchy # -# test -- A1 -- A2 -- A3 -# \- B1 +# root -- A1 -- A2 -- A3 +# +- B1 # -# P =3D set cpus.partition (0:member, 1:root, 2:isolated, -1:root inva= lid) +# P =3D set cpus.partition (0:member, 1:root, 2:isolated) # C =3D add cpu-list # S

=3D use prefix in subtree_control # T =3D put a task into cgroup -# O- =3D Write to CPU online file of +# O=3D =3D Write to CPU online file of # SETUP_A123_PARTITIONS=3D"C1-3:P1:S+ C2-3:P1:S+ C3:P1" TEST_MATRIX=3D( - # test old-A1 old-A2 old-A3 old-B1 new-A1 new-A2 new-A3 new-B1 fail ECPU= s Pstate - # ---- ------ ------ ------ ------ ------ ------ ------ ------ ---- ----= - ------ - " S+ C0-1 . . C2-3 S+ C4-5 . . 0 A2:0-= 1" - " S+ C0-1 . . C2-3 P1 . . . 0 " - " S+ C0-1 . . C2-3 P1:S+ C0-1:P1 . . 0 " - " S+ C0-1 . . C2-3 P1:S+ C1:P1 . . 0 " - " S+ C0-1:S+ . . C2-3 . . . P1 0 " - " S+ C0-1:P1 . . C2-3 S+ C1 . . 0 " - " S+ C0-1:P1 . . C2-3 S+ C1:P1 . . 0 " - " S+ C0-1:P1 . . C2-3 S+ C1:P1 . P1 0 " - " S+ C0-1:P1 . . C2-3 C4-5 . . . 0 A1:4-= 5" - " S+ C0-1:P1 . . C2-3 S+:C4-5 . . . 0 A1:4-= 5" - " S+ C0-1 . . C2-3:P1 . . . C2 0 " - " S+ C0-1 . . C2-3:P1 . . . C4-5 0 B1:4-= 5" - " S+ C0-3:P1:S+ C2-3:P1 . . . . . . 0 A1:0-= 1,A2:2-3" - " S+ C0-3:P1:S+ C2-3:P1 . . C1-3 . . . 0 A1:1,= A2:2-3" - " S+ C2-3:P1:S+ C3:P1 . . C3 . . . 0 A1:,A= 2:3 A1:P1,A2:P1" - " S+ C2-3:P1:S+ C3:P1 . . C3 P0 . . 0 A1:3,= A2:3 A1:P1,A2:P0" - " S+ C2-3:P1:S+ C2:P1 . . C2-4 . . . 0 A1:3-= 4,A2:2" - " S+ C2-3:P1:S+ C3:P1 . . C3 . . C0-2 0 A1:,B= 1:0-2 A1:P1,A2:P1" - " S+ $SETUP_A123_PARTITIONS . C2-3 . . . 0 A1:,A= 2:2,A3:3 A1:P1,A2:P1,A3:P1" + # old-A1 old-A2 old-A3 old-B1 new-A1 new-A2 new-A3 new-B1 fail ECPUs Pst= ate ISOLCPUS + # ------ ------ ------ ------ ------ ------ ------ ------ ---- ----- ---= --- -------- + " C0-1 . . C2-3 S+ C4-5 . . 0 A2:0-1" + " C0-1 . . C2-3 P1 . . . 0 " + " C0-1 . . C2-3 P1:S+ C0-1:P1 . . 0 " + " C0-1 . . C2-3 P1:S+ C1:P1 . . 0 " + " C0-1:S+ . . C2-3 . . . P1 0 " + " C0-1:P1 . . C2-3 S+ C1 . . 0 " + " C0-1:P1 . . C2-3 S+ C1:P1 . . 0 " + " C0-1:P1 . . C2-3 S+ C1:P1 . P1 0 " + " C0-1:P1 . . C2-3 C4-5 . . . 0 A1:4-5" + " C0-1:P1 . . C2-3 S+:C4-5 . . . 0 A1:4-5" + " C0-1 . . C2-3:P1 . . . C2 0 " + " C0-1 . . C2-3:P1 . . . C4-5 0 B1:4-5" + "C0-3:P1:S+ C2-3:P1 . . . . . . 0 A1:0-1,A2:= 2-3" + "C0-3:P1:S+ C2-3:P1 . . C1-3 . . . 0 A1:1,A2:2-= 3" + "C2-3:P1:S+ C3:P1 . . C3 . . . 0 A1:,A2:3 A= 1:P1,A2:P1" + "C2-3:P1:S+ C3:P1 . . C3 P0 . . 0 A1:3,A2:3 = A1:P1,A2:P0" + "C2-3:P1:S+ C2:P1 . . C2-4 . . . 0 A1:3-4,A2:= 2" + "C2-3:P1:S+ C3:P1 . . C3 . . C0-2 0 A1:,B1:0-2= A1:P1,A2:P1" + "$SETUP_A123_PARTITIONS . C2-3 . . . 0 A1:,A2:2,A= 3:3 A1:P1,A2:P1,A3:P1" =20 # CPU offlining cases: - " S+ C0-1 . . C2-3 S+ C4-5 . O2-0 0 A1:0-= 1,B1:3" - " S+ C0-3:P1:S+ C2-3:P1 . . O2-0 . . . 0 A1:0-= 1,A2:3" - " S+ C0-3:P1:S+ C2-3:P1 . . O2-0 O2-1 . . 0 A1:0-= 1,A2:2-3" - " S+ C0-3:P1:S+ C2-3:P1 . . O1-0 . . . 0 A1:0,= A2:2-3" - " S+ C0-3:P1:S+ C2-3:P1 . . O1-0 O1-1 . . 0 A1:0-= 1,A2:2-3" - " S+ C2-3:P1:S+ C3:P1 . . O3-0 O3-1 . . 0 A1:2,= A2:3 A1:P1,A2:P1" - " S+ C2-3:P1:S+ C3:P2 . . O3-0 O3-1 . . 0 A1:2,= A2:3 A1:P1,A2:P2" - " S+ C2-3:P1:S+ C3:P1 . . O2-0 O2-1 . . 0 A1:2,= A2:3 A1:P1,A2:P1" - " S+ C2-3:P1:S+ C3:P2 . . O2-0 O2-1 . . 0 A1:2,= A2:3 A1:P1,A2:P2" - " S+ C2-3:P1:S+ C3:P1 . . O2-0 . . . 0 A1:,A= 2:3 A1:P1,A2:P1" - " S+ C2-3:P1:S+ C3:P1 . . O3-0 . . . 0 A1:2,= A2: A1:P1,A2:P1" - " S+ C2-3:P1:S+ C3:P1 . . T:O2-0 . . . 0 A1:3,= A2:3 A1:P1,A2:P-1" - " S+ C2-3:P1:S+ C3:P1 . . . T:O3-0 . . 0 A1:2,= A2:2 A1:P1,A2:P-1" - " S+ $SETUP_A123_PARTITIONS . O1-0 . . . 0 A1:,A= 2:2,A3:3 A1:P1,A2:P1,A3:P1" - " S+ $SETUP_A123_PARTITIONS . O2-0 . . . 0 A1:1,= A2:,A3:3 A1:P1,A2:P1,A3:P1" - " S+ $SETUP_A123_PARTITIONS . O3-0 . . . 0 A1:1,= A2:2,A3: A1:P1,A2:P1,A3:P1" - " S+ $SETUP_A123_PARTITIONS . T:O1-0 . . . 0 A1:2-= 3,A2:2-3,A3:3 A1:P1,A2:P-1,A3:P-1" - " S+ $SETUP_A123_PARTITIONS . . T:O2-0 . . 0 A1:1,= A2:3,A3:3 A1:P1,A2:P1,A3:P-1" - " S+ $SETUP_A123_PARTITIONS . . . T:O3-0 . 0 A1:1,= A2:2,A3:2 A1:P1,A2:P1,A3:P-1" - " S+ $SETUP_A123_PARTITIONS . T:O1-0 O1-1 . . 0 A1:1,= A2:2,A3:3 A1:P1,A2:P1,A3:P1" - " S+ $SETUP_A123_PARTITIONS . . T:O2-0 O2-1 . 0 A1:1,= A2:2,A3:3 A1:P1,A2:P1,A3:P1" - " S+ $SETUP_A123_PARTITIONS . . . T:O3-0 O3-1 0 A1:1,= A2:2,A3:3 A1:P1,A2:P1,A3:P1" - " S+ $SETUP_A123_PARTITIONS . T:O1-0 O2-0 O1-1 . 0 A1:1,= A2:,A3:3 A1:P1,A2:P1,A3:P1" - " S+ $SETUP_A123_PARTITIONS . T:O1-0 O2-0 O2-1 . 0 A1:2-= 3,A2:2-3,A3:3 A1:P1,A2:P-1,A3:P-1" - - # test old-A1 old-A2 old-A3 old-B1 new-A1 new-A2 new-A3 new-B1 fail ECPU= s Pstate - # ---- ------ ------ ------ ------ ------ ------ ------ ------ ---- ----= - ------ + " C0-1 . . C2-3 S+ C4-5 . O2=3D0 0 A1:0-1,B= 1:3" + "C0-3:P1:S+ C2-3:P1 . . O2=3D0 . . . 0 A1:0-1,A= 2:3" + "C0-3:P1:S+ C2-3:P1 . . O2=3D0 O2=3D1 . . 0 A1:0-1= ,A2:2-3" + "C0-3:P1:S+ C2-3:P1 . . O1=3D0 . . . 0 A1:0,A2:= 2-3" + "C0-3:P1:S+ C2-3:P1 . . O1=3D0 O1=3D1 . . 0 A1:0-1= ,A2:2-3" + "C2-3:P1:S+ C3:P1 . . O3=3D0 O3=3D1 . . 0 A1:2,A= 2:3 A1:P1,A2:P1" + "C2-3:P1:S+ C3:P2 . . O3=3D0 O3=3D1 . . 0 A1:2,A= 2:3 A1:P1,A2:P2" + "C2-3:P1:S+ C3:P1 . . O2=3D0 O2=3D1 . . 0 A1:2,A= 2:3 A1:P1,A2:P1" + "C2-3:P1:S+ C3:P2 . . O2=3D0 O2=3D1 . . 0 A1:2,A= 2:3 A1:P1,A2:P2" + "C2-3:P1:S+ C3:P1 . . O2=3D0 . . . 0 A1:,A2:3= A1:P1,A2:P1" + "C2-3:P1:S+ C3:P1 . . O3=3D0 . . . 0 A1:2,A2:= A1:P1,A2:P1" + "C2-3:P1:S+ C3:P1 . . T:O2=3D0 . . . 0 A1:3,A2:= 3 A1:P1,A2:P-1" + "C2-3:P1:S+ C3:P1 . . . T:O3=3D0 . . 0 A1:2,A2:= 2 A1:P1,A2:P-1" + "$SETUP_A123_PARTITIONS . O1=3D0 . . . 0 A1:,A2:2= ,A3:3 A1:P1,A2:P1,A3:P1" + "$SETUP_A123_PARTITIONS . O2=3D0 . . . 0 A1:1,A2:= ,A3:3 A1:P1,A2:P1,A3:P1" + "$SETUP_A123_PARTITIONS . O3=3D0 . . . 0 A1:1,A2:= 2,A3: A1:P1,A2:P1,A3:P1" + "$SETUP_A123_PARTITIONS . T:O1=3D0 . . . 0 A1:2-3,A= 2:2-3,A3:3 A1:P1,A2:P-1,A3:P-1" + "$SETUP_A123_PARTITIONS . . T:O2=3D0 . . 0 A1:1,A2:= 3,A3:3 A1:P1,A2:P1,A3:P-1" + "$SETUP_A123_PARTITIONS . . . T:O3=3D0 . 0 A1:1,A2:= 2,A3:2 A1:P1,A2:P1,A3:P-1" + "$SETUP_A123_PARTITIONS . T:O1=3D0 O1=3D1 . . 0 A1:1,A= 2:2,A3:3 A1:P1,A2:P1,A3:P1" + "$SETUP_A123_PARTITIONS . . T:O2=3D0 O2=3D1 . 0 A1:1,A= 2:2,A3:3 A1:P1,A2:P1,A3:P1" + "$SETUP_A123_PARTITIONS . . . T:O3=3D0 O3=3D1 0 A1:1,A= 2:2,A3:3 A1:P1,A2:P1,A3:P1" + "$SETUP_A123_PARTITIONS . T:O1=3D0 O2=3D0 O1=3D1 . 0 A1:1= ,A2:,A3:3 A1:P1,A2:P1,A3:P1" + "$SETUP_A123_PARTITIONS . T:O1=3D0 O2=3D0 O2=3D1 . 0 A1:2= -3,A2:2-3,A3:3 A1:P1,A2:P-1,A3:P-1" + + # old-A1 old-A2 old-A3 old-B1 new-A1 new-A2 new-A3 new-B1 fail ECPUs Pst= ate ISOLCPUS + # ------ ------ ------ ------ ------ ------ ------ ------ ---- ----- ---= --- -------- + # + # Remote partition and cpuset.cpus.exclusive tests + # + " C0-3:S+ C1-3:S+ C2-3 . X2-3 . . . 0 A1:0-3,A2:= 1-3,A3:2-3,XA1:2-3" + " C0-3:S+ C1-3:S+ C2-3 . X2-3 X2-3:P2 . . 0 A1:0-1,A2:= 2-3,A3:2-3 A1:P0,A2:P2 2-3" + " C0-3:S+ C1-3:S+ C2-3 . X2-3 X3:P2 . . 0 A1:0-2,A2:= 3,A3:3 A1:P0,A2:P2 3" + " C0-3:S+ C1-3:S+ C2-3 . X2-3 X2-3 X2-3:P2 . 0 A1:0-1,A2:= 1,A3:2-3 A1:P0,A3:P2 2-3" + " C0-3:S+ C1-3:S+ C2-3 . X2-3 X2-3 X2-3:P2:C3 . 0 A1:0-2,A2:= 1-2,A3:3 A1:P0,A3:P2 3" + " C0-3:S+ C1-3:S+ C2-3 C2-3 . . . P2 0 A1:0-3,A2:= 1-3,A3:2-3,B1:2-3 A1:P0,A3:P0,B1:P-2" + " C0-3:S+ C1-3:S+ C2-3 C4-5 . . . P2 0 B1:4-5 B1:= P2 4-5" + " C0-3:S+ C1-3:S+ C2-3 C4 X2-3 X2-3 X2-3:P2 P2 0 A3:2-3,B1:= 4 A3:P2,B1:P2 2-4" + " C0-3:S+ C1-3:S+ C2-3 C4 X2-3 X2-3 X2-3:P2:C1-3 P2 0 A3:2-3,B1:= 4 A3:P2,B1:P2 2-4" + " C0-3:S+ C1-3:S+ C2-3 C4 X1-3 X1-3:P2 P2 . 0 A2:1,A3:2-= 3 A2:P2,A3:P2 1-3" + " C0-3:S+ C1-3:S+ C2-3 C4 X2-3 X2-3 X2-3:P2 P2:C4-5 0 A3:2-3,B1:= 4-5 A3:P2,B1:P2 2-5" + + # Nested remote/local partition tests + " C0-3:S+ C1-3:S+ C2-3 C4-5 X2-3 X2-3:P1 P2 P1 0 A1:0-1,A2:= ,A3:2-3,B1:4-5 \ + A1:P0,A2:P1,A3:P2,B1:P1 2-3" + " C0-3:S+ C1-3:S+ C2-3 C4 X2-3 X2-3:P1 P2 P1 0 A1:0-1,A2:= ,A3:2-3,B1:4 \ + A1:P0,A2:P1,A3:P2,B1:P1 2-4" + " C0-3:S+ C1-3:S+ C3 C4 X2-3 X2-3:P1 P2 P1 0 A1:0-1,A2:= 2,A3:3,B1:4 \ + A1:P0,A2:P1,A3:P2,B1:P1 2-4" + + # Remote partition offline tests + " C0-3:S+ C1-3:S+ C2-3 . X2-3 X2-3 X2-3:P2:O2=3D0 . 0 A1:0-1,A= 2:1,A3:3 A1:P0,A3:P2 2-3" + " C0-3:S+ C1-3:S+ C2-3 . X2-3 X2-3 X2-3:P2:O2=3D0 O2=3D1 0 A1:0-= 1,A2:1,A3:2-3 A1:P0,A3:P2 2-3" + " C0-3:S+ C1-3:S+ C3 . X2-3 X2-3 P2:O3=3D0 . 0 A1:0-2,A= 2:1-2,A3: A1:P0,A3:P2 3" + " C0-3:S+ C1-3:S+ C3 . X2-3 X2-3 T:P2:O3=3D0 . 0 A1:0-2,A= 2:1-2,A3:1-2 A1:P0,A3:P-2 3" + + # An invalidated remote partition cannot self-recover from hotplug + " C0-3:S+ C1-3:S+ C2 . X2-3 X2-3 T:P2:O2=3D0 O2=3D1 0 A1:0-3= ,A2:1-3,A3:2 A1:P0,A3:P-2" + + # old-A1 old-A2 old-A3 old-B1 new-A1 new-A2 new-A3 new-B1 fail ECPUs Pst= ate ISOLCPUS + # ------ ------ ------ ------ ------ ------ ------ ------ ---- ----- ---= --- -------- # # Incorrect change to cpuset.cpus invalidates partition root # # Adding CPUs to partition root that are not in parent's # cpuset.cpus is allowed, but those extra CPUs are ignored. - " S+ C2-3:P1:S+ C3:P1 . . . C2-4 . . 0 A1:,A= 2:2-3 A1:P1,A2:P1" + "C2-3:P1:S+ C3:P1 . . . C2-4 . . 0 A1:,A2:2-3= A1:P1,A2:P1" =20 # Taking away all CPUs from parent or itself if there are tasks # will make the partition invalid. - " S+ C2-3:P1:S+ C3:P1 . . T C2-3 . . 0 A1:2-= 3,A2:2-3 A1:P1,A2:P-1" - " S+ C3:P1:S+ C3 . . T P1 . . 0 A1:3,= A2:3 A1:P1,A2:P-1" - " S+ $SETUP_A123_PARTITIONS . T:C2-3 . . . 0 A1:2-= 3,A2:2-3,A3:3 A1:P1,A2:P-1,A3:P-1" - " S+ $SETUP_A123_PARTITIONS . T:C2-3:C1-3 . . . 0 A1:1,= A2:2,A3:3 A1:P1,A2:P1,A3:P1" + "C2-3:P1:S+ C3:P1 . . T C2-3 . . 0 A1:2-3,A2:= 2-3 A1:P1,A2:P-1" + " C3:P1:S+ C3 . . T P1 . . 0 A1:3,A2:3 = A1:P1,A2:P-1" + "$SETUP_A123_PARTITIONS . T:C2-3 . . . 0 A1:2-3,A2:= 2-3,A3:3 A1:P1,A2:P-1,A3:P-1" + "$SETUP_A123_PARTITIONS . T:C2-3:C1-3 . . . 0 A1:1,A2:2,= A3:3 A1:P1,A2:P1,A3:P1" =20 # Changing a partition root to member makes child partitions invalid - " S+ C2-3:P1:S+ C3:P1 . . P0 . . . 0 A1:2-= 3,A2:3 A1:P0,A2:P-1" - " S+ $SETUP_A123_PARTITIONS . C2-3 P0 . . 0 A1:2-= 3,A2:2-3,A3:3 A1:P1,A2:P0,A3:P-1" + "C2-3:P1:S+ C3:P1 . . P0 . . . 0 A1:2-3,A2:= 3 A1:P0,A2:P-1" + "$SETUP_A123_PARTITIONS . C2-3 P0 . . 0 A1:2-3,A2:= 2-3,A3:3 A1:P1,A2:P0,A3:P-1" =20 # cpuset.cpus can contains cpus not in parent's cpuset.cpus as long # as they overlap. - " S+ C2-3:P1:S+ . . . . C3-4:P1 . . 0 A1:2,= A2:3 A1:P1,A2:P1" + "C2-3:P1:S+ . . . . C3-4:P1 . . 0 A1:2,A2:3 = A1:P1,A2:P1" =20 # Deletion of CPUs distributed to child cgroup is allowed. - " S+ C0-1:P1:S+ C1 . C2-3 C4-5 . . . 0 A1:4-= 5,A2:4-5" + "C0-1:P1:S+ C1 . C2-3 C4-5 . . . 0 A1:4-5,A2:= 4-5" =20 # To become a valid partition root, cpuset.cpus must overlap parent's # cpuset.cpus. - " S+ C0-1:P1 . . C2-3 S+ C4-5:P1 . . 0 A1:0-= 1,A2:0-1 A1:P1,A2:P-1" + " C0-1:P1 . . C2-3 S+ C4-5:P1 . . 0 A1:0-1,A2:= 0-1 A1:P1,A2:P-1" =20 # Enabling partition with child cpusets is allowed - " S+ C0-1:S+ C1 . C2-3 P1 . . . 0 A1:0-= 1,A2:1 A1:P1" + " C0-1:S+ C1 . C2-3 P1 . . . 0 A1:0-1,A2:= 1 A1:P1" =20 # A partition root with non-partition root parent is invalid, but it # can be made valid if its parent becomes a partition root too. - " S+ C0-1:S+ C1 . C2-3 . P2 . . 0 A1:0-= 1,A2:1 A1:P0,A2:P-2" - " S+ C0-1:S+ C1:P2 . C2-3 P1 . . . 0 A1:0,= A2:1 A1:P1,A2:P2" + " C0-1:S+ C1 . C2-3 . P2 . . 0 A1:0-1,A2:= 1 A1:P0,A2:P-2" + " C0-1:S+ C1:P2 . C2-3 P1 . . . 0 A1:0,A2:1 = A1:P1,A2:P2" =20 # A non-exclusive cpuset.cpus change will invalidate partition and its si= blings - " S+ C0-1:P1 . . C2-3 C0-2 . . . 0 A1:0-= 2,B1:2-3 A1:P-1,B1:P0" - " S+ C0-1:P1 . . P1:C2-3 C0-2 . . . 0 A1:0-2,= B1:2-3 A1:P-1,B1:P-1" - " S+ C0-1 . . P1:C2-3 C0-2 . . . 0 A1:0-2,= B1:2-3 A1:P0,B1:P-1" + " C0-1:P1 . . C2-3 C0-2 . . . 0 A1:0-2,B1:= 2-3 A1:P-1,B1:P0" + " C0-1:P1 . . P1:C2-3 C0-2 . . . 0 A1:0-2,B1:= 2-3 A1:P-1,B1:P-1" + " C0-1 . . P1:C2-3 C0-2 . . . 0 A1:0-2,B1:= 2-3 A1:P0,B1:P-1" =20 - # test old-A1 old-A2 old-A3 old-B1 new-A1 new-A2 new-A3 new-B1 fail ECPU= s Pstate - # ---- ------ ------ ------ ------ ------ ------ ------ ------ ---- ----= - ------ + # old-A1 old-A2 old-A3 old-B1 new-A1 new-A2 new-A3 new-B1 fail ECPUs Pst= ate ISOLCPUS + # ------ ------ ------ ------ ------ ------ ------ ------ ---- ----- ---= --- -------- # Failure cases: =20 # A task cannot be added to a partition with no cpu - " S+ C2-3:P1:S+ C3:P1 . . O2-0:T . . . 1 A1:,A= 2:3 A1:P1,A2:P1" + "C2-3:P1:S+ C3:P1 . . O2=3D0:T . . . 1 A1:,A2:3= A1:P1,A2:P1" + + # cpuset.cpus.exclusive must be a subset of cpuset.cpus & parent's cpuset= .cpus.exclusive + " C0-3:S+ C1-3:S+ C2-3 . X2-4 . . . 1" + " C0-3:S+ C1-3:S+ C2-3 . X1-2 X2-3 . . 1" ) =20 # # Write to the cpu online file -# $1 - - where =3D cpu number, value to be written +# $1 - =3D where =3D cpu number, value to be written # write_cpu_online() { - CPU=3D${1%-*} - VAL=3D${1#*-} + CPU=3D${1%=3D*} + VAL=3D${1#*=3D} CPUFILE=3D//sys/devices/system/cpu/cpu${CPU}/online if [[ $VAL -eq 0 ]] then @@ -349,11 +400,12 @@ set_ctrl_state() TMPMSG=3D/tmp/.msg_$$ CGRP=3D$1 STATE=3D$2 - SHOWERR=3D${3}${VERBOSE} + SHOWERR=3D${3} CTRL=3D${CTRL:=3D$CONTROLLER} HASERR=3D0 REDIRECT=3D"2> $TMPMSG" [[ -z "$STATE" || "$STATE" =3D '.' ]] && return 0 + [[ $VERBOSE -gt 0 ]] && SHOWERR=3D1 =20 rm -f $TMPMSG for CMD in $(echo $STATE | sed -e "s/:/ /g") @@ -362,12 +414,18 @@ set_ctrl_state() SFILE=3D$CGRP/cgroup.subtree_control PFILE=3D$CGRP/cpuset.cpus.partition CFILE=3D$CGRP/cpuset.cpus + XFILE=3D$CGRP/cpuset.cpus.exclusive S=3D$(expr substr $CMD 1 1) if [[ $S =3D S ]] then PREFIX=3D${CMD#?} COMM=3D"echo ${PREFIX}${CTRL} > $SFILE" eval $COMM $REDIRECT + elif [[ $S =3D X ]] + then + CPUS=3D${CMD#?} + COMM=3D"echo $CPUS > $XFILE" + eval $COMM $REDIRECT elif [[ $S =3D C ]] then CPUS=3D${CMD#?} @@ -430,7 +488,7 @@ online_cpus() [[ -n "OFFLINE_CPUS" ]] && { for C in $OFFLINE_CPUS do - write_cpu_online ${C}-1 + write_cpu_online ${C}=3D1 done } } @@ -443,18 +501,25 @@ reset_cgroup_states() echo 0 > $CGROUP2/cgroup.procs online_cpus rmdir A1/A2/A3 A1/A2 A1 B1 > /dev/null 2>&1 - set_ctrl_state . S- + pause 0.02 + set_ctrl_state . R- pause 0.01 } =20 dump_states() { - for DIR in A1 A1/A2 A1/A2/A3 B1 + for DIR in . A1 A1/A2 A1/A2/A3 B1 do + CPUS=3D$DIR/cpuset.cpus ECPUS=3D$DIR/cpuset.cpus.effective + XCPUS=3D$DIR/cpuset.cpus.exclusive PRS=3D$DIR/cpuset.cpus.partition + PCPUS=3D$DIR/.__DEBUG__.cpuset.cpus.subpartitions + [[ -e $CPUS ]] && echo "$CPUS: $(cat $CPUS)" + [[ -e $XCPUS ]] && echo "$XCPUS: $(cat $XCPUS)" [[ -e $ECPUS ]] && echo "$ECPUS: $(cat $ECPUS)" [[ -e $PRS ]] && echo "$PRS: $(cat $PRS)" + [[ -e $PCPUS ]] && echo "$PCPUS: $(cat $PCPUS)" done } =20 @@ -470,11 +535,17 @@ check_effective_cpus() set -- $(echo $CHK | sed -e "s/:/ /g") CGRP=3D$1 CPUS=3D$2 + if [[ $CGRP =3D X* ]] + then + CGRP=3D${CGRP#X} + FILE=3Dcpuset.cpus.exclusive + else + FILE=3Dcpuset.cpus.effective + fi [[ $CGRP =3D A2 ]] && CGRP=3DA1/A2 [[ $CGRP =3D A3 ]] && CGRP=3DA1/A2/A3 - FILE=3D$CGRP/cpuset.cpus.effective - [[ -e $FILE ]] || return 1 - [[ $CPUS =3D $(cat $FILE) ]] || return 1 + [[ -e $CGRP/$FILE ]] || return 1 + [[ $CPUS =3D $(cat $CGRP/$FILE) ]] || return 1 done } =20 @@ -524,6 +595,64 @@ check_cgroup_states() return 0 } =20 +# +# Get isolated (including offline) CPUs by looking at +# /sys/kernel/debug/sched/domains and compare that with the expected value. +# +# Note that a sched domain of just 1 CPU will be considered isolated. +# +# $1 - expected isolated cpu list +# +check_isolcpus() +{ + EXPECT_VAL=3D$1 + ISOLCPUS=3D + LASTISOLCPU=3D + SCHED_DOMAINS=3D/sys/kernel/debug/sched/domains + [[ -d $SCHED_DOMAINS ]] || return + + for ((CPU=3D0; CPU < $NR_CPUS; CPU++)) + do + [[ -n "$(ls ${SCHED_DOMAINS}/cpu$CPU)" ]] && continue + + if [[ -z "$LASTISOLCPU" ]] + then + ISOLCPUS=3D$CPU + LASTISOLCPU=3D$CPU + elif [[ "$LASTISOLCPU" -eq $((CPU - 1)) ]] + then + echo $ISOLCPUS | grep -q "\<$LASTISOLCPU\$" + if [[ $? -eq 0 ]] + then + ISOLCPUS=3D${ISOLCPUS}- + fi + LASTISOLCPU=3D$CPU + else + if [[ $ISOLCPUS =3D *- ]] + then + ISOLCPUS=3D${ISOLCPUS}$LASTISOLCPU + fi + ISOLCPUS=3D${ISOLCPUS},$CPU + LASTISOLCPU=3D$CPU + fi + done + [[ "$ISOLCPUS" =3D *- ]] && ISOLCPUS=3D${ISOLCPUS}$LASTISOLCPU + [[ $EXPECT_VAL =3D $ISOLCPUS ]] +} + +test_fail() +{ + TESTNUM=3D$1 + TESTTYPE=3D$2 + ADDINFO=3D$3 + echo "Test $TEST[$TESTNUM] failed $TESTTYPE check!" + [[ -n "$ADDINFO" ]] && echo "*** $ADDINFO ***" + eval echo \"\${$TEST[$I]}\" + echo + dump_states + exit 1 +} + # # Run cpuset state transition test # $1 - test matrix name @@ -536,88 +665,80 @@ run_state_test() { TEST=3D$1 CONTROLLER=3Dcpuset - CPULIST=3D0-6 I=3D0 eval CNT=3D"\${#$TEST[@]}" =20 reset_cgroup_states - echo $CPULIST > cpuset.cpus - echo root > cpuset.cpus.partition console_msg "Running state transition test ..." =20 while [[ $I -lt $CNT ]] do echo "Running test $I ..." > /dev/console + [[ $VERBOSE -gt 1 ]] && { + echo "" + eval echo \"\${$TEST[$I]}\" + } eval set -- "\${$TEST[$I]}" - ROOT=3D$1 - OLD_A1=3D$2 - OLD_A2=3D$3 - OLD_A3=3D$4 - OLD_B1=3D$5 - NEW_A1=3D$6 - NEW_A2=3D$7 - NEW_A3=3D$8 - NEW_B1=3D$9 - RESULT=3D${10} - ECPUS=3D${11} - STATES=3D${12} - - set_ctrl_state_noerr . $ROOT + OLD_A1=3D$1 + OLD_A2=3D$2 + OLD_A3=3D$3 + OLD_B1=3D$4 + NEW_A1=3D$5 + NEW_A2=3D$6 + NEW_A3=3D$7 + NEW_B1=3D$8 + RESULT=3D$9 + ECPUS=3D${10} + STATES=3D${11} + ICPUS=3D${12} + + set_ctrl_state_noerr B1 $OLD_B1 set_ctrl_state_noerr A1 $OLD_A1 set_ctrl_state_noerr A1/A2 $OLD_A2 set_ctrl_state_noerr A1/A2/A3 $OLD_A3 - set_ctrl_state_noerr B1 $OLD_B1 RETVAL=3D0 set_ctrl_state A1 $NEW_A1; ((RETVAL +=3D $?)) set_ctrl_state A1/A2 $NEW_A2; ((RETVAL +=3D $?)) set_ctrl_state A1/A2/A3 $NEW_A3; ((RETVAL +=3D $?)) set_ctrl_state B1 $NEW_B1; ((RETVAL +=3D $?)) =20 - [[ $RETVAL -ne $RESULT ]] && { - echo "Test $TEST[$I] failed result check!" - eval echo \"\${$TEST[$I]}\" - dump_states - exit 1 - } + [[ $RETVAL -ne $RESULT ]] && test_fail $I result =20 [[ -n "$ECPUS" && "$ECPUS" !=3D . ]] && { check_effective_cpus $ECPUS - [[ $? -ne 0 ]] && { - echo "Test $TEST[$I] failed effective CPU check!" - eval echo \"\${$TEST[$I]}\" - echo - dump_states - exit 1 - } + [[ $? -ne 0 ]] && test_fail $I "effective CPU" } =20 - [[ -n "$STATES" ]] && { + [[ -n "$STATES" && "$STATES" !=3D . ]] && { check_cgroup_states $STATES - [[ $? -ne 0 ]] && { - echo "FAILED: Test $TEST[$I] failed states check!" - eval echo \"\${$TEST[$I]}\" - echo - dump_states - exit 1 - } + [[ $? -ne 0 ]] && test_fail $I states } =20 + # Compare the expected isolated CPUs with the actual ones, + # if available + [[ -n "$ICPUS" ]] && { + check_isolcpus $ICPUS + [[ $? -ne 0 ]] && test_fail $I "isolated CPU" \ + "Expect $ICPUS, get $ISOLCPUS instead" + } reset_cgroup_states # # Check to see if effective cpu list changes # - pause 0.05 NEWLIST=3D$(cat cpuset.cpus.effective) + [[ $NEWLIST !=3D $CPULIST ]] && { + # Wait a bit longer & recheck + pause 0.05 + NEWLIST=3D$(cat cpuset.cpus.effective) + } [[ $NEWLIST !=3D $CPULIST ]] && { echo "Effective cpus changed to $NEWLIST after test $I!" exit 1 } - [[ -n "$VERBOSE" ]] && echo "Test $I done." + [[ $VERBOSE -gt 0 ]] && echo "Test $I done." ((I++)) done echo "All $I tests of $TEST PASSED." - - echo member > cpuset.cpus.partition } =20 # @@ -642,6 +763,7 @@ test_inotify() { ERR=3D0 PRS=3D/tmp/.prs_$$ + cd $CGROUP2/test [[ -f $WAIT_INOTIFY ]] || { echo "wait_inotify not found, inotify test SKIPPED." return @@ -655,7 +777,7 @@ test_inotify() rm -f $PRS wait_inotify $PWD/cpuset.cpus.partition $PRS & pause 0.01 - set_ctrl_state . "O1-0" + set_ctrl_state . "O1=3D0" pause 0.01 check_cgroup_states ".:P-1" if [[ $? -ne 0 ]] @@ -689,5 +811,3 @@ run_state_test TEST_MATRIX test_isolated test_inotify echo "All tests PASSED." -cd .. -rmdir test --=20 2.31.1