From nobody Thu Dec 18 07:57:06 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96D17C41513 for ; Tue, 15 Aug 2023 15:34:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238043AbjHOPdf (ORCPT ); Tue, 15 Aug 2023 11:33:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57464 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238115AbjHOPdB (ORCPT ); Tue, 15 Aug 2023 11:33:01 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 230A51BE6 for ; Tue, 15 Aug 2023 08:32:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1692113454; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AJ8b+6/Az1sefgsD4LZWYKV/ZRwh79oYPUej1g5eSqA=; b=NMlIyYBm/+b4S/riCDk2wReUmgfQ8OiA1aubCaIAj8h/oyQYBVG5AfExH8AAwNttosm4Wg 2F8Cwj8Xv/Rb2lR1NRnTmLJn+X6f2FByynMjPuv22Iyo+TfRYxXTNQp32rRlGTG1bWprjx Tmnmce3URTA4elxFjjwRfCnT09yUbeU= Received: from mimecast-mx02.redhat.com (66.187.233.73 [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-591-zTLLgcNnPeCPtha9Wc6wDQ-1; Tue, 15 Aug 2023 11:30:50 -0400 X-MC-Unique: zTLLgcNnPeCPtha9Wc6wDQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 15B5B3814946; Tue, 15 Aug 2023 15:30:50 +0000 (UTC) Received: from llong.com (unknown [10.22.18.67]) by smtp.corp.redhat.com (Postfix) with ESMTP id 78C7C40D2839; Tue, 15 Aug 2023 15:30:49 +0000 (UTC) From: Waiman Long To: Tejun Heo , Zefan Li , Johannes Weiner , Christian Brauner , Jonathan Corbet , Shuah Khan Cc: cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Juri Lelli , Dietmar Eggemann , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Giuseppe Scrivano , Waiman Long Subject: [PATCH-cgroup v6 6/6] cgroup/cpuset: Extend test_cpuset_prs.sh to test remote partition Date: Tue, 15 Aug 2023 11:30:27 -0400 Message-Id: <20230815153027.633355-7-longman@redhat.com> In-Reply-To: <20230815153027.633355-1-longman@redhat.com> References: <20230815153027.633355-1-longman@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This patch extends the test_cpuset_prs.sh test script to support testing the new remote partition type and the new "cpuset.cpus.exclusive" and "cpuset.cpus.exclusive.effective" control files by adding new tests for them. In addition, the following changes are also made: 1) Run the state transition tests directly under root to ease testing of remote partition and remove the unneeded test column. 2) Add a column to for the list of expected isolated CPUs and compare it with the actual value by looking at the state of /sys/kernel/debug/sched/domains which will be available if the verbose flag is set. Signed-off-by: Waiman Long --- .../selftests/cgroup/test_cpuset_prs.sh | 428 ++++++++++++------ 1 file changed, 287 insertions(+), 141 deletions(-) diff --git a/tools/testing/selftests/cgroup/test_cpuset_prs.sh b/tools/test= ing/selftests/cgroup/test_cpuset_prs.sh index 4afb132e4e4f..e623a6b4034f 100755 --- a/tools/testing/selftests/cgroup/test_cpuset_prs.sh +++ b/tools/testing/selftests/cgroup/test_cpuset_prs.sh @@ -3,7 +3,7 @@ # # Test for cpuset v2 partition root state (PRS) # -# The sched verbose flag is set, if available, so that the console log +# The sched verbose flag can be optionally set so that the console log # can be examined for the correct setting of scheduling domain. # =20 @@ -22,27 +22,27 @@ WAIT_INOTIFY=3D$(cd $(dirname $0); pwd)/wait_inotify # Find cgroup v2 mount point CGROUP2=3D$(mount -t cgroup2 | head -1 | awk -e '{print $3}') [[ -n "$CGROUP2" ]] || skip_test "Cgroup v2 mount point not found!" +SUBPARTS_CPUS=3D$CGROUP2/.__DEBUG__.cpuset.cpus.subpartitions +CPULIST=3D$(cat $CGROUP2/cpuset.cpus.effective) =20 -CPUS=3D$(lscpu | grep "^CPU(s):" | sed -e "s/.*:[[:space:]]*//") -[[ $CPUS -lt 8 ]] && skip_test "Test needs at least 8 cpus available!" +NR_CPUS=3D$(lscpu | grep "^CPU(s):" | sed -e "s/.*:[[:space:]]*//") +[[ $NR_CPUS -lt 8 ]] && skip_test "Test needs at least 8 cpus available!" =20 # Set verbose flag and delay factor PROG=3D$1 -VERBOSE=3D +VERBOSE=3D0 DELAY_FACTOR=3D1 SCHED_DEBUG=3D while [[ "$1" =3D -* ]] do case "$1" in - -v) VERBOSE=3D1 + -v) ((VERBOSE++)) # Enable sched/verbose can slow thing down [[ $DELAY_FACTOR -eq 1 ]] && DELAY_FACTOR=3D2 - break ;; -d) DELAY_FACTOR=3D$2 shift - break ;; *) echo "Usage: $PROG [-v] [-d " exit @@ -52,7 +52,7 @@ do done =20 # Set sched verbose flag if available when "-v" option is specified -if [[ -n "$VERBOSE" && -d /sys/kernel/debug/sched ]] +if [[ $VERBOSE -gt 0 && -d /sys/kernel/debug/sched ]] then # Used to restore the original setting during cleanup SCHED_DEBUG=3D$(cat /sys/kernel/debug/sched/verbose) @@ -61,14 +61,26 @@ fi =20 cd $CGROUP2 echo +cpuset > cgroup.subtree_control + +# +# If cpuset has been set up and used in child cgroups, we may not be able = to +# create partition under root cgroup because of the CPU exclusivity rule. +# So we are going to skip the test if this is the case. +# [[ -d test ]] || mkdir test -cd test +echo 0-6 > test/cpuset.cpus +echo root > test/cpuset.cpus.partition +cat test/cpuset.cpus.partition | grep -q invalid +RESULT=3D$? +echo member > test/cpuset.cpus.partition +echo "" > test/cpuset.cpus +[[ $RESULT -eq 0 ]] && skip_test "Child cgroups are using cpuset!" =20 cleanup() { online_cpus + cd $CGROUP2 rmdir A1/A2/A3 A1/A2 A1 B1 > /dev/null 2>&1 - cd .. rmdir test > /dev/null 2>&1 [[ -n "$SCHED_DEBUG" ]] && echo "$SCHED_DEBUG" > /sys/kernel/debug/sched/verbose @@ -103,7 +115,7 @@ test_partition() [[ $? -eq 0 ]] || exit 1 ACTUAL_VAL=3D$(cat cpuset.cpus.partition) [[ $ACTUAL_VAL !=3D $EXPECTED_VAL ]] && { - echo "cpuset.cpus.partition: expect $EXPECTED_VAL, found $EXPECTED_VAL" + echo "cpuset.cpus.partition: expect $EXPECTED_VAL, found $ACTUAL_VAL" echo "Test FAILED" exit 1 } @@ -114,7 +126,7 @@ test_effective_cpus() EXPECTED_VAL=3D$1 ACTUAL_VAL=3D$(cat cpuset.cpus.effective) [[ "$ACTUAL_VAL" !=3D "$EXPECTED_VAL" ]] && { - echo "cpuset.cpus.effective: expect '$EXPECTED_VAL', found '$EXPECTED_VA= L'" + echo "cpuset.cpus.effective: expect '$EXPECTED_VAL', found '$ACTUAL_VAL'" echo "Test FAILED" exit 1 } @@ -139,6 +151,7 @@ test_add_proc() # test_isolated() { + cd $CGROUP2/test echo 2-3 > cpuset.cpus TYPE=3D$(cat cpuset.cpus.partition) [[ $TYPE =3D member ]] || echo member > cpuset.cpus.partition @@ -203,125 +216,186 @@ test_isolated() # # Cgroup test hierarchy # -# test -- A1 -- A2 -- A3 -# \- B1 +# root -- A1 -- A2 -- A3 +# +- B1 # -# P =3D set cpus.partition (0:member, 1:root, 2:isolated, -1:root inva= lid) +# P =3D set cpus.partition (0:member, 1:root, 2:isolated) # C =3D add cpu-list # S

=3D use prefix in subtree_control # T =3D put a task into cgroup -# O- =3D Write to CPU online file of +# O=3D =3D Write to CPU online file of # SETUP_A123_PARTITIONS=3D"C1-3:P1:S+ C2-3:P1:S+ C3:P1" TEST_MATRIX=3D( - # test old-A1 old-A2 old-A3 old-B1 new-A1 new-A2 new-A3 new-B1 fail ECPU= s Pstate - # ---- ------ ------ ------ ------ ------ ------ ------ ------ ---- ----= - ------ - " S+ C0-1 . . C2-3 S+ C4-5 . . 0 A2:0-= 1" - " S+ C0-1 . . C2-3 P1 . . . 0 " - " S+ C0-1 . . C2-3 P1:S+ C0-1:P1 . . 0 " - " S+ C0-1 . . C2-3 P1:S+ C1:P1 . . 0 " - " S+ C0-1:S+ . . C2-3 . . . P1 0 " - " S+ C0-1:P1 . . C2-3 S+ C1 . . 0 " - " S+ C0-1:P1 . . C2-3 S+ C1:P1 . . 0 " - " S+ C0-1:P1 . . C2-3 S+ C1:P1 . P1 0 " - " S+ C0-1:P1 . . C2-3 C4-5 . . . 0 A1:4-= 5" - " S+ C0-1:P1 . . C2-3 S+:C4-5 . . . 0 A1:4-= 5" - " S+ C0-1 . . C2-3:P1 . . . C2 0 " - " S+ C0-1 . . C2-3:P1 . . . C4-5 0 B1:4-= 5" - " S+ C0-3:P1:S+ C2-3:P1 . . . . . . 0 A1:0-= 1,A2:2-3" - " S+ C0-3:P1:S+ C2-3:P1 . . C1-3 . . . 0 A1:1,= A2:2-3" - " S+ C2-3:P1:S+ C3:P1 . . C3 . . . 0 A1:,A= 2:3 A1:P1,A2:P1" - " S+ C2-3:P1:S+ C3:P1 . . C3 P0 . . 0 A1:3,= A2:3 A1:P1,A2:P0" - " S+ C2-3:P1:S+ C2:P1 . . C2-4 . . . 0 A1:3-= 4,A2:2" - " S+ C2-3:P1:S+ C3:P1 . . C3 . . C0-2 0 A1:,B= 1:0-2 A1:P1,A2:P1" - " S+ $SETUP_A123_PARTITIONS . C2-3 . . . 0 A1:,A= 2:2,A3:3 A1:P1,A2:P1,A3:P1" + # old-A1 old-A2 old-A3 old-B1 new-A1 new-A2 new-A3 new-B1 fail ECPUs Pst= ate ISOLCPUS + # ------ ------ ------ ------ ------ ------ ------ ------ ---- ----- ---= --- -------- + " C0-1 . . C2-3 S+ C4-5 . . 0 A2:0-1" + " C0-1 . . C2-3 P1 . . . 0 " + " C0-1 . . C2-3 P1:S+ C0-1:P1 . . 0 " + " C0-1 . . C2-3 P1:S+ C1:P1 . . 0 " + " C0-1:S+ . . C2-3 . . . P1 0 " + " C0-1:P1 . . C2-3 S+ C1 . . 0 " + " C0-1:P1 . . C2-3 S+ C1:P1 . . 0 " + " C0-1:P1 . . C2-3 S+ C1:P1 . P1 0 " + " C0-1:P1 . . C2-3 C4-5 . . . 0 A1:4-5" + " C0-1:P1 . . C2-3 S+:C4-5 . . . 0 A1:4-5" + " C0-1 . . C2-3:P1 . . . C2 0 " + " C0-1 . . C2-3:P1 . . . C4-5 0 B1:4-5" + "C0-3:P1:S+ C2-3:P1 . . . . . . 0 A1:0-1,A2:= 2-3" + "C0-3:P1:S+ C2-3:P1 . . C1-3 . . . 0 A1:1,A2:2-= 3" + "C2-3:P1:S+ C3:P1 . . C3 . . . 0 A1:,A2:3 A= 1:P1,A2:P1" + "C2-3:P1:S+ C3:P1 . . C3 P0 . . 0 A1:3,A2:3 = A1:P1,A2:P0" + "C2-3:P1:S+ C2:P1 . . C2-4 . . . 0 A1:3-4,A2:= 2" + "C2-3:P1:S+ C3:P1 . . C3 . . C0-2 0 A1:,B1:0-2= A1:P1,A2:P1" + "$SETUP_A123_PARTITIONS . C2-3 . . . 0 A1:,A2:2,A= 3:3 A1:P1,A2:P1,A3:P1" =20 # CPU offlining cases: - " S+ C0-1 . . C2-3 S+ C4-5 . O2-0 0 A1:0-= 1,B1:3" - " S+ C0-3:P1:S+ C2-3:P1 . . O2-0 . . . 0 A1:0-= 1,A2:3" - " S+ C0-3:P1:S+ C2-3:P1 . . O2-0 O2-1 . . 0 A1:0-= 1,A2:2-3" - " S+ C0-3:P1:S+ C2-3:P1 . . O1-0 . . . 0 A1:0,= A2:2-3" - " S+ C0-3:P1:S+ C2-3:P1 . . O1-0 O1-1 . . 0 A1:0-= 1,A2:2-3" - " S+ C2-3:P1:S+ C3:P1 . . O3-0 O3-1 . . 0 A1:2,= A2:3 A1:P1,A2:P1" - " S+ C2-3:P1:S+ C3:P2 . . O3-0 O3-1 . . 0 A1:2,= A2:3 A1:P1,A2:P2" - " S+ C2-3:P1:S+ C3:P1 . . O2-0 O2-1 . . 0 A1:2,= A2:3 A1:P1,A2:P1" - " S+ C2-3:P1:S+ C3:P2 . . O2-0 O2-1 . . 0 A1:2,= A2:3 A1:P1,A2:P2" - " S+ C2-3:P1:S+ C3:P1 . . O2-0 . . . 0 A1:,A= 2:3 A1:P1,A2:P1" - " S+ C2-3:P1:S+ C3:P1 . . O3-0 . . . 0 A1:2,= A2: A1:P1,A2:P1" - " S+ C2-3:P1:S+ C3:P1 . . T:O2-0 . . . 0 A1:3,= A2:3 A1:P1,A2:P-1" - " S+ C2-3:P1:S+ C3:P1 . . . T:O3-0 . . 0 A1:2,= A2:2 A1:P1,A2:P-1" - " S+ $SETUP_A123_PARTITIONS . O1-0 . . . 0 A1:,A= 2:2,A3:3 A1:P1,A2:P1,A3:P1" - " S+ $SETUP_A123_PARTITIONS . O2-0 . . . 0 A1:1,= A2:,A3:3 A1:P1,A2:P1,A3:P1" - " S+ $SETUP_A123_PARTITIONS . O3-0 . . . 0 A1:1,= A2:2,A3: A1:P1,A2:P1,A3:P1" - " S+ $SETUP_A123_PARTITIONS . T:O1-0 . . . 0 A1:2-= 3,A2:2-3,A3:3 A1:P1,A2:P-1,A3:P-1" - " S+ $SETUP_A123_PARTITIONS . . T:O2-0 . . 0 A1:1,= A2:3,A3:3 A1:P1,A2:P1,A3:P-1" - " S+ $SETUP_A123_PARTITIONS . . . T:O3-0 . 0 A1:1,= A2:2,A3:2 A1:P1,A2:P1,A3:P-1" - " S+ $SETUP_A123_PARTITIONS . T:O1-0 O1-1 . . 0 A1:1,= A2:2,A3:3 A1:P1,A2:P1,A3:P1" - " S+ $SETUP_A123_PARTITIONS . . T:O2-0 O2-1 . 0 A1:1,= A2:2,A3:3 A1:P1,A2:P1,A3:P1" - " S+ $SETUP_A123_PARTITIONS . . . T:O3-0 O3-1 0 A1:1,= A2:2,A3:3 A1:P1,A2:P1,A3:P1" - " S+ $SETUP_A123_PARTITIONS . T:O1-0 O2-0 O1-1 . 0 A1:1,= A2:,A3:3 A1:P1,A2:P1,A3:P1" - " S+ $SETUP_A123_PARTITIONS . T:O1-0 O2-0 O2-1 . 0 A1:2-= 3,A2:2-3,A3:3 A1:P1,A2:P-1,A3:P-1" - - # test old-A1 old-A2 old-A3 old-B1 new-A1 new-A2 new-A3 new-B1 fail ECPU= s Pstate - # ---- ------ ------ ------ ------ ------ ------ ------ ------ ---- ----= - ------ + " C0-1 . . C2-3 S+ C4-5 . O2=3D0 0 A1:0-1,B= 1:3" + "C0-3:P1:S+ C2-3:P1 . . O2=3D0 . . . 0 A1:0-1,A= 2:3" + "C0-3:P1:S+ C2-3:P1 . . O2=3D0 O2=3D1 . . 0 A1:0-1= ,A2:2-3" + "C0-3:P1:S+ C2-3:P1 . . O1=3D0 . . . 0 A1:0,A2:= 2-3" + "C0-3:P1:S+ C2-3:P1 . . O1=3D0 O1=3D1 . . 0 A1:0-1= ,A2:2-3" + "C2-3:P1:S+ C3:P1 . . O3=3D0 O3=3D1 . . 0 A1:2,A= 2:3 A1:P1,A2:P1" + "C2-3:P1:S+ C3:P2 . . O3=3D0 O3=3D1 . . 0 A1:2,A= 2:3 A1:P1,A2:P2" + "C2-3:P1:S+ C3:P1 . . O2=3D0 O2=3D1 . . 0 A1:2,A= 2:3 A1:P1,A2:P1" + "C2-3:P1:S+ C3:P2 . . O2=3D0 O2=3D1 . . 0 A1:2,A= 2:3 A1:P1,A2:P2" + "C2-3:P1:S+ C3:P1 . . O2=3D0 . . . 0 A1:,A2:3= A1:P1,A2:P1" + "C2-3:P1:S+ C3:P1 . . O3=3D0 . . . 0 A1:2,A2:= A1:P1,A2:P1" + "C2-3:P1:S+ C3:P1 . . T:O2=3D0 . . . 0 A1:3,A2:= 3 A1:P1,A2:P-1" + "C2-3:P1:S+ C3:P1 . . . T:O3=3D0 . . 0 A1:2,A2:= 2 A1:P1,A2:P-1" + "$SETUP_A123_PARTITIONS . O1=3D0 . . . 0 A1:,A2:2= ,A3:3 A1:P1,A2:P1,A3:P1" + "$SETUP_A123_PARTITIONS . O2=3D0 . . . 0 A1:1,A2:= ,A3:3 A1:P1,A2:P1,A3:P1" + "$SETUP_A123_PARTITIONS . O3=3D0 . . . 0 A1:1,A2:= 2,A3: A1:P1,A2:P1,A3:P1" + "$SETUP_A123_PARTITIONS . T:O1=3D0 . . . 0 A1:2-3,A= 2:2-3,A3:3 A1:P1,A2:P-1,A3:P-1" + "$SETUP_A123_PARTITIONS . . T:O2=3D0 . . 0 A1:1,A2:= 3,A3:3 A1:P1,A2:P1,A3:P-1" + "$SETUP_A123_PARTITIONS . . . T:O3=3D0 . 0 A1:1,A2:= 2,A3:2 A1:P1,A2:P1,A3:P-1" + "$SETUP_A123_PARTITIONS . T:O1=3D0 O1=3D1 . . 0 A1:1,A= 2:2,A3:3 A1:P1,A2:P1,A3:P1" + "$SETUP_A123_PARTITIONS . . T:O2=3D0 O2=3D1 . 0 A1:1,A= 2:2,A3:3 A1:P1,A2:P1,A3:P1" + "$SETUP_A123_PARTITIONS . . . T:O3=3D0 O3=3D1 0 A1:1,A= 2:2,A3:3 A1:P1,A2:P1,A3:P1" + "$SETUP_A123_PARTITIONS . T:O1=3D0 O2=3D0 O1=3D1 . 0 A1:1= ,A2:,A3:3 A1:P1,A2:P1,A3:P1" + "$SETUP_A123_PARTITIONS . T:O1=3D0 O2=3D0 O2=3D1 . 0 A1:2= -3,A2:2-3,A3:3 A1:P1,A2:P-1,A3:P-1" + + # old-A1 old-A2 old-A3 old-B1 new-A1 new-A2 new-A3 new-B1 fail ECPUs Pst= ate ISOLCPUS + # ------ ------ ------ ------ ------ ------ ------ ------ ---- ----- ---= --- -------- + # + # Remote partition and cpuset.cpus.exclusive tests + # + " C0-3:S+ C1-3:S+ C2-3 . X2-3 . . . 0 A1:0-3,A2:= 1-3,A3:2-3,XA1:2-3" + " C0-3:S+ C1-3:S+ C2-3 . X2-3 X2-3:P2 . . 0 A1:0-1,A2:= 2-3,A3:2-3 A1:P0,A2:P2 2-3" + " C0-3:S+ C1-3:S+ C2-3 . X2-3 X3:P2 . . 0 A1:0-2,A2:= 3,A3:3 A1:P0,A2:P2 3" + " C0-3:S+ C1-3:S+ C2-3 . X2-3 X2-3 X2-3:P2 . 0 A1:0-1,A2:= 1,A3:2-3 A1:P0,A3:P2 2-3" + " C0-3:S+ C1-3:S+ C2-3 . X2-3 X2-3 X2-3:P2:C3 . 0 A1:0-2,A2:= 1-2,A3:3 A1:P0,A3:P2 3" + " C0-3:S+ C1-3:S+ C2-3 C2-3 . . . P2 0 A1:0-3,A2:= 1-3,A3:2-3,B1:2-3 A1:P0,A3:P0,B1:P-2" + " C0-3:S+ C1-3:S+ C2-3 C4-5 . . . P2 0 B1:4-5 B1:= P2 4-5" + " C0-3:S+ C1-3:S+ C2-3 C4 X2-3 X2-3 X2-3:P2 P2 0 A3:2-3,B1:= 4 A3:P2,B1:P2 2-4" + " C0-3:S+ C1-3:S+ C2-3 C4 X2-3 X2-3 X2-3:P2:C1-3 P2 0 A3:2-3,B1:= 4 A3:P2,B1:P2 2-4" + " C0-3:S+ C1-3:S+ C2-3 C4 X1-3 X1-3:P2 P2 . 0 A2:1,A3:2-= 3 A2:P2,A3:P2 1-3" + " C0-3:S+ C1-3:S+ C2-3 C4 X2-3 X2-3 X2-3:P2 P2:C4-5 0 A3:2-3,B1:= 4-5 A3:P2,B1:P2 2-5" + + # Nested remote/local partition tests + " C0-3:S+ C1-3:S+ C2-3 C4-5 X2-3 X2-3:P1 P2 P1 0 A1:0-1,A2:= ,A3:2-3,B1:4-5 \ + A1:P0,A2:P1,A3:P2,B1:P1 2-3" + " C0-3:S+ C1-3:S+ C2-3 C4 X2-3 X2-3:P1 P2 P1 0 A1:0-1,A2:= ,A3:2-3,B1:4 \ + A1:P0,A2:P1,A3:P2,B1:P1 2-4" + " C0-3:S+ C1-3:S+ C3 C4 X2-3 X2-3:P1 P2 P1 0 A1:0-1,A2:= 2,A3:3,B1:4 \ + A1:P0,A2:P1,A3:P2,B1:P1 2-4" + + # Remote partition offline tests + " C0-3:S+ C1-3:S+ C2-3 . X2-3 X2-3 X2-3:P2:O2=3D0 . 0 A1:0-1,A= 2:1,A3:3 A1:P0,A3:P2 2-3" + " C0-3:S+ C1-3:S+ C2-3 . X2-3 X2-3 X2-3:P2:O2=3D0 O2=3D1 0 A1:0-= 1,A2:1,A3:2-3 A1:P0,A3:P2 2-3" + " C0-3:S+ C1-3:S+ C3 . X2-3 X2-3 P2:O3=3D0 . 0 A1:0-2,A= 2:1-2,A3: A1:P0,A3:P2 3" + " C0-3:S+ C1-3:S+ C3 . X2-3 X2-3 T:P2:O3=3D0 . 0 A1:0-2,A= 2:1-2,A3:1-2 A1:P0,A3:P-2 3" + + # An invalidated remote partition cannot self-recover from hotplug + " C0-3:S+ C1-3:S+ C2 . X2-3 X2-3 T:P2:O2=3D0 O2=3D1 0 A1:0-3= ,A2:1-3,A3:2 A1:P0,A3:P-2" + + # cpus_allowed/exclusive_cpus update test + " C0-3:X2-3:S+ C1-3:X2-3:S+ C2-3:X2-3 \ + . C4 . P2 . 0 A1:4,A2:4,XA2:,XA3:,A3:4 \ + A1:P0,A3:P-2 ." + " C0-3:X2-3:S+ C1-3:X2-3:S+ C2-3:X2-3 \ + . X1 . P2 . 0 A1:0-3,A2:1-3,XA1:1,XA2:,XA3:,A= 3:2-3 \ + A1:P0,A3:P-2 ." + " C0-3:X2-3:S+ C1-3:X2-3:S+ C2-3:X2-3 \ + . . C3 P2 . 0 A1:0-2,A2:0-2,XA2:3,XA3:3,A3:3 \ + A1:P0,A3:P2 3" + " C0-3:X2-3:S+ C1-3:X2-3:S+ C2-3:X2-3 \ + . . X3 P2 . 0 A1:0-2,A2:1-2,XA2:3,XA3:3,A3:3 \ + A1:P0,A3:P2 3" + " C0-3:X2-3:S+ C1-3:X2-3:S+ C2-3:X2-3:P2 \ + . . X3 . . 0 A1:0-3,A2:1-3,XA2:3,XA3:3,A3:2-= 3 \ + A1:P0,A3:P-2 ." + " C0-3:X2-3:S+ C1-3:X2-3:S+ C2-3:X2-3:P2 \ + . . C3 . . 0 A1:0-3,A2:3,XA2:3,XA3:3,A3:3 \ + A1:P0,A3:P-2 ." + " C0-3:X2-3:S+ C1-3:X2-3:S+ C2-3:X2-3:P2 \ + . C4 . . . 0 A1:4,A2:4,A3:4,XA1:,XA2:,XA3 \ + A1:P0,A3:P-2 ." + + # old-A1 old-A2 old-A3 old-B1 new-A1 new-A2 new-A3 new-B1 fail ECPUs Pst= ate ISOLCPUS + # ------ ------ ------ ------ ------ ------ ------ ------ ---- ----- ---= --- -------- # # Incorrect change to cpuset.cpus invalidates partition root # # Adding CPUs to partition root that are not in parent's # cpuset.cpus is allowed, but those extra CPUs are ignored. - " S+ C2-3:P1:S+ C3:P1 . . . C2-4 . . 0 A1:,A= 2:2-3 A1:P1,A2:P1" + "C2-3:P1:S+ C3:P1 . . . C2-4 . . 0 A1:,A2:2-3= A1:P1,A2:P1" =20 # Taking away all CPUs from parent or itself if there are tasks # will make the partition invalid. - " S+ C2-3:P1:S+ C3:P1 . . T C2-3 . . 0 A1:2-= 3,A2:2-3 A1:P1,A2:P-1" - " S+ C3:P1:S+ C3 . . T P1 . . 0 A1:3,= A2:3 A1:P1,A2:P-1" - " S+ $SETUP_A123_PARTITIONS . T:C2-3 . . . 0 A1:2-= 3,A2:2-3,A3:3 A1:P1,A2:P-1,A3:P-1" - " S+ $SETUP_A123_PARTITIONS . T:C2-3:C1-3 . . . 0 A1:1,= A2:2,A3:3 A1:P1,A2:P1,A3:P1" + "C2-3:P1:S+ C3:P1 . . T C2-3 . . 0 A1:2-3,A2:= 2-3 A1:P1,A2:P-1" + " C3:P1:S+ C3 . . T P1 . . 0 A1:3,A2:3 = A1:P1,A2:P-1" + "$SETUP_A123_PARTITIONS . T:C2-3 . . . 0 A1:2-3,A2:= 2-3,A3:3 A1:P1,A2:P-1,A3:P-1" + "$SETUP_A123_PARTITIONS . T:C2-3:C1-3 . . . 0 A1:1,A2:2,= A3:3 A1:P1,A2:P1,A3:P1" =20 # Changing a partition root to member makes child partitions invalid - " S+ C2-3:P1:S+ C3:P1 . . P0 . . . 0 A1:2-= 3,A2:3 A1:P0,A2:P-1" - " S+ $SETUP_A123_PARTITIONS . C2-3 P0 . . 0 A1:2-= 3,A2:2-3,A3:3 A1:P1,A2:P0,A3:P-1" + "C2-3:P1:S+ C3:P1 . . P0 . . . 0 A1:2-3,A2:= 3 A1:P0,A2:P-1" + "$SETUP_A123_PARTITIONS . C2-3 P0 . . 0 A1:2-3,A2:= 2-3,A3:3 A1:P1,A2:P0,A3:P-1" =20 # cpuset.cpus can contains cpus not in parent's cpuset.cpus as long # as they overlap. - " S+ C2-3:P1:S+ . . . . C3-4:P1 . . 0 A1:2,= A2:3 A1:P1,A2:P1" + "C2-3:P1:S+ . . . . C3-4:P1 . . 0 A1:2,A2:3 = A1:P1,A2:P1" =20 # Deletion of CPUs distributed to child cgroup is allowed. - " S+ C0-1:P1:S+ C1 . C2-3 C4-5 . . . 0 A1:4-= 5,A2:4-5" + "C0-1:P1:S+ C1 . C2-3 C4-5 . . . 0 A1:4-5,A2:= 4-5" =20 # To become a valid partition root, cpuset.cpus must overlap parent's # cpuset.cpus. - " S+ C0-1:P1 . . C2-3 S+ C4-5:P1 . . 0 A1:0-= 1,A2:0-1 A1:P1,A2:P-1" + " C0-1:P1 . . C2-3 S+ C4-5:P1 . . 0 A1:0-1,A2:= 0-1 A1:P1,A2:P-1" =20 # Enabling partition with child cpusets is allowed - " S+ C0-1:S+ C1 . C2-3 P1 . . . 0 A1:0-= 1,A2:1 A1:P1" + " C0-1:S+ C1 . C2-3 P1 . . . 0 A1:0-1,A2:= 1 A1:P1" =20 # A partition root with non-partition root parent is invalid, but it # can be made valid if its parent becomes a partition root too. - " S+ C0-1:S+ C1 . C2-3 . P2 . . 0 A1:0-= 1,A2:1 A1:P0,A2:P-2" - " S+ C0-1:S+ C1:P2 . C2-3 P1 . . . 0 A1:0,= A2:1 A1:P1,A2:P2" + " C0-1:S+ C1 . C2-3 . P2 . . 0 A1:0-1,A2:= 1 A1:P0,A2:P-2" + " C0-1:S+ C1:P2 . C2-3 P1 . . . 0 A1:0,A2:1 = A1:P1,A2:P2" =20 # A non-exclusive cpuset.cpus change will invalidate partition and its si= blings - " S+ C0-1:P1 . . C2-3 C0-2 . . . 0 A1:0-= 2,B1:2-3 A1:P-1,B1:P0" - " S+ C0-1:P1 . . P1:C2-3 C0-2 . . . 0 A1:0-2,= B1:2-3 A1:P-1,B1:P-1" - " S+ C0-1 . . P1:C2-3 C0-2 . . . 0 A1:0-2,= B1:2-3 A1:P0,B1:P-1" + " C0-1:P1 . . C2-3 C0-2 . . . 0 A1:0-2,B1:= 2-3 A1:P-1,B1:P0" + " C0-1:P1 . . P1:C2-3 C0-2 . . . 0 A1:0-2,B1:= 2-3 A1:P-1,B1:P-1" + " C0-1 . . P1:C2-3 C0-2 . . . 0 A1:0-2,B1:= 2-3 A1:P0,B1:P-1" =20 - # test old-A1 old-A2 old-A3 old-B1 new-A1 new-A2 new-A3 new-B1 fail ECPU= s Pstate - # ---- ------ ------ ------ ------ ------ ------ ------ ------ ---- ----= - ------ + # old-A1 old-A2 old-A3 old-B1 new-A1 new-A2 new-A3 new-B1 fail ECPUs Pst= ate ISOLCPUS + # ------ ------ ------ ------ ------ ------ ------ ------ ---- ----- ---= --- -------- # Failure cases: =20 # A task cannot be added to a partition with no cpu - " S+ C2-3:P1:S+ C3:P1 . . O2-0:T . . . 1 A1:,A= 2:3 A1:P1,A2:P1" + "C2-3:P1:S+ C3:P1 . . O2=3D0:T . . . 1 A1:,A2:3= A1:P1,A2:P1" + + # Changes to cpuset.cpus.exclusive that violate exclusivity rule is rejec= ted + " C0-3 . . C4-5 X0-3 . . X3-5 1 A1:0-3,B1:= 4-5" + ) =20 # # Write to the cpu online file -# $1 - - where =3D cpu number, value to be written +# $1 - =3D where =3D cpu number, value to be written # write_cpu_online() { - CPU=3D${1%-*} - VAL=3D${1#*-} + CPU=3D${1%=3D*} + VAL=3D${1#*=3D} CPUFILE=3D//sys/devices/system/cpu/cpu${CPU}/online if [[ $VAL -eq 0 ]] then @@ -349,11 +423,12 @@ set_ctrl_state() TMPMSG=3D/tmp/.msg_$$ CGRP=3D$1 STATE=3D$2 - SHOWERR=3D${3}${VERBOSE} + SHOWERR=3D${3} CTRL=3D${CTRL:=3D$CONTROLLER} HASERR=3D0 REDIRECT=3D"2> $TMPMSG" [[ -z "$STATE" || "$STATE" =3D '.' ]] && return 0 + [[ $VERBOSE -gt 0 ]] && SHOWERR=3D1 =20 rm -f $TMPMSG for CMD in $(echo $STATE | sed -e "s/:/ /g") @@ -362,12 +437,18 @@ set_ctrl_state() SFILE=3D$CGRP/cgroup.subtree_control PFILE=3D$CGRP/cpuset.cpus.partition CFILE=3D$CGRP/cpuset.cpus + XFILE=3D$CGRP/cpuset.cpus.exclusive S=3D$(expr substr $CMD 1 1) if [[ $S =3D S ]] then PREFIX=3D${CMD#?} COMM=3D"echo ${PREFIX}${CTRL} > $SFILE" eval $COMM $REDIRECT + elif [[ $S =3D X ]] + then + CPUS=3D${CMD#?} + COMM=3D"echo $CPUS > $XFILE" + eval $COMM $REDIRECT elif [[ $S =3D C ]] then CPUS=3D${CMD#?} @@ -430,7 +511,7 @@ online_cpus() [[ -n "OFFLINE_CPUS" ]] && { for C in $OFFLINE_CPUS do - write_cpu_online ${C}-1 + write_cpu_online ${C}=3D1 done } } @@ -443,18 +524,27 @@ reset_cgroup_states() echo 0 > $CGROUP2/cgroup.procs online_cpus rmdir A1/A2/A3 A1/A2 A1 B1 > /dev/null 2>&1 - set_ctrl_state . S- + pause 0.02 + set_ctrl_state . R- pause 0.01 } =20 dump_states() { - for DIR in A1 A1/A2 A1/A2/A3 B1 + for DIR in . A1 A1/A2 A1/A2/A3 B1 do + CPUS=3D$DIR/cpuset.cpus ECPUS=3D$DIR/cpuset.cpus.effective + XCPUS=3D$DIR/cpuset.cpus.exclusive + XECPUS=3D$DIR/cpuset.cpus.exclusive.effective PRS=3D$DIR/cpuset.cpus.partition - [[ -e $ECPUS ]] && echo "$ECPUS: $(cat $ECPUS)" - [[ -e $PRS ]] && echo "$PRS: $(cat $PRS)" + PCPUS=3D$DIR/.__DEBUG__.cpuset.cpus.subpartitions + [[ -e $CPUS ]] && echo "$CPUS: $(cat $CPUS)" + [[ -e $XCPUS ]] && echo "$XCPUS: $(cat $XCPUS)" + [[ -e $ECPUS ]] && echo "$ECPUS: $(cat $ECPUS)" + [[ -e $XECPUS ]] && echo "$XECPUS: $(cat $XECPUS)" + [[ -e $PRS ]] && echo "$PRS: $(cat $PRS)" + [[ -e $PCPUS ]] && echo "$PCPUS: $(cat $PCPUS)" done } =20 @@ -470,11 +560,17 @@ check_effective_cpus() set -- $(echo $CHK | sed -e "s/:/ /g") CGRP=3D$1 CPUS=3D$2 + if [[ $CGRP =3D X* ]] + then + CGRP=3D${CGRP#X} + FILE=3Dcpuset.cpus.exclusive.effective + else + FILE=3Dcpuset.cpus.effective + fi [[ $CGRP =3D A2 ]] && CGRP=3DA1/A2 [[ $CGRP =3D A3 ]] && CGRP=3DA1/A2/A3 - FILE=3D$CGRP/cpuset.cpus.effective - [[ -e $FILE ]] || return 1 - [[ $CPUS =3D $(cat $FILE) ]] || return 1 + [[ -e $CGRP/$FILE ]] || return 1 + [[ $CPUS =3D $(cat $CGRP/$FILE) ]] || return 1 done } =20 @@ -524,6 +620,65 @@ check_cgroup_states() return 0 } =20 +# +# Get isolated (including offline) CPUs by looking at +# /sys/kernel/debug/sched/domains and compare that with the expected value. +# +# Note that a sched domain of just 1 CPU will be considered isolated. +# +# $1 - expected isolated cpu list +# +check_isolcpus() +{ + EXPECT_VAL=3D$1 + ISOLCPUS=3D + LASTISOLCPU=3D + SCHED_DOMAINS=3D/sys/kernel/debug/sched/domains + [[ -d $SCHED_DOMAINS ]] || return + [[ $EXPECT_VAL =3D . ]] && EXPECT_VAL=3D + + for ((CPU=3D0; CPU < $NR_CPUS; CPU++)) + do + [[ -n "$(ls ${SCHED_DOMAINS}/cpu$CPU)" ]] && continue + + if [[ -z "$LASTISOLCPU" ]] + then + ISOLCPUS=3D$CPU + LASTISOLCPU=3D$CPU + elif [[ "$LASTISOLCPU" -eq $((CPU - 1)) ]] + then + echo $ISOLCPUS | grep -q "\<$LASTISOLCPU\$" + if [[ $? -eq 0 ]] + then + ISOLCPUS=3D${ISOLCPUS}- + fi + LASTISOLCPU=3D$CPU + else + if [[ $ISOLCPUS =3D *- ]] + then + ISOLCPUS=3D${ISOLCPUS}$LASTISOLCPU + fi + ISOLCPUS=3D${ISOLCPUS},$CPU + LASTISOLCPU=3D$CPU + fi + done + [[ "$ISOLCPUS" =3D *- ]] && ISOLCPUS=3D${ISOLCPUS}$LASTISOLCPU + [[ "$EXPECT_VAL" =3D "$ISOLCPUS" ]] +} + +test_fail() +{ + TESTNUM=3D$1 + TESTTYPE=3D$2 + ADDINFO=3D$3 + echo "Test $TEST[$TESTNUM] failed $TESTTYPE check!" + [[ -n "$ADDINFO" ]] && echo "*** $ADDINFO ***" + eval echo \${$TEST[$I]} + echo + dump_states + exit 1 +} + # # Run cpuset state transition test # $1 - test matrix name @@ -536,88 +691,80 @@ run_state_test() { TEST=3D$1 CONTROLLER=3Dcpuset - CPULIST=3D0-6 I=3D0 eval CNT=3D"\${#$TEST[@]}" =20 reset_cgroup_states - echo $CPULIST > cpuset.cpus - echo root > cpuset.cpus.partition console_msg "Running state transition test ..." =20 while [[ $I -lt $CNT ]] do echo "Running test $I ..." > /dev/console + [[ $VERBOSE -gt 1 ]] && { + echo "" + eval echo \"\${$TEST[$I]}\" + } eval set -- "\${$TEST[$I]}" - ROOT=3D$1 - OLD_A1=3D$2 - OLD_A2=3D$3 - OLD_A3=3D$4 - OLD_B1=3D$5 - NEW_A1=3D$6 - NEW_A2=3D$7 - NEW_A3=3D$8 - NEW_B1=3D$9 - RESULT=3D${10} - ECPUS=3D${11} - STATES=3D${12} - - set_ctrl_state_noerr . $ROOT + OLD_A1=3D$1 + OLD_A2=3D$2 + OLD_A3=3D$3 + OLD_B1=3D$4 + NEW_A1=3D$5 + NEW_A2=3D$6 + NEW_A3=3D$7 + NEW_B1=3D$8 + RESULT=3D$9 + ECPUS=3D${10} + STATES=3D${11} + ICPUS=3D${12} + + set_ctrl_state_noerr B1 $OLD_B1 set_ctrl_state_noerr A1 $OLD_A1 set_ctrl_state_noerr A1/A2 $OLD_A2 set_ctrl_state_noerr A1/A2/A3 $OLD_A3 - set_ctrl_state_noerr B1 $OLD_B1 RETVAL=3D0 set_ctrl_state A1 $NEW_A1; ((RETVAL +=3D $?)) set_ctrl_state A1/A2 $NEW_A2; ((RETVAL +=3D $?)) set_ctrl_state A1/A2/A3 $NEW_A3; ((RETVAL +=3D $?)) set_ctrl_state B1 $NEW_B1; ((RETVAL +=3D $?)) =20 - [[ $RETVAL -ne $RESULT ]] && { - echo "Test $TEST[$I] failed result check!" - eval echo \"\${$TEST[$I]}\" - dump_states - exit 1 - } + [[ $RETVAL -ne $RESULT ]] && test_fail $I result =20 [[ -n "$ECPUS" && "$ECPUS" !=3D . ]] && { check_effective_cpus $ECPUS - [[ $? -ne 0 ]] && { - echo "Test $TEST[$I] failed effective CPU check!" - eval echo \"\${$TEST[$I]}\" - echo - dump_states - exit 1 - } + [[ $? -ne 0 ]] && test_fail $I "effective CPU" } =20 - [[ -n "$STATES" ]] && { + [[ -n "$STATES" && "$STATES" !=3D . ]] && { check_cgroup_states $STATES - [[ $? -ne 0 ]] && { - echo "FAILED: Test $TEST[$I] failed states check!" - eval echo \"\${$TEST[$I]}\" - echo - dump_states - exit 1 - } + [[ $? -ne 0 ]] && test_fail $I states } =20 + # Compare the expected isolated CPUs with the actual ones, + # if available + [[ -n "$ICPUS" ]] && { + check_isolcpus $ICPUS + [[ $? -ne 0 ]] && test_fail $I "isolated CPU" \ + "Expect $ICPUS, get $ISOLCPUS instead" + } reset_cgroup_states # # Check to see if effective cpu list changes # - pause 0.05 NEWLIST=3D$(cat cpuset.cpus.effective) + [[ $NEWLIST !=3D $CPULIST ]] && { + # Wait a bit longer & recheck + pause 0.05 + NEWLIST=3D$(cat cpuset.cpus.effective) + } [[ $NEWLIST !=3D $CPULIST ]] && { echo "Effective cpus changed to $NEWLIST after test $I!" exit 1 } - [[ -n "$VERBOSE" ]] && echo "Test $I done." + [[ $VERBOSE -gt 0 ]] && echo "Test $I done." ((I++)) done echo "All $I tests of $TEST PASSED." - - echo member > cpuset.cpus.partition } =20 # @@ -642,6 +789,7 @@ test_inotify() { ERR=3D0 PRS=3D/tmp/.prs_$$ + cd $CGROUP2/test [[ -f $WAIT_INOTIFY ]] || { echo "wait_inotify not found, inotify test SKIPPED." return @@ -655,7 +803,7 @@ test_inotify() rm -f $PRS wait_inotify $PWD/cpuset.cpus.partition $PRS & pause 0.01 - set_ctrl_state . "O1-0" + set_ctrl_state . "O1=3D0" pause 0.01 check_cgroup_states ".:P-1" if [[ $? -ne 0 ]] @@ -689,5 +837,3 @@ run_state_test TEST_MATRIX test_isolated test_inotify echo "All tests PASSED." -cd .. -rmdir test --=20 2.31.1