From nobody Mon Jun 8 04:15:26 2026 Received: from mail-dl1-f73.google.com (mail-dl1-f73.google.com [74.125.82.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E6FC537F739 for ; Tue, 2 Jun 2026 07:31:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.82.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780385521; cv=none; b=sYomP+2USpSMQm+2GYeU7eqxnwEzByeABC5DC6w/XLz5Hsv22StGlxSWty+1eNODG+iD+AJt+CJ92nSZMgUqad/DuDDgiHrSx5wqJ0FVXqcoG6Ew2fT07D+RzjaaY6+Mbv+GhfQWuhUkDXCOwenJlIbTMptLobq+bT0xySjZiqg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780385521; c=relaxed/simple; bh=4OF4CoTID6RNso9R76NGGFryDxCJQuD42DBxRoyx3vM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=cyLZrOJpc/dW7CER3j9BS6EvQUrhqNPviqaOgzcAVD3yFHIcxNipfvUDmzdo0wsqOAMIGZ6BmbD6o2B0XdrzSzI1xXVE+LDE3l+u5QPmkSttEy1NpjwJ5seWZcyG+3eKSuF18/PX3eiwA5xqL+Ka7KyH2uCjFKufNcjfe1tYeAM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--irogers.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=MOVCwLSw; arc=none smtp.client-ip=74.125.82.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--irogers.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="MOVCwLSw" Received: by mail-dl1-f73.google.com with SMTP id a92af1059eb24-137dd5100c7so3689740c88.1 for ; Tue, 02 Jun 2026 00:31:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780385516; x=1780990316; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nzzt1hK/sF2nYHHamiQNl90dUvzbnZsDg6tX+vvjQQk=; b=MOVCwLSw3QSf3YfYrgrbaqfnsU237U+ZxUJJVk9s9JuVmJ3FErCV/mq8Gsh+kx988G 6ZM2NkOqWRJzrrowxBl7Fzz4pXHXLaTcw1NKIyDMj2TQa9hwjJbaSVmkpyqy5B9F2s0e yPsKFnUQX5hFfrrxmpRjDLlLHlKCdpI5LnBh+JKb5UjigZS7jPsaICwAu3Ce30GdKnKp NcCmk5EasLWYJjobDdVr89iyWkfVPLFD8R4kD4kxvw+8aOxwiXlpanbX/a8jz/RJ7CKR QVE9JZKAQTUSvhe1+BoH9qvAIHpjFidM2gz6WhcnmSIK70A5ahDzk5wMTE8VdX96beeA zwLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780385516; x=1780990316; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nzzt1hK/sF2nYHHamiQNl90dUvzbnZsDg6tX+vvjQQk=; b=piRn9gayyjee/t9To4Ad1pyMNeqWfnOW9uS0Yzzklm9JqPCaRF4bvhmaoE8quIgqtN rJj7iYZqmYhlMbMo7MEdW0BujqHUOut6ijMl7WIcCFhlg70/3wl+xmUTvdn6LFFkW4oV S1L1U65ZxPWOn5Z6eq+xLO6QPbUvXJzPBfGsaCBOXHcx+9HInNOc6aSUDCki3Y81w47E t76l/izLOrBcbcL57hxIehf0qxAVCO5S7vTflKrgdnqDMvndpW4yKf3mYMYVrmyPAza5 V92TBGmqoEBOW0y/2AyOuWp9mRKtA36FbJrd7j6b6KHXQzcAaR2gPZk4QA4slf0/hPIv bGBw== X-Forwarded-Encrypted: i=1; AFNElJ9qnINBOWzcpNBcwcQhL4AeSiVqZpzEh2F2Il5O6M7D96WDrXuy4wx1DKrGRbyp7h+/fENUQ/sQws3Nxfs=@vger.kernel.org X-Gm-Message-State: AOJu0YztoZif0CNvwlWOGFlqeVnfMLaTxSCxhQgZ+bEGYkhyxEnJjSlU z9V3qWwA9rb5u8suKzjVgGBj7AYYO1k7C6k8ZXYJ7tW6ZhAisLb7QgNQrdLAcp4MMD8kPsXuuWI R4QNxs67Khg== X-Received: from dlea3-n2.prod.google.com ([2002:a05:701b:4203:20b0:135:dd6a:be57]) (user=irogers job=prod-delivery.src-stubby-dispatcher) by 2002:a05:7022:41a0:b0:137:ec47:8ff0 with SMTP id a92af1059eb24-137ec479cefmr2004734c88.40.1780385515469; Tue, 02 Jun 2026 00:31:55 -0700 (PDT) Date: Tue, 2 Jun 2026 00:31:23 -0700 In-Reply-To: <20260602073132.2653307-1-irogers@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260601061401.1541457-1-irogers@google.com> <20260602073132.2653307-1-irogers@google.com> X-Mailer: git-send-email 2.54.0.929.g9b7fa37559-goog Message-ID: <20260602073132.2653307-9-irogers@google.com> Subject: [PATCH v7] perf test: Refactor parallel poll loop to drain all pipes simultaneously From: Ian Rogers To: irogers@google.com, acme@kernel.org, namhyung@kernel.org Cc: adrian.hunter@intel.com, alexander.shishkin@linux.intel.com, james.clark@linaro.org, jolsa@kernel.org, linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, mingo@redhat.com, peterz@infradead.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When running tests in parallel with verbose output (-v), child processes write to pipes. If a test produces significant output (e.G. Granite Rapids metric parsing printing hundreds of lines), it fills the 64KB pipe buffer and blocks. Previously, the parent harness (finish_test) only polled the pipe of the current test waiting to be printed. Other children blocked indefinitely until the parent reached them, severely sequentializing execution. Address this by implementing finish_tests_parallel() to poll and drain output pipes from all running children simultaneously into per-child buffers, employing safe strbuf_addstr string operations alongside thorough variable orderings for strict ISO C90 compliance. Reaping occurs out of order as children finish, while final result printing remains strictly in order. This drops parallel verbose execution time for the PMU events suite from ~35 seconds down to ~5.9 seconds. Assisted-by: Gemini-CLI:Google Gemini 3 Signed-off-by: Ian Rogers --- tools/perf/tests/builtin-test.c | 242 +++++++++++++++++++++++++++++++- 1 file changed, 240 insertions(+), 2 deletions(-) diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-tes= t.c index 2ccb52a776cc..0b33bc74826c 100644 --- a/tools/perf/tests/builtin-test.c +++ b/tools/perf/tests/builtin-test.c @@ -302,6 +302,9 @@ struct child_test { struct test_suite *test; int suite_num; int test_case_num; + struct strbuf err_output; + int result; + bool done; }; =20 static jmp_buf run_test_jmp_buf; @@ -356,6 +359,11 @@ static int run_test_child(struct child_process *proces= s) =20 #define TEST_RUNNING -3 =20 +static struct pollfd *global_pfds; +static size_t *global_pfd_indices; + +static int strbuf_addstr_safe(struct strbuf *sb, const char *s); + static int print_test_result(struct test_suite *t, int curr_suite, int cur= r_test_case, int result, int width, int running) { @@ -499,16 +507,241 @@ static void finish_test(struct child_test **child_te= sts, int running_test, int c } /* Clean up child process. */ ret =3D finish_command(&child_test->process); + child_test->process.pid =3D 0; + if (child_test->err_output.len > 0) { + struct strbuf merged =3D STRBUF_INIT; + + if (child_test->err_output.buf) + strbuf_addstr_safe(&merged, child_test->err_output.buf); + if (err_output.buf) + strbuf_addstr_safe(&merged, err_output.buf); + strbuf_release(&err_output); + err_output =3D merged; + } if (verbose > 1 || (verbose =3D=3D 1 && ret =3D=3D TEST_FAIL)) fprintf(stderr, "%s", err_output.buf); =20 strbuf_release(&err_output); + strbuf_release(&child_test->err_output); print_test_result(t, curr_suite, curr_test_case, ret, width, /*running=3D= */0); if (err > 0) close(err); zfree(&child_tests[running_test]); } =20 +static int strbuf_addstr_safe(struct strbuf *sb, const char *s) +{ + sigset_t set, oldset; + int ret; + + sigemptyset(&set); + sigaddset(&set, SIGINT); + sigaddset(&set, SIGTERM); + sigprocmask(SIG_BLOCK, &set, &oldset); + ret =3D strbuf_addstr(sb, s); + sigprocmask(SIG_SETMASK, &oldset, NULL); + return ret; +} + +static void drain_child_process_err(struct child_test *child) +{ + char buf[512]; + ssize_t len; + + while ((len =3D read(child->process.err, buf, sizeof(buf) - 1)) > 0) { + buf[len] =3D '\0'; + strbuf_addstr_safe(&child->err_output, buf); + } +} + +static void handle_child_pipe_activity(struct child_test *child, short rev= ents) +{ + if (!revents) + return; + + drain_child_process_err(child); + /* + * If the child closed its end of the pipe (EOF) or encountered + * an error, close the file descriptor immediately and set it + * to -1. This removes it from the pfds array for subsequent + * iterations, preventing a tight CPU busy-loop while waiting + * for the process itself to exit. + */ + if (revents & (POLLHUP | POLLERR | POLLNVAL)) { + close(child->process.err); + child->process.err =3D -1; + } +} + +static int finish_tests_parallel(struct child_test **child_tests, size_t n= um_tests, int width) +{ + size_t next_to_print =3D 0; + struct pollfd *pfds; + size_t *pfd_indices; + size_t num_pfds =3D 0; + int last_running =3D -1; + size_t i; + int last_suite_printed =3D -1; + + global_pfds =3D calloc(num_tests, sizeof(*pfds)); + global_pfd_indices =3D calloc(num_tests, sizeof(*pfd_indices)); + pfds =3D global_pfds; + pfd_indices =3D global_pfd_indices; + if (!pfds || !pfd_indices) { + free(pfds); + free(pfd_indices); + global_pfds =3D NULL; + global_pfd_indices =3D NULL; + return -ENOMEM; + } + + for (i =3D 0; i < num_tests; i++) { + struct child_test *child =3D child_tests[i]; + + if (!child) + continue; + strbuf_init(&child->err_output, 0); + if (child->process.err > 0) + fcntl(child->process.err, F_SETFL, O_NONBLOCK); + } + + while (next_to_print < num_tests) { + size_t running_count =3D 0; + size_t p; + + while (next_to_print < num_tests && + (!child_tests[next_to_print] || child_tests[next_to_print]->done)) + next_to_print++; + + if (next_to_print >=3D num_tests) + break; + + num_pfds =3D 0; + + for (i =3D next_to_print; i < num_tests; i++) { + struct child_test *child =3D child_tests[i]; + + if (!child || child->done) + continue; + + if (!check_if_command_finished(&child->process)) + running_count++; + + if (child->process.err > 0) { + pfds[num_pfds].fd =3D child->process.err; + pfds[num_pfds].events =3D POLLIN | POLLERR | POLLHUP | POLLNVAL; + pfd_indices[num_pfds] =3D i; + num_pfds++; + } + } + + if (perf_use_color_default && running_count !=3D (size_t)last_running) { + struct child_test *next_child =3D child_tests[next_to_print]; + + if (last_running !=3D -1) + fprintf(debug_file(), PERF_COLOR_DELETE_LINE); + + if (next_child) { + if (test_suite__num_test_cases(next_child->test) > 1 && + last_suite_printed !=3D next_child->suite_num) { + pr_info("%3d: %-*s:\n", next_child->suite_num + 1, width, + test_description(next_child->test, -1)); + last_suite_printed =3D next_child->suite_num; + } + print_test_result(next_child->test, next_child->suite_num, + next_child->test_case_num, TEST_RUNNING, width, + running_count); + } + last_running =3D running_count; + } + + if (num_pfds =3D=3D 0) { + if (running_count > 0) + usleep(10 * 1000); + } else { + int pret =3D poll(pfds, num_pfds, 100); + + if (pret > 0) { + for (p =3D 0; p < num_pfds; p++) { + size_t idx =3D pfd_indices[p]; + + handle_child_pipe_activity(child_tests[idx], + pfds[p].revents); + } + } + } + + for (i =3D next_to_print; i < num_tests; i++) { + struct child_test *child =3D child_tests[i]; + + if (!child || child->done) + continue; + + if (check_if_command_finished(&child->process)) { + if (child->process.err > 0) { + drain_child_process_err(child); + close(child->process.err); + child->process.err =3D -1; + } + child->result =3D finish_command(&child->process); + child->process.pid =3D 0; + child->done =3D true; + } + } + + while (next_to_print < num_tests) { + struct child_test *child =3D child_tests[next_to_print]; + + if (!child) { + next_to_print++; + continue; + } + if (!child->done) + break; + + if (perf_use_color_default && last_running !=3D -1) { + fprintf(debug_file(), PERF_COLOR_DELETE_LINE); + last_running =3D -1; + } + + if (test_suite__num_test_cases(child->test) > 1 && + last_suite_printed !=3D child->suite_num) { + pr_info("%3d: %-*s:\n", child->suite_num + 1, width, + test_description(child->test, -1)); + last_suite_printed =3D child->suite_num; + } + + if (verbose > 1) { + if (test_suite__num_test_cases(child->test) > 1) { + pr_info("%3d.%1d: %s:\n", child->suite_num + 1, + child->test_case_num + 1, + test_description(child->test, + child->test_case_num)); + } else { + pr_info("%3d: %s:\n", child->suite_num + 1, + test_description(child->test, -1)); + } + } + + if (verbose > 1 || (verbose =3D=3D 1 && child->result =3D=3D TEST_FAIL)) + fprintf(stderr, "%s", child->err_output.buf); + + print_test_result(child->test, child->suite_num, child->test_case_num, + child->result, width, 0); + strbuf_release(&child->err_output); + child_tests[next_to_print] =3D NULL; + zfree(&child); + next_to_print++; + } + } + + free(global_pfds); + free(global_pfd_indices); + global_pfds =3D NULL; + global_pfd_indices =3D NULL; + return 0; +} + static int start_test(struct test_suite *test, int curr_suite, int curr_te= st_case, struct child_test **child, int width, int pass) { @@ -671,8 +904,9 @@ static int __cmd_test(struct test_suite **suites, int a= rgc, const char *argv[], } if (!sequential) { /* Parallel mode starts tests but doesn't finish them. Do that now. */ - for (size_t x =3D 0; x < num_tests; x++) - finish_test(child_tests, x, num_tests, width); + err =3D finish_tests_parallel(child_tests, num_tests, width); + if (err) + goto err_out; } } err_out: @@ -683,6 +917,10 @@ static int __cmd_test(struct test_suite **suites, int = argc, const char *argv[], for (size_t x =3D 0; x < num_tests; x++) finish_test(child_tests, x, num_tests, width); } + free(global_pfds); + free(global_pfd_indices); + global_pfds =3D NULL; + global_pfd_indices =3D NULL; free(child_tests); return err; } --=20 2.54.0.929.g9b7fa37559-goog