From nobody Thu Dec 18 01:38:06 2025 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A85CE156879 for ; Wed, 18 Dec 2024 20:20:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734553259; cv=none; b=iSX8pmGptBCnsJk0mXRq3UJYnFOTPgvIwUnLVoHeDXB2OSAAhqtz7nbY1uytKuMqQA2q/QPyKTlaYKdxy9ZeobAMNa7Va4Y5E+63Tdh24d4yLKQhnoOJ06V8ob/s0XUP1KPmhU1TyzDzMuKPztU1L13jqVe/VsCsqIJyulqECkg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734553259; c=relaxed/simple; bh=KEM/E0Si5RbMcLSm2mW/BWLExpu+IJ76jQq+OhCS59s=; h=Date:In-Reply-To:Mime-Version:Message-ID:Subject:From:To:Cc: Content-Type; b=BplnDUn9/Nw/+aweFcA+Pngd7YTHMUWFjO72VMqKTQHbzJF8z8jLyRG8+qwEgnyV4lKmEkEflgUVjjvreEKTn1pV769nxbxF5rOJQQ6RWOkuSMEMO0ZoVwcyk9v9lSwVvk+/h/W1yYMCbMw07aW0g3O071arbheNzTShu4HqtK4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--elsk.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=UTxoouM0; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--elsk.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="UTxoouM0" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2ef775ec883so60914a91.1 for ; Wed, 18 Dec 2024 12:20:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1734553257; x=1735158057; darn=vger.kernel.org; h=cc:to:from:subject:message-id:mime-version:in-reply-to:date:from:to :cc:subject:date:message-id:reply-to; bh=72JS/mGeqQkAuY5dd/3ZFQi+TNYKkC9MoYq9IVHGNUE=; b=UTxoouM06Sq21UAGjq5pn2TM/KRrVCuNwOwZO61Ed6l7kJ9MMI0oHizGhXXxqibm1d Y6uxjex3vo6JGt5JXBY0o/83ulCXJU4XEDPMZJlJCmy8rv47t5gqfPBaKrKNX9fKwaQU 9YDJh1HLUW/bRCgW/esi2gkMekuNeT/2ABZp4hnEwypvcyJUjdWkPLORiFm7LNbFW2aj eLh+KrLl1QN6enwbb9ctB9b7Ag1l5oJVG7lWd6Jzb0RnTUkIhK1cldFlWt6chHSF3MSG 91jr2/CvtIEocG5avAb5O7M3TVvl2d5krsvkVY3mtD3lyGE/it2WaN/qyXxQMkFB+S3n bsJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734553257; x=1735158057; h=cc:to:from:subject:message-id:mime-version:in-reply-to:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=72JS/mGeqQkAuY5dd/3ZFQi+TNYKkC9MoYq9IVHGNUE=; b=i4qv7pXTiUEapepj+UPbNwSSpmrYylsuri4+7U4Te0QMQbYHvcy+9jJxtSeNuluJbc EBKb/JGmz6gezEnYcU/07rhTa6J4hpu62mqYiRb8/R7l4KoGFFyWxkNI29wVVvZav3Dv qzb6Kmf3jXxOZJ1DB9TrViMR3zGuLf1r/iuIEABHSS5pybr5lAEHPoNz6KEmZUPB5u7E Q7VhHGo4x0dvKp31ZlxUxw4t7Dg6oVJUT7zJSQzLMJtRvgvSnIAIC+KCSZ0fyo/Tq3dU UD+sX3kiA2ANVxQ8PTp7hs0pTfjOY8D4chPZHjFJXpX6VV7EMj5DJg9FrVijp00vZKmz Zdbg== X-Forwarded-Encrypted: i=1; AJvYcCW+2m3d/I6JmmbP4aIG9bcSt7OXfqR5wLOKEvdyIY0aGsxngVqrqpkp0+sLEaebTDK8HwUaCkYs3J63ek4=@vger.kernel.org X-Gm-Message-State: AOJu0Yz2latMYIvXfMnnAtxFnNog7K/0KhwElouGLDzhNlkCJrBp5tjw n0rs+dzh9ojZF//TbnNWh31PT2ZD7BgoPkhluEc3piBdX2Lf9uV9jLTSg+Ax2t8FG4E+OQ== X-Google-Smtp-Source: AGHT+IFmh9W/k66e/72jTDjX5bjPtuR5/cijiHvyZS+xwMH6EUuNvYdt9nA1vNPdYwkZx0Swi8biXhmx X-Received: from pjbph13.prod.google.com ([2002:a17:90b:3bcd:b0:2e0:52d7:183e]) (user=elsk job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:2f0e:b0:2ee:aed2:c15c with SMTP id 98e67ed59e1d1-2f2e9378343mr5391361a91.28.1734553256913; Wed, 18 Dec 2024 12:20:56 -0800 (PST) Date: Wed, 18 Dec 2024 20:20:11 +0000 In-Reply-To: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241218202021.17276-1-elsk@google.com> Subject: [PATCH v3] kheaders: prevent `find` from seeing perl temp files From: HONG Yifan To: Masahiro Yamada , Miguel Ojeda , Matthias Maennich Cc: HONG Yifan , kernel-team@android.com, linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Symptom: The command find ... | xargs ... perl -i occasionally triggers error messages like the following, with the build still succeeding: Can't open /kernel/.tmp_cpio_dir/include/dt-bindings/clock/XX= NX4nW9: No such file or directory. Analysis: With strace, the root cause has been identified to be `perl -i` creating temporary files inside $cpio_dir, which causes `find` to see the temporary files and emit the names. `find` is likely implemented with readdir. POSIX `readdir` says: If a file is removed from or added to the directory after the most recent call to opendir() or rewinddir(), whether a subsequent call to readdir() returns an entry for that file is unspecified. So if the libc that `find` links against choose to return that entry in readdir(), a possible sequence of events is the following: 1. find emits foo.h 2. xargs executes `perl -i foo.h` 3. perl (pid=3D100) creates temporary file `XXXXXXXX` 4. find sees file `XXXXXXXX` and emit it 5. PID 100 exits, cleaning up the temporary file `XXXXXXXX` 6. xargs executes `perl -i XXXXXXXX` 7. perl (pid=3D200) tries to read the file, but it doesn't exist any more. ... triggering the error message. One can reproduce the bug with the following command (assuming PWD contains the list of headers in kheaders.tar.xz) for i in $(seq 100); do find -type f -print0 | xargs -0 -P8 -n1 perl -pi -e 'BEGIN {undef $/;}; s/\/\*((?!SPDX= ).)*?\*\///smg;'; done With a `find` linking against musl libc, the error message is emitted 6/100 times. The fix: This change stores the results of `find` before feeding them into xargs. find and xargs will no longer be able to see temporary files that perl creates after this change. Signed-off-by: HONG Yifan --- v3: (this patch) Change from `cat contents.txt | xargs` to `xargs < contents.txt` to pass shellcheck. Fix typo in commit message. v2: https://lore.kernel.org/all/20241206000012.440827-1-elsk@google.com/ change from `find *.h | xargs perl` to `find > file; cat file | xargs perl` because Masahiro discovered that t= he approach in v1 still causes find to see temporary files. The new approa= ch is more robust. v1: https://lore.kernel.org/all/20241107005831.15434-1-elsk@google.com/ kernel/gen_kheaders.sh | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh index 7e1340da5aca..3b58761b4690 100755 --- a/kernel/gen_kheaders.sh +++ b/kernel/gen_kheaders.sh @@ -84,8 +84,13 @@ for f in $dir_list; done | cpio --quiet -pdu $cpio_dir >/dev/null 2>&1 # Remove comments except SDPX lines -find $cpio_dir -type f -print0 | - xargs -0 -P8 -n1 perl -pi -e 'BEGIN {undef $/;}; s/\/\*((?!SPDX).)*?\*\//= /smg;' +# Use a temporary file to store directory contents to prevent find/xargs f= rom +# seeing temporary files created by perl. +find $cpio_dir -type f -print0 > "${cpio_dir}.contents.txt" +xargs -0 -P8 -n1 \ + perl -pi -e 'BEGIN {undef $/;}; s/\/\*((?!SPDX).)*?\*\///smg;' \ + < "${cpio_dir}.contents.txt" +rm -f "${cpio_dir}.contents.txt" # Create archive and try to normalize metadata for reproducibility. tar "${KBUILD_BUILD_TIMESTAMP:+--mtime=3D$KBUILD_BUILD_TIMESTAMP}" \ -- 2.47.1.613.gc27f4b7a9f-goog