From nobody Sun Nov 24 15:54:47 2024 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BF7C41C3301 for ; Mon, 4 Nov 2024 16:19:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730737170; cv=none; b=kpzN10jOVcVCWYBCNjK2ptSesgrcdYIlWVGFEtuMnSnDebsuk9VsvzV2YKaaVL3IfCP00hjULJ4Y2S68sd+ffu8FZumvMVd4jZrPI9mj633iVPxeHOYf3uAifrPhoweH8Mb06DJy+mzFrdS5dwTl7Ye3M1IQwQkQUd5/13BLe2I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730737170; c=relaxed/simple; bh=lUc5TertTv6v/DNguptt7P2xUp+ooVpcLKkDyjzxF5E=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=bLLdGruKZVKNZ+R+gj3X++UoF0drouHEUB0WeZFwlNTCrQIy6uIs9Cap6ED2+3d01/251gSGKw1o2AdyCHaDBQM4YIg1tfzAST1eUU093Ypi7rnDLwFHnfpi/D7YZjVL1HG9Aghw5qSNz67J6k/L8ryF8JRwobqmQLhQ6XVpwnQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--elver.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Sd4wqmLV; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--elver.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Sd4wqmLV" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-37d609ef9f7so2286830f8f.3 for ; Mon, 04 Nov 2024 08:19:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1730737166; x=1731341966; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=P9zIGQ63tMUKGGfo5aLFc1OFxRhvBbjK0JxcXzvrQ2E=; b=Sd4wqmLVaX/Xglj1JO7x6cawt9hiVRm50+PsxRNYH977iwa3bkxMssSjB3g1/w3iKV 3c2aVbh/Z3JJRrM1WkSLi4My5/etysSQY1ZixbjpHXZ14QYwjdbLnQRs2NKpZkzalyTw zz0RqxraR7Vh/jMoG7bhZTSOUQQDpX75x+vtIZvXJVNvc6NR7/NuKg+q1KCyQiaZVJSd wMbCKDRVdIlRbJkgFz+XxQGnK74sl+My18GXiWJwFl60RNIiyGSNDy3Etk3dD4ex0ZFI 2IM9yxOE+DpCNujiurpq1XD23o/UYyQGmi0Mhik5tDcgdKdmEq1LQYACJ+5kT1uX2LdD j3NA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730737166; x=1731341966; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=P9zIGQ63tMUKGGfo5aLFc1OFxRhvBbjK0JxcXzvrQ2E=; b=wsfqYeXuHCDLSNKPQVXAYju1mr+e6iyYJqsIqerjImq+qDN3IfKaLlZ0QGKi7Xrk+s pukoA+cAsGqRjhXq4WMK82Ug+6CXTMewFtUUYBiWQLY+kVgyJHnNDoJ/QIzhtYdFuhZN BJAp6LJDKOrirWdhdej4RXwoMsVRp4H9s+oRchsy64hZydLaVEppFO+HXlQ3kKIVfkOG bZZ0bXR0wlWwQUPJoH08L2NQzPx257gWiuIj98Z6fApqxc5w1cP7PeQsaNN5giZ6xeqT AU3vbIystWY1Mypbrla4+ioaigcswBmIudn+rgo/cJGTiyvxodsUfcwvvDpTBScWre8D qcmA== X-Forwarded-Encrypted: i=1; AJvYcCWTMpPzsvzdwqxyIBt093OL0P3ayreb5RIDf0si6KChUYa3VtbzxDnNC1P4Z3BL89d6VGJN1isr0MbHFv4=@vger.kernel.org X-Gm-Message-State: AOJu0YwjEbgdys2oBAl0BBVqCRZLT1DzgJG3YNkPdCTFQFNBCcaKMiEF Pp+IZaEv+3NDowgQItAlpK95i03HNwYCfNXwq5CucYHOD9suVN3TFqj1oW1HYfN250NgaJOIsw= = X-Google-Smtp-Source: AGHT+IHS0+6h/UBLulc4DyiTN6UHNskaVEbiMgEJ6V0E0xUDQO9SGhU+3nV/ZEnM+3FaKpvsFAFewAG/VQ== X-Received: from elver.muc.corp.google.com ([2a00:79e0:9c:201:dc4d:3b27:d746:73ee]) (user=elver job=sendgmr) by 2002:a5d:44d0:0:b0:37e:d5a2:b104 with SMTP id ffacd0b85a97d-381be7cf9ecmr7013f8f.6.1730737165815; Mon, 04 Nov 2024 08:19:25 -0800 (PST) Date: Mon, 4 Nov 2024 16:43:05 +0100 In-Reply-To: <20241104161910.780003-1-elver@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241104161910.780003-1-elver@google.com> X-Mailer: git-send-email 2.47.0.163.g1226f6d8fa-goog Message-ID: <20241104161910.780003-2-elver@google.com> Subject: [PATCH v2 1/5] time/sched_clock: Swap update_clock_read_data() latch writes From: Marco Elver To: elver@google.com, Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng Cc: "Paul E. McKenney" , Thomas Gleixner , Mark Rutland , Dmitry Vyukov , kasan-dev@googlegroups.com, linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Swap the writes to the odd and even copies to make the writer critical section look like all other seqcount_latch writers. Signed-off-by: Marco Elver --- v2: * New patch. --- kernel/time/sched_clock.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/kernel/time/sched_clock.c b/kernel/time/sched_clock.c index 68d6c1190ac7..85595fcf6aa2 100644 --- a/kernel/time/sched_clock.c +++ b/kernel/time/sched_clock.c @@ -119,9 +119,6 @@ unsigned long long notrace sched_clock(void) */ static void update_clock_read_data(struct clock_read_data *rd) { - /* update the backup (odd) copy with the new data */ - cd.read_data[1] =3D *rd; - /* steer readers towards the odd copy */ raw_write_seqcount_latch(&cd.seq); =20 @@ -130,6 +127,9 @@ static void update_clock_read_data(struct clock_read_da= ta *rd) =20 /* switch readers back to the even copy */ raw_write_seqcount_latch(&cd.seq); + + /* update the backup (odd) copy with the new data */ + cd.read_data[1] =3D *rd; } =20 /* --=20 2.47.0.163.g1226f6d8fa-goog From nobody Sun Nov 24 15:54:47 2024 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B6A0D1C4A01 for ; Mon, 4 Nov 2024 16:19:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730737171; cv=none; b=U8G393CfFsXILPrwLcptHeN3CraHJPPwAEcGxUev17UTIB43WWlAY8L/eg1W6ohSNibA6UOqxqEpzhyhd7kVsLibCOLRAA+AToLny0erME/Hw24nYYMBYXmbXMlNNLN+xAClpQscgwVtpxAr/1OoemCR7tt8lEJ3O1ASuvjTRgI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730737171; c=relaxed/simple; bh=Z+EtvtNAVddb0LGVKPzUJYIz4oWFMbDXlfVxfY+njgY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=QlNxidg62/bLSZsXQ/JnlfY0cJuUupe8ZFVIJA4wPM1MWrSpK3M3p7IJNVmquwojOSnsP33yTtkQ5EINv80V/8CgTIHSV1Px5gdHLY6css6mZitcbCC/DOLk+qpx52NPpKnR10Jc3qU9cv/RiIyeptJt7nYWj+T1DL0zk/7eKBY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--elver.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=diyB5BCR; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--elver.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="diyB5BCR" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6ea8a5e862eso19102737b3.0 for ; Mon, 04 Nov 2024 08:19:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1730737169; x=1731341969; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=F81a+6CGvKtmngjrkoxGzbhKVD48308MotgDbr+zIK4=; b=diyB5BCRikRV80sb7qpnw3Y/dd3LS3Ho4L1SN+NKe0azpjw05jPIop5owEPrr/VKZN aFTaWrBAzmtfNmQ3ZX8GMZ5ekLlEP+WUzrprfrlGwnPfRRNakdrzoYayrahPySMhNVLx VHCMtVgxzB6az8f4b5N1BtKc06EWoSZMZkgXtWt96nZredqocQo9lYngTT127Tjys2GI v1RYptZ4xFrNAbFVVlMnBa7ls5ore5O2U6MS+ASj7IwVbyMKFHoUMvx49JgTz8Zgyf4U UZn1E+4D2xxe9FrXM5H5aFU6nQFUmjtJwiMVTgeDyWRZ1OqqFlI1y4N9VeWpv8U4vIDD wPiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730737169; x=1731341969; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=F81a+6CGvKtmngjrkoxGzbhKVD48308MotgDbr+zIK4=; b=vBZr/d6qyaxl4n2fB0Y9ZoTyF5rkdCQkHwGWIprGJacWr2KgYsq5vvFx9UVJiVkPfV uWIBsvcEG63/k5Kac88TCpXtksRvdiNtvvxV6CL0AEeuB6872pAAT9ya1YaxVsZkJk34 9G/8ioi88qtF+ngIrYTomEbRuqgWljKvmVD+LGXWlgliXSEg87uO6BnO4ek/6FaJDFR0 SuQU0a59XE+urDClUlckkUYQ6vu1r1+aWCHl6Qb3cyIlW4IS30lE6iDHj5LkQPjffOqm +/CyEl0SmRTg8bfs0dtNiuW02WzPmfd9sCvqTkk94useO2parCfM0de/vyD6DM2ligQQ BIYQ== X-Forwarded-Encrypted: i=1; AJvYcCUiU4iquhmB54JU4dkl0B4xo228p2uEODUwrggmbRC5WrBxyhe8ZNvJdDp1Kd6rL+02ryFrD2igr2RsqjI=@vger.kernel.org X-Gm-Message-State: AOJu0Yz7lOxSg4E9XVQseucKS/L+98wp6xzxa3bOY3XodbopC/SrTJpA bxJgs/Gupc47WhksobyWl9Ulk4YajO9gA88udRZ8B4rSzBSux1qr6zsx4MK0e6daTIfqr9c83Q= = X-Google-Smtp-Source: AGHT+IEVgE/MIej8RE4qbEDqP1YW+GMNJ/swOW+IJ2MrFumHiQViz2zIr7y971A6SVkDwhd+PHYo+OVUbQ== X-Received: from elver.muc.corp.google.com ([2a00:79e0:9c:201:dc4d:3b27:d746:73ee]) (user=elver job=sendgmr) by 2002:a0d:c601:0:b0:6dd:bb6e:ec89 with SMTP id 00721157ae682-6ea55787f6bmr1375427b3.2.1730737168609; Mon, 04 Nov 2024 08:19:28 -0800 (PST) Date: Mon, 4 Nov 2024 16:43:06 +0100 In-Reply-To: <20241104161910.780003-1-elver@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241104161910.780003-1-elver@google.com> X-Mailer: git-send-email 2.47.0.163.g1226f6d8fa-goog Message-ID: <20241104161910.780003-3-elver@google.com> Subject: [PATCH v2 2/5] time/sched_clock: Broaden sched_clock()'s instrumentation coverage From: Marco Elver To: elver@google.com, Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng Cc: "Paul E. McKenney" , Thomas Gleixner , Mark Rutland , Dmitry Vyukov , kasan-dev@googlegroups.com, linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Most of sched_clock()'s implementation is ineligible for instrumentation due to relying on sched_clock_noinstr(). Split the implementation off into an __always_inline function __sched_clock(), which is then used by the noinstr and instrumentable version, to allow more of sched_clock() to be covered by various instrumentation. This will allow instrumentation with the various sanitizers (KASAN, KCSAN, KMSAN, UBSAN). For KCSAN, we know that raw seqcount_latch usage without annotations will result in false positive reports: tell it that all of __sched_clock() is "atomic" for the latch writer; later changes in this series will take care of the readers. Link: https://lore.kernel.org/all/20241030204815.GQ14555@noisy.programming.= kicks-ass.net/ Co-developed-by: Peter Zijlstra (Intel) Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Marco Elver --- v2: * New patch. --- kernel/time/sched_clock.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/kernel/time/sched_clock.c b/kernel/time/sched_clock.c index 85595fcf6aa2..29bdf309dae8 100644 --- a/kernel/time/sched_clock.c +++ b/kernel/time/sched_clock.c @@ -80,7 +80,7 @@ notrace int sched_clock_read_retry(unsigned int seq) return raw_read_seqcount_latch_retry(&cd.seq, seq); } =20 -unsigned long long noinstr sched_clock_noinstr(void) +static __always_inline unsigned long long __sched_clock(void) { struct clock_read_data *rd; unsigned int seq; @@ -98,11 +98,23 @@ unsigned long long noinstr sched_clock_noinstr(void) return res; } =20 +unsigned long long noinstr sched_clock_noinstr(void) +{ + return __sched_clock(); +} + unsigned long long notrace sched_clock(void) { unsigned long long ns; preempt_disable_notrace(); - ns =3D sched_clock_noinstr(); + /* + * All of __sched_clock() is a seqcount_latch reader critical section, + * but relies on the raw helpers which are uninstrumented. For KCSAN, + * mark all accesses in __sched_clock() as atomic. + */ + kcsan_nestable_atomic_begin(); + ns =3D __sched_clock(); + kcsan_nestable_atomic_end(); preempt_enable_notrace(); return ns; } --=20 2.47.0.163.g1226f6d8fa-goog From nobody Sun Nov 24 15:54:47 2024 Received: from mail-ed1-f73.google.com (mail-ed1-f73.google.com [209.85.208.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1DFCE1C75FA for ; Mon, 4 Nov 2024 16:19:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730737175; cv=none; b=f2s73TxzPUlcJBtje1NjUOmmX2GLDxo06NHbWq8Tfmhz1NZ4OMFdxbQpFEnxtCHHdcNgdXRwCQLkx7uEfOJY84I6aQweYuLXUyIbSnveQaXiZczPUmH/IUiJ+ReiqVK6kOLdbdNDAP93lha1u5wZ8W+C6yCc4hcdZN/CVcBy+k8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730737175; c=relaxed/simple; bh=Up6Nl7a0O4dGH9BKNaqMISi3+v/Z0UXTsR+1mKX6xTE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=KhGmMD/cziGaG2nwfarxKIgjAt/dBWyGWuuy3uf6BY94oSvACaf3gNaZdRrSVjkGxHpOqG7pIYfTSaQZHWi2L09Jb71tULy0zbd6IITABbvS/7L+JlaN3yQlnil/v17AGOZcK5SnGKmlxWbKdF2nGr9d/TaXwLgd0YSfq6AMsq4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--elver.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=x0gPAIoF; arc=none smtp.client-ip=209.85.208.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--elver.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="x0gPAIoF" Received: by mail-ed1-f73.google.com with SMTP id 4fb4d7f45d1cf-5cbadbef8edso2980771a12.3 for ; Mon, 04 Nov 2024 08:19:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1730737171; x=1731341971; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=18LLtGEOGzu+73DHRju+ymYzFojyLpzj5nOA/GmHSqU=; b=x0gPAIoFxTbBNqkE+RXlvqaV72b26E2WQVWxcUgyEB7904T2ZfA3apDunA2GZT8ZLo 7jOhUMMEYZQXTtOcZyXOWrkmESRSfEaiOKAYHSCJErR/49GfEEL1qfQZI5jM15LkEMqH rLpU+0fDNAVHfVxBG+ZQvpKgG+F8IaBFAOYyAAZ2OWie+xTU2JVHGQLjxbF3BYgCTUI8 2SPPOyDTHFH2MySx3t7R8wcCqaNfUJuct/qFHdsmMcTf6Oy6OGaDDFIrwtSxKja+xBf+ A05Qxl15JdNsA1P5ri+Bj/dXhD9j+xoLdXRQe0a0ogBTjFs1eyvqL6BPjllP4ML6fmfv Om6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730737171; x=1731341971; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=18LLtGEOGzu+73DHRju+ymYzFojyLpzj5nOA/GmHSqU=; b=EwyrU5LkNZGUc++3XHQYCSTwkLxJ6LxRvILSVt4XE7FyBOKPN2mdiuH8bzZ42EQEa/ /WXkUPSOn1ifn7tWZbdTSR/9ek6ZCAUry9XpVYIEfn41jx5QuehiXLLEWDL5pdmJKNtU mTMYqIHzGOBJT0hMvD9x0o/yCw/vSt61q+S7BYorF3ihwYEuxwddqnK+Pmheicckt1hQ 45YEjNnMAwEJLZbcnkZOU7nni+hxTZsW0lMID8ufaVeQcHrwk/NS3mnoZcfjB0IDAXKg IP1HM0leAtSw1e+ujqqqPJxIVI2qeaIs43OZD9XIHm/UCHTMVCLv8kfLb4fPbKK0mOxp yfKw== X-Forwarded-Encrypted: i=1; AJvYcCUCXzzBzf1Nx/Br3guhgukVanCvsu408KfU4jqjH/87OQRLXzJPmsfql+u7nkHjQbzEKHmBuJevVkaRV14=@vger.kernel.org X-Gm-Message-State: AOJu0YyfoeXGpMP8G8pQFvAt59nu8WxPu9fU1cr88vWPgV+wlQWWoRmZ QBibcnH9JfYegnjdkKYWFBsiXui7U514EHeR6wURPBK6e4Q+1uP7t/62ci6O6Ljc/RDIASrNnQ= = X-Google-Smtp-Source: AGHT+IETLznCWJ7078mtEN2zw0LHcGVF522oHU07UWHIjL8ZqDdYAFPjAmLuAomnIYwKTJ+LzM/igFJm+Q== X-Received: from elver.muc.corp.google.com ([2a00:79e0:9c:201:dc4d:3b27:d746:73ee]) (user=elver job=sendgmr) by 2002:a05:6402:2350:b0:5cb:c081:92b2 with SMTP id 4fb4d7f45d1cf-5cea966ac74mr5044a12.1.1730737171303; Mon, 04 Nov 2024 08:19:31 -0800 (PST) Date: Mon, 4 Nov 2024 16:43:07 +0100 In-Reply-To: <20241104161910.780003-1-elver@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241104161910.780003-1-elver@google.com> X-Mailer: git-send-email 2.47.0.163.g1226f6d8fa-goog Message-ID: <20241104161910.780003-4-elver@google.com> Subject: [PATCH v2 3/5] kcsan, seqlock: Support seqcount_latch_t From: Marco Elver To: elver@google.com, Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng Cc: "Paul E. McKenney" , Thomas Gleixner , Mark Rutland , Dmitry Vyukov , kasan-dev@googlegroups.com, linux-kernel@vger.kernel.org, Alexander Potapenko Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" While fuzzing an arm64 kernel, Alexander Potapenko reported: | BUG: KCSAN: data-race in ktime_get_mono_fast_ns / timekeeping_update | | write to 0xffffffc082e74248 of 56 bytes by interrupt on cpu 0: | update_fast_timekeeper kernel/time/timekeeping.c:430 [inline] | timekeeping_update+0x1d8/0x2d8 kernel/time/timekeeping.c:768 | timekeeping_advance+0x9e8/0xb78 kernel/time/timekeeping.c:2344 | update_wall_time+0x18/0x38 kernel/time/timekeeping.c:2360 | [...] | | read to 0xffffffc082e74258 of 8 bytes by task 5260 on cpu 1: | __ktime_get_fast_ns kernel/time/timekeeping.c:372 [inline] | ktime_get_mono_fast_ns+0x88/0x174 kernel/time/timekeeping.c:489 | init_srcu_struct_fields+0x40c/0x530 kernel/rcu/srcutree.c:263 | init_srcu_struct+0x14/0x20 kernel/rcu/srcutree.c:311 | [...] | | value changed: 0x000002f875d33266 -> 0x000002f877416866 | | Reported by Kernel Concurrency Sanitizer on: | CPU: 1 UID: 0 PID: 5260 Comm: syz.2.7483 Not tainted 6.12.0-rc3-dirty #78 This is a false positive data race between a seqcount latch writer and a re= ader accessing stale data. Since its introduction, KCSAN has never understood the seqcount_latch interface (due to being unannotated). Unlike the regular seqlock interface, the seqcount_latch interface for latch writers never has had a well-defined critical section, making it difficult = to teach tooling where the critical section starts and ends. Introduce an instrumentable (non-raw) seqcount_latch interface, with which we can clearly denote writer critical sections. This both helps readability and tooling like KCSAN to understand when the writer is done updating all latch copies. Link: https://lore.kernel.org/all/20241030204815.GQ14555@noisy.programming.= kicks-ass.net/ Reported-by: Alexander Potapenko Fixes: 88ecd153be95 ("seqlock, kcsan: Add annotations for KCSAN") Co-developed-by: Peter Zijlstra (Intel) Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Marco Elver --- v2: * Introduce new interface, courtesy of Peter Zijlstra. Adjust documentation along with its introduction. --- Documentation/locking/seqlock.rst | 2 +- include/linux/seqlock.h | 86 +++++++++++++++++++++++++------ 2 files changed, 72 insertions(+), 16 deletions(-) diff --git a/Documentation/locking/seqlock.rst b/Documentation/locking/seql= ock.rst index bfda1a5fecad..ec6411d02ac8 100644 --- a/Documentation/locking/seqlock.rst +++ b/Documentation/locking/seqlock.rst @@ -153,7 +153,7 @@ Use seqcount_latch_t when the write side sections canno= t be protected from interruption by readers. This is typically the case when the read side can be invoked from NMI handlers. =20 -Check `raw_write_seqcount_latch()` for more information. +Check `write_seqcount_latch()` for more information. =20 =20 .. _seqlock_t: diff --git a/include/linux/seqlock.h b/include/linux/seqlock.h index fffeb754880f..45eee0e5dca0 100644 --- a/include/linux/seqlock.h +++ b/include/linux/seqlock.h @@ -621,6 +621,23 @@ static __always_inline unsigned raw_read_seqcount_latc= h(const seqcount_latch_t * return READ_ONCE(s->seqcount.sequence); } =20 +/** + * read_seqcount_latch() - pick even/odd latch data copy + * @s: Pointer to seqcount_latch_t + * + * See write_seqcount_latch() for details and a full reader/writer usage + * example. + * + * Return: sequence counter raw value. Use the lowest bit as an index for + * picking which data copy to read. The full counter must then be checked + * with read_seqcount_latch_retry(). + */ +static __always_inline unsigned read_seqcount_latch(const seqcount_latch_t= *s) +{ + kcsan_atomic_next(KCSAN_SEQLOCK_REGION_MAX); + return raw_read_seqcount_latch(s); +} + /** * raw_read_seqcount_latch_retry() - end a seqcount_latch_t read section * @s: Pointer to seqcount_latch_t @@ -635,9 +652,34 @@ raw_read_seqcount_latch_retry(const seqcount_latch_t *= s, unsigned start) return unlikely(READ_ONCE(s->seqcount.sequence) !=3D start); } =20 +/** + * read_seqcount_latch_retry() - end a seqcount_latch_t read section + * @s: Pointer to seqcount_latch_t + * @start: count, from read_seqcount_latch() + * + * Return: true if a read section retry is required, else false + */ +static __always_inline int +read_seqcount_latch_retry(const seqcount_latch_t *s, unsigned start) +{ + kcsan_atomic_next(0); + return raw_read_seqcount_latch_retry(s, start); +} + /** * raw_write_seqcount_latch() - redirect latch readers to even/odd copy * @s: Pointer to seqcount_latch_t + */ +static __always_inline void raw_write_seqcount_latch(seqcount_latch_t *s) +{ + smp_wmb(); /* prior stores before incrementing "sequence" */ + s->seqcount.sequence++; + smp_wmb(); /* increment "sequence" before following stores */ +} + +/** + * write_seqcount_latch_begin() - redirect latch readers to odd copy + * @s: Pointer to seqcount_latch_t * * The latch technique is a multiversion concurrency control method that a= llows * queries during non-atomic modifications. If you can guarantee queries n= ever @@ -665,17 +707,11 @@ raw_read_seqcount_latch_retry(const seqcount_latch_t = *s, unsigned start) * * void latch_modify(struct latch_struct *latch, ...) * { - * smp_wmb(); // Ensure that the last data[1] update is visible - * latch->seq.sequence++; - * smp_wmb(); // Ensure that the seqcount update is visible - * + * write_seqcount_latch_begin(&latch->seq); * modify(latch->data[0], ...); - * - * smp_wmb(); // Ensure that the data[0] update is visible - * latch->seq.sequence++; - * smp_wmb(); // Ensure that the seqcount update is visible - * + * write_seqcount_latch(&latch->seq); * modify(latch->data[1], ...); + * write_seqcount_latch_end(&latch->seq); * } * * The query will have a form like:: @@ -686,13 +722,13 @@ raw_read_seqcount_latch_retry(const seqcount_latch_t = *s, unsigned start) * unsigned seq, idx; * * do { - * seq =3D raw_read_seqcount_latch(&latch->seq); + * seq =3D read_seqcount_latch(&latch->seq); * * idx =3D seq & 0x01; * entry =3D data_query(latch->data[idx], ...); * * // This includes needed smp_rmb() - * } while (raw_read_seqcount_latch_retry(&latch->seq, seq)); + * } while (read_seqcount_latch_retry(&latch->seq, seq)); * * return entry; * } @@ -716,11 +752,31 @@ raw_read_seqcount_latch_retry(const seqcount_latch_t = *s, unsigned start) * When data is a dynamic data structure; one should use regular RCU * patterns to manage the lifetimes of the objects within. */ -static inline void raw_write_seqcount_latch(seqcount_latch_t *s) +static __always_inline void write_seqcount_latch_begin(seqcount_latch_t *s) { - smp_wmb(); /* prior stores before incrementing "sequence" */ - s->seqcount.sequence++; - smp_wmb(); /* increment "sequence" before following stores */ + kcsan_nestable_atomic_begin(); + raw_write_seqcount_latch(s); +} + +/** + * write_seqcount_latch() - redirect latch readers to even copy + * @s: Pointer to seqcount_latch_t + */ +static __always_inline void write_seqcount_latch(seqcount_latch_t *s) +{ + raw_write_seqcount_latch(s); +} + +/** + * write_seqcount_latch_end() - end a seqcount_latch_t write section + * @s: Pointer to seqcount_latch_t + * + * Marks the end of a seqcount_latch_t writer section, after all copies of= the + * latch-protected data have been updated. + */ +static __always_inline void write_seqcount_latch_end(seqcount_latch_t *s) +{ + kcsan_nestable_atomic_end(); } =20 #define __SEQLOCK_UNLOCKED(lockname) \ --=20 2.47.0.163.g1226f6d8fa-goog From nobody Sun Nov 24 15:54:47 2024 Received: from mail-ej1-f73.google.com (mail-ej1-f73.google.com [209.85.218.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DE7271C07F7 for ; Mon, 4 Nov 2024 16:19:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730737177; cv=none; b=AbojaQ9jy3KgRliBhCDNMAE9Aqm8DzSwmiysGO3UibyNcM5Aq+b5uNKc8V8aj6o1KRHZ0LYHgVPkT7v9LZZfBQa0Xhw0iM43W06gkbVtFTyxB6gDXW4MUomIuuCL/JTdicVC6qaBCd0XwaL7G1R++J1pqbDNtMgO7ITnQ4KJPl8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730737177; c=relaxed/simple; bh=2Zf6Wh7WwMTJz5DSSx3o7r0wmY9f2/IMOwLW/s4U3UQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=asfLZxpawLz32cFh/eb/BQScI2+wW9bsVIJaZv30USC/BbNSyuopTMY1sl9tkwc4AODV7wGQEQOZYBxTMsaHhh5fJ2g8/oVJzeZKzCRni5BIj51zLIr5TfClYpBw/7ePfQsW8dZCHKKokpU4tyEGsPdTL6QuyN8+R1oySHTbNEY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--elver.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=OpJAcVhN; arc=none smtp.client-ip=209.85.218.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--elver.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="OpJAcVhN" Received: by mail-ej1-f73.google.com with SMTP id a640c23a62f3a-a9a1be34c61so150222166b.1 for ; Mon, 04 Nov 2024 08:19:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1730737174; x=1731341974; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=D01NtyTMk6i2oodCmqKoOULZz5EC2eZ4CIhcaLc7Lms=; b=OpJAcVhNuDY/xvw8cHPEeBBJktdGIcXB7dAGEs5Lw2+N10jK8DKZP4f1uCbxVHU+nO iiWhrtkeeWSkGtZXZ1Q6J+eHj1nuX0VdRLhrCDtSx3WFe0JF0fTmPiOrskuUfvBWV6iT LskDZRwINdN4zhirCkT1PkZZpFCDJB6lRF2I7G/vnK7+0txm3Zg+BY6P6+Qt37RmIBjx /vmydqSATjAPUxakqo4DMzNMpCaN9NFEC/iCpOobpESZrxQF3ANYm8Uz2/cgEBK88+Qj dtruB9c+FLhnPLqVj9paGhbFkdgzUiDd7GY0XoUv6kilgVqqVHWsG89dNzXerbYyZab5 aHEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730737174; x=1731341974; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=D01NtyTMk6i2oodCmqKoOULZz5EC2eZ4CIhcaLc7Lms=; b=Va3sYkCmu4mPku9cM+bpXOzgN50CMeDhDJWl88zA8CygeRjiX7UM1HT0Yr2m95Ipc5 W1mMcv5h/aXAR5LBZOkevQKIIxYzOb6yuiZxqjNodljA/KgqU7+J2TrDfXHXfItQWgPu 90kmcTyfmRYHRhAz4Y4xPjcHLTiPJzVP4mcdBbNBH9bQnV5Xks4YmHOaZjClSIWvSBMd 3R/OAo2dIOE8U4dFxz8RIaGIR9RKiirn8Es4HuwN3ljy4nwXTX+7gQlp1mM9Op/a5TyR LQe6Dup5CMaeyvao0RWPXMudKEB793Mz47bNmXiYUv8yFutrYdSybs+Q76oCdfc3OORM GltQ== X-Forwarded-Encrypted: i=1; AJvYcCUjBL/5PgIMUjEw6gVMTo50d+9jYe6HQwBIF1lGbdXmEzmF23kT8DV7gBsLsi7BCIS7mGtdNkjh2s9hd34=@vger.kernel.org X-Gm-Message-State: AOJu0Ywj5/QYzzop/AACFn5nJnIMHA5gu3ub5G+W4+xYY8xT52fnnZx7 GMP2rtgEuFfJyDJlFmR1icepza9OxHCjZSo+HHdccck90dfqdYHOmXluNU/BC+FNUEjqhRv9+g= = X-Google-Smtp-Source: AGHT+IE04dEG4Y5MUmrGycRsDe0dmIy+dy7/bFzFTwDynvSgS0Tjs/YvxNAZJ2cTxdozb20CkU05HfKJJA== X-Received: from elver.muc.corp.google.com ([2a00:79e0:9c:201:dc4d:3b27:d746:73ee]) (user=elver job=sendgmr) by 2002:a17:906:c0c1:b0:a99:fa8a:9783 with SMTP id a640c23a62f3a-a9e654bee59mr296866b.3.1730737174017; Mon, 04 Nov 2024 08:19:34 -0800 (PST) Date: Mon, 4 Nov 2024 16:43:08 +0100 In-Reply-To: <20241104161910.780003-1-elver@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241104161910.780003-1-elver@google.com> X-Mailer: git-send-email 2.47.0.163.g1226f6d8fa-goog Message-ID: <20241104161910.780003-5-elver@google.com> Subject: [PATCH v2 4/5] seqlock, treewide: Switch to non-raw seqcount_latch interface From: Marco Elver To: elver@google.com, Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng Cc: "Paul E. McKenney" , Thomas Gleixner , Mark Rutland , Dmitry Vyukov , kasan-dev@googlegroups.com, linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Switch all instrumentable users of the seqcount_latch interface over to the non-raw interface. Link: https://lore.kernel.org/all/20241030204815.GQ14555@noisy.programming.= kicks-ass.net/ Co-developed-by: Peter Zijlstra (Intel) Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Marco Elver --- v2: * New patch. --- arch/x86/kernel/tsc.c | 5 +++-- include/linux/rbtree_latch.h | 20 +++++++++++--------- kernel/printk/printk.c | 9 +++++---- kernel/time/sched_clock.c | 12 +++++++----- kernel/time/timekeeping.c | 12 +++++++----- 5 files changed, 33 insertions(+), 25 deletions(-) diff --git a/arch/x86/kernel/tsc.c b/arch/x86/kernel/tsc.c index dfe6847fd99e..67aeaba4ba9c 100644 --- a/arch/x86/kernel/tsc.c +++ b/arch/x86/kernel/tsc.c @@ -174,10 +174,11 @@ static void __set_cyc2ns_scale(unsigned long khz, int= cpu, unsigned long long ts =20 c2n =3D per_cpu_ptr(&cyc2ns, cpu); =20 - raw_write_seqcount_latch(&c2n->seq); + write_seqcount_latch_begin(&c2n->seq); c2n->data[0] =3D data; - raw_write_seqcount_latch(&c2n->seq); + write_seqcount_latch(&c2n->seq); c2n->data[1] =3D data; + write_seqcount_latch_end(&c2n->seq); } =20 static void set_cyc2ns_scale(unsigned long khz, int cpu, unsigned long lon= g tsc_now) diff --git a/include/linux/rbtree_latch.h b/include/linux/rbtree_latch.h index 6a0999c26c7c..2f630eb8307e 100644 --- a/include/linux/rbtree_latch.h +++ b/include/linux/rbtree_latch.h @@ -14,7 +14,7 @@ * * If we need to allow unconditional lookups (say as required for NMI cont= ext * usage) we need a more complex setup; this data structure provides this = by - * employing the latch technique -- see @raw_write_seqcount_latch -- to + * employing the latch technique -- see @write_seqcount_latch_begin -- to * implement a latched RB-tree which does allow for unconditional lookups = by * virtue of always having (at least) one stable copy of the tree. * @@ -132,7 +132,7 @@ __lt_find(void *key, struct latch_tree_root *ltr, int i= dx, * @ops: operators defining the node order * * It inserts @node into @root in an ordered fashion such that we can alwa= ys - * observe one complete tree. See the comment for raw_write_seqcount_latch= (). + * observe one complete tree. See the comment for write_seqcount_latch_beg= in(). * * The inserts use rcu_assign_pointer() to publish the element such that t= he * tree structure is stored before we can observe the new @node. @@ -145,10 +145,11 @@ latch_tree_insert(struct latch_tree_node *node, struct latch_tree_root *root, const struct latch_tree_ops *ops) { - raw_write_seqcount_latch(&root->seq); + write_seqcount_latch_begin(&root->seq); __lt_insert(node, root, 0, ops->less); - raw_write_seqcount_latch(&root->seq); + write_seqcount_latch(&root->seq); __lt_insert(node, root, 1, ops->less); + write_seqcount_latch_end(&root->seq); } =20 /** @@ -159,7 +160,7 @@ latch_tree_insert(struct latch_tree_node *node, * * Removes @node from the trees @root in an ordered fashion such that we c= an * always observe one complete tree. See the comment for - * raw_write_seqcount_latch(). + * write_seqcount_latch_begin(). * * It is assumed that @node will observe one RCU quiescent state before be= ing * reused of freed. @@ -172,10 +173,11 @@ latch_tree_erase(struct latch_tree_node *node, struct latch_tree_root *root, const struct latch_tree_ops *ops) { - raw_write_seqcount_latch(&root->seq); + write_seqcount_latch_begin(&root->seq); __lt_erase(node, root, 0); - raw_write_seqcount_latch(&root->seq); + write_seqcount_latch(&root->seq); __lt_erase(node, root, 1); + write_seqcount_latch_end(&root->seq); } =20 /** @@ -204,9 +206,9 @@ latch_tree_find(void *key, struct latch_tree_root *root, unsigned int seq; =20 do { - seq =3D raw_read_seqcount_latch(&root->seq); + seq =3D read_seqcount_latch(&root->seq); node =3D __lt_find(key, root, seq & 1, ops->comp); - } while (raw_read_seqcount_latch_retry(&root->seq, seq)); + } while (read_seqcount_latch_retry(&root->seq, seq)); =20 return node; } diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c index beb808f4c367..19911c8fa7b6 100644 --- a/kernel/printk/printk.c +++ b/kernel/printk/printk.c @@ -560,10 +560,11 @@ bool printk_percpu_data_ready(void) /* Must be called under syslog_lock. */ static void latched_seq_write(struct latched_seq *ls, u64 val) { - raw_write_seqcount_latch(&ls->latch); + write_seqcount_latch_begin(&ls->latch); ls->val[0] =3D val; - raw_write_seqcount_latch(&ls->latch); + write_seqcount_latch(&ls->latch); ls->val[1] =3D val; + write_seqcount_latch_end(&ls->latch); } =20 /* Can be called from any context. */ @@ -574,10 +575,10 @@ static u64 latched_seq_read_nolock(struct latched_seq= *ls) u64 val; =20 do { - seq =3D raw_read_seqcount_latch(&ls->latch); + seq =3D read_seqcount_latch(&ls->latch); idx =3D seq & 0x1; val =3D ls->val[idx]; - } while (raw_read_seqcount_latch_retry(&ls->latch, seq)); + } while (read_seqcount_latch_retry(&ls->latch, seq)); =20 return val; } diff --git a/kernel/time/sched_clock.c b/kernel/time/sched_clock.c index 29bdf309dae8..fcca4e72f1ef 100644 --- a/kernel/time/sched_clock.c +++ b/kernel/time/sched_clock.c @@ -71,13 +71,13 @@ static __always_inline u64 cyc_to_ns(u64 cyc, u32 mult,= u32 shift) =20 notrace struct clock_read_data *sched_clock_read_begin(unsigned int *seq) { - *seq =3D raw_read_seqcount_latch(&cd.seq); + *seq =3D read_seqcount_latch(&cd.seq); return cd.read_data + (*seq & 1); } =20 notrace int sched_clock_read_retry(unsigned int seq) { - return raw_read_seqcount_latch_retry(&cd.seq, seq); + return read_seqcount_latch_retry(&cd.seq, seq); } =20 static __always_inline unsigned long long __sched_clock(void) @@ -132,16 +132,18 @@ unsigned long long notrace sched_clock(void) static void update_clock_read_data(struct clock_read_data *rd) { /* steer readers towards the odd copy */ - raw_write_seqcount_latch(&cd.seq); + write_seqcount_latch_begin(&cd.seq); =20 /* now its safe for us to update the normal (even) copy */ cd.read_data[0] =3D *rd; =20 /* switch readers back to the even copy */ - raw_write_seqcount_latch(&cd.seq); + write_seqcount_latch(&cd.seq); =20 /* update the backup (odd) copy with the new data */ cd.read_data[1] =3D *rd; + + write_seqcount_latch_end(&cd.seq); } =20 /* @@ -279,7 +281,7 @@ void __init generic_sched_clock_init(void) */ static u64 notrace suspended_sched_clock_read(void) { - unsigned int seq =3D raw_read_seqcount_latch(&cd.seq); + unsigned int seq =3D read_seqcount_latch(&cd.seq); =20 return cd.read_data[seq & 1].epoch_cyc; } diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index 7e6f409bf311..18752983e834 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -411,7 +411,7 @@ static inline u64 timekeeping_get_ns(const struct tk_re= ad_base *tkr) * We want to use this from any context including NMI and tracing / * instrumenting the timekeeping code itself. * - * Employ the latch technique; see @raw_write_seqcount_latch. + * Employ the latch technique; see @write_seqcount_latch. * * So if a NMI hits the update of base[0] then it will use base[1] * which is still consistent. In the worst case this can result is a @@ -424,16 +424,18 @@ static void update_fast_timekeeper(const struct tk_re= ad_base *tkr, struct tk_read_base *base =3D tkf->base; =20 /* Force readers off to base[1] */ - raw_write_seqcount_latch(&tkf->seq); + write_seqcount_latch_begin(&tkf->seq); =20 /* Update base[0] */ memcpy(base, tkr, sizeof(*base)); =20 /* Force readers back to base[0] */ - raw_write_seqcount_latch(&tkf->seq); + write_seqcount_latch(&tkf->seq); =20 /* Update base[1] */ memcpy(base + 1, base, sizeof(*base)); + + write_seqcount_latch_end(&tkf->seq); } =20 static __always_inline u64 __ktime_get_fast_ns(struct tk_fast *tkf) @@ -443,11 +445,11 @@ static __always_inline u64 __ktime_get_fast_ns(struct= tk_fast *tkf) u64 now; =20 do { - seq =3D raw_read_seqcount_latch(&tkf->seq); + seq =3D read_seqcount_latch(&tkf->seq); tkr =3D tkf->base + (seq & 0x01); now =3D ktime_to_ns(tkr->base); now +=3D __timekeeping_get_ns(tkr); - } while (raw_read_seqcount_latch_retry(&tkf->seq, seq)); + } while (read_seqcount_latch_retry(&tkf->seq, seq)); =20 return now; } --=20 2.47.0.163.g1226f6d8fa-goog From nobody Sun Nov 24 15:54:47 2024 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8A76E1C9B68 for ; Mon, 4 Nov 2024 16:19:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730737180; cv=none; b=sP2Z00ui/tQt7AmNdco2DvIFKRmqKfNr7Ys3O307J7QSGxWYkF3Z44M8BWI23DlTo3SrtvSnsttdLxTe2FwAdvX0Qzr7JOlKnRXC6c77pbN7q1HO3ZVD9PwEiZ+1TalVjJTppQ/9yoCE+QGkkFEem2gX3MYBqjhsMoaOKYOIeVg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730737180; c=relaxed/simple; bh=5Xt4Fkz1MQW5w8UbS0tu5hc+USWbleuAt8BZVlSI+ks=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=K2W10/rnW2T14TbhM5yRB9gXmTlZ4kFr6R677GcO7uuY8KTo0hRUjYHusuKXRTpRrydNDD3SUiIyPQcXUCuY/i+M1EngF/iTg1JOtGeWZjto9JeOVEZD2a+OphIuoaAVp+ojJbxu6p7MiMYBEZo7SyJHsnH4FEi+qQLhdwhXDDM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--elver.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=c9klFaVi; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--elver.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="c9klFaVi" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-37d531a19a9so2366763f8f.1 for ; Mon, 04 Nov 2024 08:19:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1730737177; x=1731341977; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=OKj0s9xjK+saMHMRM1ZIudsUlk4SnRJVysUql1YTd+A=; b=c9klFaViDRvVoW2eYQ0LJD/cjOOBknFad0ba2xjUurXry0KimKtNitoVlo3d49iyfl 7XDxXrX7p3gAi1DoDSmmVuttxfqvld+7VklKqLuo90ZEqI16Drqo2FwWmPaTnLUcWt0M uBS5Su7uKMKXwt9xBUkPfE4AHEkW/feXHRfKkHWpIJSHvQcZMkU8Q5HGbkgNG2vhGaCf MtwZYQIdr0lkrb1DOazWjV/LmnhDJd3tfn5YuZR/ANGuejJ7S+tFrIrXQHLiDS53LQyC zXV7fNvHp6tdIbu2Utu0cYwJ5mbNdewc2TYUdjvxHHFZuHI+arZI1TA3++nCVkyP1Se7 BA9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730737177; x=1731341977; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OKj0s9xjK+saMHMRM1ZIudsUlk4SnRJVysUql1YTd+A=; b=lESbIVyJodZihd98YkT7ioBe7AawcYDSstLbxgrtFgk3qpzOgeVhfOM88NwK3UHtZ9 anI0qO+gjizdNxWqvttgIR9QU8JEn7JgvAcNI/UgQiyzQbOKlKSgPd7mlMo3iVT7pzPN OPR1iORHfOPvmGpkOp3H5QCWnBKnv+PJxTD72FtW6GHHg291axmsGBOVg0ehkOSKDDDk u8Qr+K+CQYcZusCVvF0e7Zv16ZHAihVGH4lEUA+xMjrUG3JRKFViMfF9iW0Uy1TFnsxT l6H6BJIbRJ5xPXdYcva8h6vlSDouJHr8ArN5ZPH5ioDs5yS+NiOZ68yxz+ThhsD1akZe cObg== X-Forwarded-Encrypted: i=1; AJvYcCWUbYOwioaBaW+xgWiwOs6xDEPMZBzgLclMYtnsYMGFBSk75nIjoEfrv2Dd8dTC4w36nRYnB3qpZ7UVFms=@vger.kernel.org X-Gm-Message-State: AOJu0YzowYo1inMPnptJSoRWWUNYPaxXj6L0OoH50hpG8g4XsarADvLW PYB9S2PwG6grXm/eSROXFm2y8miSWit5Pk44cv7/ANJMRPGgmjMKdb5zlwI1MFxLC3aXF1acxw= = X-Google-Smtp-Source: AGHT+IFYYQcvbot0gcoCP5b6QfEJHJaRpAbBNrY7qSGTa/FrddCLO5titnk6bblSV7Xeur6I4XZZDbNdVQ== X-Received: from elver.muc.corp.google.com ([2a00:79e0:9c:201:dc4d:3b27:d746:73ee]) (user=elver job=sendgmr) by 2002:a5d:44d0:0:b0:37e:d5a2:b104 with SMTP id ffacd0b85a97d-381be7cf9ecmr7016f8f.6.1730737176936; Mon, 04 Nov 2024 08:19:36 -0800 (PST) Date: Mon, 4 Nov 2024 16:43:09 +0100 In-Reply-To: <20241104161910.780003-1-elver@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241104161910.780003-1-elver@google.com> X-Mailer: git-send-email 2.47.0.163.g1226f6d8fa-goog Message-ID: <20241104161910.780003-6-elver@google.com> Subject: [PATCH v2 5/5] kcsan, seqlock: Fix incorrect assumption in read_seqbegin() From: Marco Elver To: elver@google.com, Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng Cc: "Paul E. McKenney" , Thomas Gleixner , Mark Rutland , Dmitry Vyukov , kasan-dev@googlegroups.com, linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" During testing of the preceding changes, I noticed that in some cases, current->kcsan_ctx.in_flat_atomic remained true until task exit. This is obviously wrong, because _all_ accesses for the given task will be treated as atomic, resulting in false negatives i.e. missed data races. Debugging led to fs/dcache.c, where we can see this usage of seqlock: struct dentry *d_lookup(const struct dentry *parent, const struct qstr *na= me) { struct dentry *dentry; unsigned seq; do { seq =3D read_seqbegin(&rename_lock); dentry =3D __d_lookup(parent, name); if (dentry) break; } while (read_seqretry(&rename_lock, seq)); [...] As can be seen, read_seqretry() is never called if dentry !=3D NULL; consequently, current->kcsan_ctx.in_flat_atomic will never be reset to false by read_seqretry(). Give up on the wrong assumption of "assume closing read_seqretry()", and rely on the already-present annotations in read_seqcount_begin/retry(). Fixes: 88ecd153be95 ("seqlock, kcsan: Add annotations for KCSAN") Signed-off-by: Marco Elver --- v2: * New patch. --- include/linux/seqlock.h | 12 +----------- 1 file changed, 1 insertion(+), 11 deletions(-) diff --git a/include/linux/seqlock.h b/include/linux/seqlock.h index 45eee0e5dca0..5298765d6ca4 100644 --- a/include/linux/seqlock.h +++ b/include/linux/seqlock.h @@ -810,11 +810,7 @@ static __always_inline void write_seqcount_latch_end(s= eqcount_latch_t *s) */ static inline unsigned read_seqbegin(const seqlock_t *sl) { - unsigned ret =3D read_seqcount_begin(&sl->seqcount); - - kcsan_atomic_next(0); /* non-raw usage, assume closing read_seqretry() */ - kcsan_flat_atomic_begin(); - return ret; + return read_seqcount_begin(&sl->seqcount); } =20 /** @@ -830,12 +826,6 @@ static inline unsigned read_seqbegin(const seqlock_t *= sl) */ static inline unsigned read_seqretry(const seqlock_t *sl, unsigned start) { - /* - * Assume not nested: read_seqretry() may be called multiple times when - * completing read critical section. - */ - kcsan_flat_atomic_end(); - return read_seqcount_retry(&sl->seqcount, start); } =20 --=20 2.47.0.163.g1226f6d8fa-goog