[PATCH RFC 03/33] locking: Introduce <linux/thread_safety.h>

Bart Van Assche posted 33 patches 10 months, 1 week ago
[PATCH RFC 03/33] locking: Introduce <linux/thread_safety.h>
Posted by Bart Van Assche 10 months, 1 week ago
Introduce a new kernel header with the Clang thread-safety attributes.
If:
- a struct that represents a synchronization object is annotated with the
  CAPABILITY() attribute,
- the operations on that synchronization object are annotated with the
  ACQUIRE() and RELEASE() attributes,
- if variables or members that should be guarded by a synchronization
  object are annotated with GUARDED_BY(),

then the Clang compiler verifies the following if -Wthread-safety is
enabled:
- Whether or not locking in a function implementation matches the
  thread-safety attributes in the function declaration. No annotation
  is necessary if a lock call is followed by an unlock call. For other
  patterns, annotation is required.
- Whether or not the requirements of the GUARDED_BY() annotations are
  met.

Some highlights from the Clang thread-safety attribute documentation:
- Alias analysis is not performed on thread-safety attribute arguments.
  Hence the expansion of some local variables in subsequent patches.
- Most thread-safety attributes affect the function interface.
  NO_THREAD_SAFETY_ANALYSIS only affects the function definition.
- If a private struct definition (in a .c file) includes a
  synchronization object, annotations of functions in .h files must
  not refer to the name of the private struct. A possible solution is
  to define a capability with DEFINE_CAPABILITY and to use the name of
  that capability in the thread-safety annotations.

A few notes from me:
- Thread-safety attributes are not included in function pointer types.
  In other words, when passing an annotated function as an argument to
  another function, the thread-safety attributes are discarded.
- Annotating conditional locking functions that return a pointer is not
  yet supported by Clang.

More information is available here:
https://clang.llvm.org/docs/ThreadSafetyAnalysis.html

In case anyone would be interested, the equivalent Qemu header file is
available here:
https://github.com/qemu/qemu/blob/master/include/qemu/clang-tsa.h

Cc: Ingo Molnar <mingo@redhat.com>
Cc: Will Deacon <will@kernel.org>
Cc: Boqun Feng <boqun.feng@gmail.com> (LOCKDEP & RUST)
Cc: Waiman Long <longman@redhat.com>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 include/linux/thread_safety.h | 141 ++++++++++++++++++++++++++++++++++
 1 file changed, 141 insertions(+)
 create mode 100644 include/linux/thread_safety.h

diff --git a/include/linux/thread_safety.h b/include/linux/thread_safety.h
new file mode 100644
index 000000000000..e23175223a18
--- /dev/null
+++ b/include/linux/thread_safety.h
@@ -0,0 +1,141 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _THREAD_SAFETY_H_
+#define _THREAD_SAFETY_H_
+
+/* See also https://clang.llvm.org/docs/ThreadSafetyAnalysis.html */
+
+/*
+ * Enable thread safety attributes only for clang. The attributes can be safely
+ * ignored when compiling with other compilers.
+ */
+#if defined(__clang__)
+#define THREAD_ANNOTATION_ATTRIBUTE_(...) __attribute__((__VA_ARGS__))
+#else
+#define THREAD_ANNOTATION_ATTRIBUTE_(...)
+#endif
+
+/*
+ * Macro for applying a capability as an attribute to a type definition.
+ * This macro can be used in struct definitions and also in typedefs.
+ * @x must be a string.
+ */
+#define CAPABILITY(x) THREAD_ANNOTATION_ATTRIBUTE_(capability(x))
+
+/*
+ * Macro for defining a capability name that is not tied to an existing type.
+ * @capability_name is declared as an external variable. Any attempt to
+ * read or modify that external variable will result in a linker error.
+ */
+#define DEFINE_CAPABILITY(capability_name)	      \
+	extern const struct {} CAPABILITY(#capability_name) capability_name
+
+/*
+ * Attribute for structure members that declares that the structure members are
+ * protected by the given capability.
+ */
+#define GUARDED_BY(x) THREAD_ANNOTATION_ATTRIBUTE_(guarded_by(x))
+
+/*
+ * Attribute for pointer structure members that declares that the contents
+ * of these pointers are protected by the given capability.
+ */
+#define PT_GUARDED_BY(x) THREAD_ANNOTATION_ATTRIBUTE_(pt_guarded_by(x))
+
+/*
+ * Attribute for instances of data structures that declares that the given
+ * capabilities must be acquired before the annotated data structure.
+ */
+#define ACQUIRED_BEFORE(...)					\
+	THREAD_ANNOTATION_ATTRIBUTE_(acquired_before(__VA_ARGS__))
+
+/*
+ * Attribute for instances of data structures that declares that the given
+ * capabilities must be acquired after the annotated data structure.
+ */
+#define ACQUIRED_AFTER(...)					\
+	THREAD_ANNOTATION_ATTRIBUTE_(acquired_after(__VA_ARGS__))
+
+/*
+ * Function attribute that declares that the caller must have exclusive access
+ * to the given capabilities.
+ */
+#define REQUIRES(...)							\
+	THREAD_ANNOTATION_ATTRIBUTE_(requires_capability(__VA_ARGS__))
+
+/*
+ * Function attribute that declares that the caller must have shared access
+ * to the given capabilities.
+ */
+#define REQUIRES_SHARED(...)						\
+	THREAD_ANNOTATION_ATTRIBUTE_(requires_shared_capability(__VA_ARGS__))
+
+/*
+ * Function attribute that declares that the function acquires the given
+ * capability.
+ */
+#define ACQUIRE(...)						\
+	THREAD_ANNOTATION_ATTRIBUTE_(acquire_capability(__VA_ARGS__))
+
+/*
+ * Function attribute that declares that the function acquires the given
+ * shared capability.
+ */
+#define ACQUIRE_SHARED(...)						\
+	THREAD_ANNOTATION_ATTRIBUTE_(acquire_shared_capability(__VA_ARGS__))
+
+/*
+ * Function attribute that declares that the function releases the given
+ * capability.
+ */
+#define RELEASE(...)						\
+	THREAD_ANNOTATION_ATTRIBUTE_(release_capability(__VA_ARGS__))
+
+/*
+ * Function attribute that declares that the function releases the given
+ * shared capability.
+ */
+#define RELEASE_SHARED(...)						\
+	THREAD_ANNOTATION_ATTRIBUTE_(release_shared_capability(__VA_ARGS__))
+
+/*
+ * Function attribute that declares that a function only acquires the given
+ * capability (2nd argument) for a given return value (first argument).
+ */
+#define TRY_ACQUIRE(...)						\
+	THREAD_ANNOTATION_ATTRIBUTE_(try_acquire_capability(__VA_ARGS__))
+
+/*
+ * Function attribute that declares that a function only acquires the given
+ * shared capability (2nd argument) for a given return value (first argument).
+ */
+#define TRY_ACQUIRE_SHARED(...)						\
+	THREAD_ANNOTATION_ATTRIBUTE_(try_acquire_shared_capability(__VA_ARGS__))
+
+/*
+ * Function attribute that declares that the caller must not hold the given
+ * capabilities.
+ */
+#define EXCLUDES(...) THREAD_ANNOTATION_ATTRIBUTE_(locks_excluded(__VA_ARGS__))
+
+/*
+ * Tell the compiler that the given capability is held.
+ */
+#define ASSERT_CAPABILITY(x) THREAD_ANNOTATION_ATTRIBUTE_(assert_capability(x))
+
+/*
+ * Tell the compiler that the given shared capability is held.
+ */
+#define ASSERT_SHARED_CAPABILITY(x)				\
+	THREAD_ANNOTATION_ATTRIBUTE_(assert_shared_capability(x))
+
+/*
+ * Function attribute that declares that a function returns a pointer to a
+ * capability.
+ */
+#define RETURN_CAPABILITY(x) THREAD_ANNOTATION_ATTRIBUTE_(lock_returned(x))
+
+/* Function attribute that disables thread-safety analysis. */
+#define NO_THREAD_SAFETY_ANALYSIS				\
+	THREAD_ANNOTATION_ATTRIBUTE_(no_thread_safety_analysis)
+
+#endif /* _THREAD_SAFETY_H_ */
Re: [PATCH RFC 03/33] locking: Introduce <linux/thread_safety.h>
Posted by Christoph Hellwig 10 months, 1 week ago
> - a struct that represents a synchronization object is annotated with the
>   CAPABILITY() attribute,
> - the operations on that synchronization object are annotated with the
>   ACQUIRE() and RELEASE() attributes,
> - if variables or members that should be guarded by a synchronization
>   object are annotated with GUARDED_BY(),

Those are all nasty shouting names, without and good prefixing.

But more importantly ACQUIRE() and RELEASE() seems to duplicate the
existing __acquires/__releases annotations from sparse.  We really need
to find away to unify them instead of duplicating the annotations.
Re: [PATCH RFC 03/33] locking: Introduce <linux/thread_safety.h>
Posted by Bart Van Assche 10 months, 1 week ago
On 2/6/25 7:53 PM, Christoph Hellwig wrote:
>> - a struct that represents a synchronization object is annotated with the
>>    CAPABILITY() attribute,
>> - the operations on that synchronization object are annotated with the
>>    ACQUIRE() and RELEASE() attributes,
>> - if variables or members that should be guarded by a synchronization
>>    object are annotated with GUARDED_BY(),
> 
> Those are all nasty shouting names, without and good prefixing.
> 
> But more importantly ACQUIRE() and RELEASE() seems to duplicate the
> existing __acquires/__releases annotations from sparse.  We really need
> to find away to unify them instead of duplicating the annotations.

I think that we are better off to drop support for the sparse locking
annotations. Linus added the macro __cond_acquires() two years ago in
the Linux kernel but support for that macro has never been implemented
in sparse. In the description of commit 4a557a5d1a61 ("sparse: introduce
conditional lock acquire function attribute") I found the following URL:
https://lore.kernel.org/all/CAHk-=wjZfO9hGqJ2_hGQG3U_XzSh9_XaXze=HgPdvJbgrvASfA@mail.gmail.com/

That URL points to an e-mail from Linus Torvalds with a patch for sparse
that implements support for __cond_acquires(). It seems to me that the
sparse patch has never been applied to the sparse code base (the git URL
for sparse is available at https://sparse.docs.kernel.org/en/latest/).

So while __cond_acquires() can be translated into a Clang annotation,
__cond_acquires() is not supported by sparse. There are other
challenges:
* The argument of __acquire() and __release() can be a "capability" or
   the address of a synchronization object. A few examples where the
   argument represents a capability:

   __acquire(RCU);
   __acquire(RCU_BH);
   __acquire(RCU_SCHED);
   __acquire(bitlock);

   An example where the argument represents the address of a
   synchronization object:

   static inline void do_raw_spin_lock(raw_spinlock_t *lock)
         __acquires(lock)
   {
	__acquire(lock);
	arch_spin_lock(&lock->raw_lock);
	mmiowb_spin_lock();
   }

   If __acquire() and __release() would have to be supported for both
   sparse and Clang, that would probably involve modifying all instances
   of these two macros across the entire kernel tree and making it
   explicit whether the argument is a capability or the address of a
   synchronization object.

* Clang verifies the thread-safety attribute arguments at compile time
   while sparse does not. An example from the scheduler:

extern
struct rq *__task_rq_lock(struct task_struct *p, struct rq_flags *rf)
	__acquires(rq->lock);

   An example from the workqueue implementation:

static void maybe_create_worker(struct worker_pool *pool)
__releases(&pool->lock)
__acquires(&pool->lock)
{
[ ... ]
}

   If __acquires() and __releases() are converted into Clang function
   attributes then compilation fails because Clang verifies the
   expressions passed to function attributes before the function body is
   compiled.

The team that is maintaining sparse is small (a single person?) and the
rate of change of sparse is low (the latest commit was published more
than a year ago). If we support both sparse locking annotations and the
Clang thread-safety annotations then we will end up fixing bugs in both
compilers. I propose to focus the maintenance effort on the project with
the highest velocity (Clang) and to drop support for the sparse locking
annotations.

Thanks,

Bart.
Re: [PATCH RFC 03/33] locking: Introduce <linux/thread_safety.h>
Posted by Marco Elver 10 months, 1 week ago
On Fri, 7 Feb 2025 at 23:34, Bart Van Assche <bvanassche@acm.org> wrote:
[...]
> > Those are all nasty shouting names, without and good prefixing.
> >
> > But more importantly ACQUIRE() and RELEASE() seems to duplicate the
> > existing __acquires/__releases annotations from sparse.  We really need
> > to find away to unify them instead of duplicating the annotations.
>
> I think that we are better off to drop support for the sparse locking
> annotations. Linus added the macro __cond_acquires() two years ago in

+1 to dropping Sparse support -- although we can retain the same keywords.

[...]
> * The argument of __acquire() and __release() can be a "capability" or
>    the address of a synchronization object. A few examples where the
>    argument represents a capability:

Not a problem, and can be solved by introducing "token" instances...

>    __acquire(RCU);
>    __acquire(RCU_BH);
>    __acquire(RCU_SCHED);

For RCU: https://lore.kernel.org/lkml/20250206181711.1902989-16-elver@google.com/

>    __acquire(bitlock);

For bitlock: https://lore.kernel.org/lkml/20250206181711.1902989-15-elver@google.com/

>    An example where the argument represents the address of a
>    synchronization object:
>
>    static inline void do_raw_spin_lock(raw_spinlock_t *lock)
>          __acquires(lock)
>    {
>         __acquire(lock);
>         arch_spin_lock(&lock->raw_lock);
>         mmiowb_spin_lock();
>    }
>
>    If __acquire() and __release() would have to be supported for both
>    sparse and Clang, that would probably involve modifying all instances
>    of these two macros across the entire kernel tree and making it
>    explicit whether the argument is a capability or the address of a
>    synchronization object.

This is not a problem as I have demonstrated with my approach. Using
token instances is also the way we'd be able to support things like
preempt-disable-enable, irq-disable-enable, etc.

In general I'm in favor of dropping Sparse support, but the majority
of existing annotations can be reused, simplifying our work.
Re: [PATCH RFC 03/33] locking: Introduce <linux/thread_safety.h>
Posted by Bart Van Assche 10 months, 1 week ago
On 2/7/25 3:19 PM, Marco Elver wrote:
> This is not a problem as I have demonstrated with my approach.

That's not what my email is about. My email is about whether or not to
continue supporting the sparse locking context annotations.

Bart.
Re: [PATCH RFC 03/33] locking: Introduce <linux/thread_safety.h>
Posted by Marco Elver 10 months, 1 week ago
On Fri, 7 Feb 2025 at 04:53, Christoph Hellwig <hch@lst.de> wrote:
>
> > - a struct that represents a synchronization object is annotated with the
> >   CAPABILITY() attribute,
> > - the operations on that synchronization object are annotated with the
> >   ACQUIRE() and RELEASE() attributes,
> > - if variables or members that should be guarded by a synchronization
> >   object are annotated with GUARDED_BY(),
>
> Those are all nasty shouting names, without and good prefixing.
>
> But more importantly ACQUIRE() and RELEASE() seems to duplicate the
> existing __acquires/__releases annotations from sparse.  We really need
> to find away to unify them instead of duplicating the annotations.

This is exactly what my approach tries to do:
https://lore.kernel.org/all/20250206181711.1902989-1-elver@google.com/T/#u
(Make it work with the existing keywords and extend them.)