From nobody Sun May 19 06:04:49 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1586268562; cv=none; d=zohomail.com; s=zohoarc; b=K6N97KAwHSluGtgXhuCsL6HEjUU2Nvj+/Bt94XAPO4IgTsTeLdtHJFUcnafiz9rYspD5r3zeSiMUNkXip1De/BCYFysNLER9/ZyxsB0gPael0p+6oco+eD3iYul/YPUxrxPt1fvKucPxCnkv+7siVjlHTlfp6yZ4nCQ0/qauTvw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1586268562; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=4DQ3hIkcyi14yXey+VNW+FKPZly8Pt2h1rVqJaCYpxY=; b=PR0o7Yy7x+OJccIjsasxmukKSfD5pljfQeXkNZsiOyqOvA7J/gjciGSTJFc7mE2lu1gI5WacT3k9JZXCW0c6gM1yTkdXbUs1KxmvgWQKpCAhPqo8HPFaQIRKpwIpP36G56iTbEz1GC0flA9A1ReaYPoKSLhZ3r+0vh8XhKaYyLw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1586268562123424.7433222495989; Tue, 7 Apr 2020 07:09:22 -0700 (PDT) Received: from localhost ([::1]:47876 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jLoum-0002t6-MA for importer@patchew.org; Tue, 07 Apr 2020 10:09:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51232) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jLotZ-0000lt-50 for qemu-devel@nongnu.org; Tue, 07 Apr 2020 10:08:09 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jLotU-00016p-Q8 for qemu-devel@nongnu.org; Tue, 07 Apr 2020 10:08:05 -0400 Received: from us-smtp-1.mimecast.com ([205.139.110.61]:52118 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1jLotU-00016a-JE for qemu-devel@nongnu.org; Tue, 07 Apr 2020 10:08:00 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-248-TV9Ti-V1PpKYRcHOXDS4kg-1; Tue, 07 Apr 2020 10:07:52 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 5CE9D107ACC4; Tue, 7 Apr 2020 14:07:51 +0000 (UTC) Received: from virtlab701.virt.lab.eng.bos.redhat.com (virtlab701.virt.lab.eng.bos.redhat.com [10.19.152.228]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9E6DA5C553; Tue, 7 Apr 2020 14:07:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1586268480; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4DQ3hIkcyi14yXey+VNW+FKPZly8Pt2h1rVqJaCYpxY=; b=MtBoW1oBbyMuoqDsr9jYdEom63Uv2PXWGOq2mxa7v0albBtQl8R71QcybWT9+NJ3jP4dZ2 M9lS4bXXYHe3QUgvZ6mXY95BuHZpmMtXHV7npkDM9GQs/JCyJuxtaq/B+UtCJ4075F/Y3z ELFADKUZfdcXWvxCE+BDiSlDtdi54cc= X-MC-Unique: TV9Ti-V1PpKYRcHOXDS4kg-1 From: Paolo Bonzini To: qemu-devel@nongnu.org Subject: [PATCH 1/5] atomics: convert to reStructuredText Date: Tue, 7 Apr 2020 10:07:42 -0400 Message-Id: <20200407140746.8041-2-pbonzini@redhat.com> In-Reply-To: <20200407140746.8041-1-pbonzini@redhat.com> References: <20200407140746.8041-1-pbonzini@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 205.139.110.61 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fangying1@huawei.com, stefanha@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" No attempts to fix or update the text; these are left for the next patch in the series. Signed-off-by: Paolo Bonzini Reviewed-by: Alex Benn=C3=A9e --- docs/devel/atomics.rst | 446 +++++++++++++++++++++++++++++++++++++++++ docs/devel/atomics.txt | 403 ------------------------------------- docs/devel/index.rst | 1 + 3 files changed, 447 insertions(+), 403 deletions(-) create mode 100644 docs/devel/atomics.rst delete mode 100644 docs/devel/atomics.txt diff --git a/docs/devel/atomics.rst b/docs/devel/atomics.rst new file mode 100644 index 0000000000..83ed3d6981 --- /dev/null +++ b/docs/devel/atomics.rst @@ -0,0 +1,446 @@ +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D +Atomic operations in QEMU +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D + +CPUs perform independent memory operations effectively in random order. +but this can be a problem for CPU-CPU interaction (including interactions +between QEMU and the guest). Multi-threaded programs use various tools +to instruct the compiler and the CPU to restrict the order to something +that is consistent with the expectations of the programmer. + +The most basic tool is locking. Mutexes, condition variables and +semaphores are used in QEMU, and should be the default approach to +synchronization. Anything else is considerably harder, but it's +also justified more often than one would like. The two tools that +are provided by ``qemu/atomic.h`` are memory barriers and atomic operation= s. + +Macros defined by ``qemu/atomic.h`` fall in three camps: + +- compiler barriers: ``barrier()``; + +- weak atomic access and manual memory barriers: ``atomic_read()``, + ``atomic_set()``, ``smp_rmb()``, ``smp_wmb()``, ``smp_mb()``, ``smp_mb_a= cquire()``, + ``smp_mb_release()``, ``smp_read_barrier_depends()``; + +- sequentially consistent atomic access: everything else. + + +Compiler memory barrier +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +``barrier()`` prevents the compiler from moving the memory accesses either +side of it to the other side. The compiler barrier has no direct effect +on the CPU, which may then reorder things however it wishes. + +``barrier()`` is mostly used within ``qemu/atomic.h`` itself. On some +architectures, CPU guarantees are strong enough that blocking compiler +optimizations already ensures the correct order of execution. In this +case, ``qemu/atomic.h`` will reduce stronger memory barriers to simple +compiler barriers. + +Still, ``barrier()`` can be useful when writing code that can be interrupt= ed +by signal handlers. + + +Sequentially consistent atomic access +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Most of the operations in the ``qemu/atomic.h`` header ensure *sequential +consistency*, where "the result of any execution is the same as if the +operations of all the processors were executed in some sequential order, +and the operations of each individual processor appear in this sequence +in the order specified by its program". + +``qemu/atomic.h`` provides the following set of atomic read-modify-write +operations:: + + void atomic_inc(ptr) + void atomic_dec(ptr) + void atomic_add(ptr, val) + void atomic_sub(ptr, val) + void atomic_and(ptr, val) + void atomic_or(ptr, val) + + typeof(*ptr) atomic_fetch_inc(ptr) + typeof(*ptr) atomic_fetch_dec(ptr) + typeof(*ptr) atomic_fetch_add(ptr, val) + typeof(*ptr) atomic_fetch_sub(ptr, val) + typeof(*ptr) atomic_fetch_and(ptr, val) + typeof(*ptr) atomic_fetch_or(ptr, val) + typeof(*ptr) atomic_fetch_xor(ptr, val) + typeof(*ptr) atomic_fetch_inc_nonzero(ptr) + typeof(*ptr) atomic_xchg(ptr, val) + typeof(*ptr) atomic_cmpxchg(ptr, old, new) + +all of which return the old value of ``*ptr``. These operations are +polymorphic; they operate on any type that is as wide as a pointer. + +Similar operations return the new value of ``*ptr``:: + + typeof(*ptr) atomic_inc_fetch(ptr) + typeof(*ptr) atomic_dec_fetch(ptr) + typeof(*ptr) atomic_add_fetch(ptr, val) + typeof(*ptr) atomic_sub_fetch(ptr, val) + typeof(*ptr) atomic_and_fetch(ptr, val) + typeof(*ptr) atomic_or_fetch(ptr, val) + typeof(*ptr) atomic_xor_fetch(ptr, val) + +Sequentially consistent loads and stores can be done using:: + + atomic_fetch_add(ptr, 0) for loads + atomic_xchg(ptr, val) for stores + +However, they are quite expensive on some platforms, notably POWER and +Arm. Therefore, qemu/atomic.h provides two primitives with slightly +weaker constraints:: + + typeof(*ptr) atomic_mb_read(ptr) + void atomic_mb_set(ptr, val) + +The semantics of these primitives map to Java volatile variables, +and are strongly related to memory barriers as used in the Linux +kernel (see below). + +As long as you use atomic_mb_read and atomic_mb_set, accesses cannot +be reordered with each other, and it is also not possible to reorder +"normal" accesses around them. + +However, and this is the important difference between +atomic_mb_read/atomic_mb_set and sequential consistency, it is important +for both threads to access the same volatile variable. It is not the +case that everything visible to thread A when it writes volatile field f +becomes visible to thread B after it reads volatile field g. The store +and load have to "match" (i.e., be performed on the same volatile +field) to achieve the right semantics. + + +These operations operate on any type that is as wide as an int or smaller. + + +Weak atomic access and manual memory barriers +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Compared to sequentially consistent atomic access, programming with +weaker consistency models can be considerably more complicated. +In general, if the algorithm you are writing includes both writes +and reads on the same side, it is generally simpler to use sequentially +consistent primitives. + +When using this model, variables are accessed with: + +- ``atomic_read()`` and ``atomic_set()``; these prevent the compiler from + optimizing accesses out of existence and creating unsolicited + accesses, but do not otherwise impose any ordering on loads and + stores: both the compiler and the processor are free to reorder + them. + +- ``atomic_load_acquire()``, which guarantees the LOAD to appear to + happen, with respect to the other components of the system, + before all the LOAD or STORE operations specified afterwards. + Operations coming before ``atomic_load_acquire()`` can still be + reordered after it. + +- ``atomic_store_release()``, which guarantees the STORE to appear to + happen, with respect to the other components of the system, + after all the LOAD or STORE operations specified afterwards. + Operations coming after ``atomic_store_release()`` can still be + reordered after it. + +Restrictions to the ordering of accesses can also be specified +using the memory barrier macros: ``smp_rmb()``, ``smp_wmb()``, ``smp_mb()`= `, +``smp_mb_acquire()``, ``smp_mb_release()``, ``smp_read_barrier_depends()``. + +Memory barriers control the order of references to shared memory. +They come in six kinds: + +- ``smp_rmb()`` guarantees that all the LOAD operations specified before + the barrier will appear to happen before all the LOAD operations + specified after the barrier with respect to the other components of + the system. + + In other words, ``smp_rmb()`` puts a partial ordering on loads, but is n= ot + required to have any effect on stores. + +- ``smp_wmb()`` guarantees that all the STORE operations specified before + the barrier will appear to happen before all the STORE operations + specified after the barrier with respect to the other components of + the system. + + In other words, ``smp_wmb()`` puts a partial ordering on stores, but is = not + required to have any effect on loads. + +- ``smp_mb_acquire()`` guarantees that all the LOAD operations specified b= efore + the barrier will appear to happen before all the LOAD or STORE operations + specified after the barrier with respect to the other components of + the system. + +- ``smp_mb_release()`` guarantees that all the STORE operations specified = *after* + the barrier will appear to happen after all the LOAD or STORE operations + specified *before* the barrier with respect to the other components of + the system. + +- ``smp_mb()`` guarantees that all the LOAD and STORE operations specified + before the barrier will appear to happen before all the LOAD and + STORE operations specified after the barrier with respect to the other + components of the system. + + ``smp_mb()`` puts a partial ordering on both loads and stores. It is + stronger than both a read and a write memory barrier; it implies both + ``smp_mb_acquire()`` and ``smp_mb_release()``, but it also prevents STOR= Es + coming before the barrier from overtaking LOADs coming after the + barrier and vice versa. + +- ``smp_read_barrier_depends()`` is a weaker kind of read barrier. On + most processors, whenever two loads are performed such that the + second depends on the result of the first (e.g., the first load + retrieves the address to which the second load will be directed), + the processor will guarantee that the first LOAD will appear to happen + before the second with respect to the other components of the system. + However, this is not always true---for example, it was not true on + Alpha processors. Whenever this kind of access happens to shared + memory (that is not protected by a lock), a read barrier is needed, + and ``smp_read_barrier_depends()`` can be used instead of ``smp_rmb()``. + + Note that the first load really has to have a _data_ dependency and not + a control dependency. If the address for the second load is dependent + on the first load, but the dependency is through a conditional rather + than actually loading the address itself, then it's a _control_ + dependency and a full read barrier or better is required. + + +This is the set of barriers that is required *between* two ``atomic_read()= `` +and ``atomic_set()`` operations to achieve sequential consistency: + + +----------------+-----------------------------------------------------= --+ + | | 2nd operation = | + | +------------------+-----------------+----------------= --+ + | 1st operation | (after last) | atomic_read | atomic_set = | + +----------------+------------------+-----------------+----------------= --+ + | (before first) | .. | none | smp_mb_release(= ) | + +----------------+------------------+-----------------+----------------= --+ + | atomic_read | smp_mb_acquire() | smp_rmb() [1]_ | [2]_ = | + +----------------+------------------+-----------------+----------------= --+ + | atomic_set | none | smp_mb() [3]_ | smp_wmb() = | + +----------------+------------------+-----------------+----------------= --+ + + .. [1] Or smp_read_barrier_depends(). + + .. [2] This requires a load-store barrier. This is achieved by + either smp_mb_acquire() or smp_mb_release(). + + .. [3] This requires a store-load barrier. On most machines, the only + way to achieve this is a full barrier. + + +You can see that the two possible definitions of ``atomic_mb_read()`` +and ``atomic_mb_set()`` are the following: + + 1) | atomic_mb_read(p) =3D atomic_read(p); smp_mb_acquire() + | atomic_mb_set(p, v) =3D smp_mb_release(); atomic_set(p, v); smp_mb() + + 2) | atomic_mb_read(p) =3D smp_mb() atomic_read(p); smp_mb_acquire() + | atomic_mb_set(p, v) =3D smp_mb_release(); atomic_set(p, v); + +Usually the former is used, because ``smp_mb()`` is expensive and a program +normally has more reads than writes. Therefore it makes more sense to +make ``atomic_mb_set()`` the more expensive operation. + +There are two common cases in which atomic_mb_read and atomic_mb_set +generate too many memory barriers, and thus it can be useful to manually +place barriers, or use atomic_load_acquire/atomic_store_release instead: + +- when a data structure has one thread that is always a writer + and one thread that is always a reader, manual placement of + memory barriers makes the write side faster. Furthermore, + correctness is easy to check for in this case using the "pairing" + trick that is explained below: + + +---------------------------------------------------------------------= -+ + | thread 1 = | + +-----------------------------------+---------------------------------= -+ + | before | after = | + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+ + | :: | :: = | + | | = | + | (other writes) | = | + | atomic_mb_set(&a, x) | atomic_store_release(&a, x) = | + | atomic_mb_set(&b, y) | atomic_store_release(&b, y) = | + +-----------------------------------+---------------------------------= -+ + + +---------------------------------------------------------------------= -+ + | thread 2 = | + +-----------------------------------+---------------------------------= -+ + | before | after = | + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+ + | :: | :: = | + | | = | + | y =3D atomic_mb_read(&b) | y =3D atomic_load_acquire(&b= ) | + | x =3D atomic_mb_read(&a) | x =3D atomic_load_acquire(&a= ) | + | (other reads) | = | + +-----------------------------------+---------------------------------= -+ + + Note that the barrier between the stores in thread 1, and between + the loads in thread 2, has been optimized here to a write or a + read memory barrier respectively. On some architectures, notably + ARMv7, smp_mb_acquire and smp_mb_release are just as expensive as + smp_mb, but smp_rmb and/or smp_wmb are more efficient. + +- sometimes, a thread is accessing many variables that are otherwise + unrelated to each other (for example because, apart from the current + thread, exactly one other thread will read or write each of these + variables). In this case, it is possible to "hoist" the implicit + barriers provided by ``atomic_mb_read()`` and ``atomic_mb_set()`` outside + a loop. For example, the above definition ``atomic_mb_read()`` gives + the following transformation: + + +-----------------------------------+---------------------------------= -+ + | before | after = | + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+ + | :: | :: = | + | | = | + | n =3D 0; | n =3D 0; = | + | for (i =3D 0; i < 10; i++) | for (i =3D 0; i < 10; i++) = | + | n +=3D atomic_mb_read(&a[i]); | n +=3D atomic_read(&a[i]);= | + | | smp_mb_acquire(); = | + +-----------------------------------+---------------------------------= -+ + + Similarly, atomic_mb_set() can be transformed as follows: + + +-----------------------------------+---------------------------------= -+ + | before | after = | + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+ + | :: | :: = | + | | = | + | | smp_mb_release(); = | + | for (i =3D 0; i < 10; i++) | for (i =3D 0; i < 10; i++) = | + | atomic_mb_set(&a[i], false); | atomic_set(&a[i], false); = | + | | smp_mb(); = | + +-----------------------------------+---------------------------------= -+ + + + The other thread can still use ``atomic_mb_read()``/``atomic_mb_set()``. + +The two tricks can be combined. In this case, splitting a loop in +two lets you hoist the barriers out of the loops _and_ eliminate the +expensive ``smp_mb()``: + + +-----------------------------------+---------------------------------= -+ + | before | after = | + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+ + | :: | :: = | + | | = | + | | smp_mb_release(); = | + | for (i =3D 0; i < 10; i++) { | for (i =3D 0; i < 10; i++)= | + | atomic_mb_set(&a[i], false); | atomic_set(&a[i], false); = | + | atomic_mb_set(&b[i], false); | smb_wmb(); = | + | } | for (i =3D 0; i < 10; i++) = | + | | atomic_set(&a[i], false); = | + | | smp_mb(); = | + +-----------------------------------+---------------------------------= -+ + + +Memory barrier pairing +---------------------- + +A useful rule of thumb is that memory barriers should always, or almost +always, be paired with another barrier. In the case of QEMU, however, +note that the other barrier may actually be in a driver that runs in +the guest! + +For the purposes of pairing, ``smp_read_barrier_depends()`` and ``smp_rmb(= )`` +both count as read barriers. A read barrier shall pair with a write +barrier or a full barrier; a write barrier shall pair with a read +barrier or a full barrier. A full barrier can pair with anything. +For example: + + +--------------------+------------------------------+ + | thread 1 | thread 2 | + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D+ + | :: | :: | + | | | + | a =3D 1; | | + | smp_wmb(); | | + | b =3D 2; | x =3D b; | + | | smp_rmb(); | + | | y =3D a; | + +--------------------+------------------------------+ + +Note that the "writing" thread is accessing the variables in the +opposite order as the "reading" thread. This is expected: stores +before the write barrier will normally match the loads after the +read barrier, and vice versa. The same is true for more than 2 +access and for data dependency barriers: + + +----------------------+------------------------------+ + | thread 1 | thread 2 | + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D+ + | :: | :: | + | | | + | b[2] =3D 1; | | + | smp_wmb(); | | + | x->i =3D 2; | | + | smp_wmb(); | | + | a =3D x; | x =3D a; | + | | smp_read_barrier_depends(); | + | | y =3D x->i; | + | | smp_read_barrier_depends(); | + | | z =3D b[y]; | + +----------------------+------------------------------+ + +``smp_wmb()`` also pairs with ``atomic_mb_read()`` and ``smp_mb_acquire()`= `. +and ``smp_rmb()`` also pairs with ``atomic_mb_set()`` and ``smp_mb_release= ()``. + + +Comparison with Linux kernel memory barriers +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Here is a list of differences between Linux kernel atomic operations +and memory barriers, and the equivalents in QEMU: + +- atomic operations in Linux are always on a 32-bit int type and + use a boxed ``atomic_t`` type; atomic operations in QEMU are polymorphic + and use normal C types. + +- Originally, ``atomic_read`` and ``atomic_set`` in Linux gave no guarantee + at all. Linux 4.1 updated them to implement volatile + semantics via ``ACCESS_ONCE`` (or the more recent ``READ``/``WRITE_ONCE`= `). + + QEMU's ``atomic_read`` and ``atomic_set`` implement C11 atomic relaxed + semantics if the compiler supports it, and volatile semantics otherwise. + Both semantics prevent the compiler from doing certain transformations; + the difference is that atomic accesses are guaranteed to be atomic, + while volatile accesses aren't. Thus, in the volatile case we just cross + our fingers hoping that the compiler will generate atomic accesses, + since we assume the variables passed are machine-word sized and + properly aligned. + + No barriers are implied by ``atomic_read`` and ``atomic_set`` in either = Linux + or QEMU. + +- atomic read-modify-write operations in Linux are of three kinds: + + =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + ``atomic_OP`` returns void + ``atomic_OP_return`` returns new value of the variable + ``atomic_fetch_OP`` returns the old value of the variable + ``atomic_cmpxchg`` returns the old value of the variable + =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + + In QEMU, the second kind does not exist. Currently Linux has + atomic_fetch_or only. QEMU provides and, or, inc, dec, add, sub. + +- different atomic read-modify-write operations in Linux imply + a different set of memory barriers; in QEMU, all of them enforce + sequential consistency, which means they imply full memory barriers + before and after the operation. + +- Linux does not have an equivalent of ``atomic_mb_set()``. In particular, + note that ``smp_store_mb()`` is a little weaker than ``atomic_mb_set()``. + ``atomic_mb_read()`` compiles to the same instructions as Linux's + ``smp_load_acquire()``, but this should be treated as an implementation + detail. + +Sources +=3D=3D=3D=3D=3D=3D=3D + +- ``Documentation/memory-barriers.txt`` from the Linux kernel diff --git a/docs/devel/atomics.txt b/docs/devel/atomics.txt deleted file mode 100644 index 67bdf82628..0000000000 --- a/docs/devel/atomics.txt +++ /dev/null @@ -1,403 +0,0 @@ -CPUs perform independent memory operations effectively in random order. -but this can be a problem for CPU-CPU interaction (including interactions -between QEMU and the guest). Multi-threaded programs use various tools -to instruct the compiler and the CPU to restrict the order to something -that is consistent with the expectations of the programmer. - -The most basic tool is locking. Mutexes, condition variables and -semaphores are used in QEMU, and should be the default approach to -synchronization. Anything else is considerably harder, but it's -also justified more often than one would like. The two tools that -are provided by qemu/atomic.h are memory barriers and atomic operations. - -Macros defined by qemu/atomic.h fall in three camps: - -- compiler barriers: barrier(); - -- weak atomic access and manual memory barriers: atomic_read(), - atomic_set(), smp_rmb(), smp_wmb(), smp_mb(), smp_mb_acquire(), - smp_mb_release(), smp_read_barrier_depends(); - -- sequentially consistent atomic access: everything else. - - -COMPILER MEMORY BARRIER -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D - -barrier() prevents the compiler from moving the memory accesses either -side of it to the other side. The compiler barrier has no direct effect -on the CPU, which may then reorder things however it wishes. - -barrier() is mostly used within qemu/atomic.h itself. On some -architectures, CPU guarantees are strong enough that blocking compiler -optimizations already ensures the correct order of execution. In this -case, qemu/atomic.h will reduce stronger memory barriers to simple -compiler barriers. - -Still, barrier() can be useful when writing code that can be interrupted -by signal handlers. - - -SEQUENTIALLY CONSISTENT ATOMIC ACCESS -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D - -Most of the operations in the qemu/atomic.h header ensure *sequential -consistency*, where "the result of any execution is the same as if the -operations of all the processors were executed in some sequential order, -and the operations of each individual processor appear in this sequence -in the order specified by its program". - -qemu/atomic.h provides the following set of atomic read-modify-write -operations: - - void atomic_inc(ptr) - void atomic_dec(ptr) - void atomic_add(ptr, val) - void atomic_sub(ptr, val) - void atomic_and(ptr, val) - void atomic_or(ptr, val) - - typeof(*ptr) atomic_fetch_inc(ptr) - typeof(*ptr) atomic_fetch_dec(ptr) - typeof(*ptr) atomic_fetch_add(ptr, val) - typeof(*ptr) atomic_fetch_sub(ptr, val) - typeof(*ptr) atomic_fetch_and(ptr, val) - typeof(*ptr) atomic_fetch_or(ptr, val) - typeof(*ptr) atomic_fetch_xor(ptr, val) - typeof(*ptr) atomic_fetch_inc_nonzero(ptr) - typeof(*ptr) atomic_xchg(ptr, val) - typeof(*ptr) atomic_cmpxchg(ptr, old, new) - -all of which return the old value of *ptr. These operations are -polymorphic; they operate on any type that is as wide as a pointer. - -Similar operations return the new value of *ptr: - - typeof(*ptr) atomic_inc_fetch(ptr) - typeof(*ptr) atomic_dec_fetch(ptr) - typeof(*ptr) atomic_add_fetch(ptr, val) - typeof(*ptr) atomic_sub_fetch(ptr, val) - typeof(*ptr) atomic_and_fetch(ptr, val) - typeof(*ptr) atomic_or_fetch(ptr, val) - typeof(*ptr) atomic_xor_fetch(ptr, val) - -Sequentially consistent loads and stores can be done using: - - atomic_fetch_add(ptr, 0) for loads - atomic_xchg(ptr, val) for stores - -However, they are quite expensive on some platforms, notably POWER and -Arm. Therefore, qemu/atomic.h provides two primitives with slightly -weaker constraints: - - typeof(*ptr) atomic_mb_read(ptr) - void atomic_mb_set(ptr, val) - -The semantics of these primitives map to Java volatile variables, -and are strongly related to memory barriers as used in the Linux -kernel (see below). - -As long as you use atomic_mb_read and atomic_mb_set, accesses cannot -be reordered with each other, and it is also not possible to reorder -"normal" accesses around them. - -However, and this is the important difference between -atomic_mb_read/atomic_mb_set and sequential consistency, it is important -for both threads to access the same volatile variable. It is not the -case that everything visible to thread A when it writes volatile field f -becomes visible to thread B after it reads volatile field g. The store -and load have to "match" (i.e., be performed on the same volatile -field) to achieve the right semantics. - - -These operations operate on any type that is as wide as an int or smaller. - - -WEAK ATOMIC ACCESS AND MANUAL MEMORY BARRIERS -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D - -Compared to sequentially consistent atomic access, programming with -weaker consistency models can be considerably more complicated. -In general, if the algorithm you are writing includes both writes -and reads on the same side, it is generally simpler to use sequentially -consistent primitives. - -When using this model, variables are accessed with: - -- atomic_read() and atomic_set(); these prevent the compiler from - optimizing accesses out of existence and creating unsolicited - accesses, but do not otherwise impose any ordering on loads and - stores: both the compiler and the processor are free to reorder - them. - -- atomic_load_acquire(), which guarantees the LOAD to appear to - happen, with respect to the other components of the system, - before all the LOAD or STORE operations specified afterwards. - Operations coming before atomic_load_acquire() can still be - reordered after it. - -- atomic_store_release(), which guarantees the STORE to appear to - happen, with respect to the other components of the system, - after all the LOAD or STORE operations specified afterwards. - Operations coming after atomic_store_release() can still be - reordered after it. - -Restrictions to the ordering of accesses can also be specified -using the memory barrier macros: smp_rmb(), smp_wmb(), smp_mb(), -smp_mb_acquire(), smp_mb_release(), smp_read_barrier_depends(). - -Memory barriers control the order of references to shared memory. -They come in six kinds: - -- smp_rmb() guarantees that all the LOAD operations specified before - the barrier will appear to happen before all the LOAD operations - specified after the barrier with respect to the other components of - the system. - - In other words, smp_rmb() puts a partial ordering on loads, but is not - required to have any effect on stores. - -- smp_wmb() guarantees that all the STORE operations specified before - the barrier will appear to happen before all the STORE operations - specified after the barrier with respect to the other components of - the system. - - In other words, smp_wmb() puts a partial ordering on stores, but is not - required to have any effect on loads. - -- smp_mb_acquire() guarantees that all the LOAD operations specified before - the barrier will appear to happen before all the LOAD or STORE operations - specified after the barrier with respect to the other components of - the system. - -- smp_mb_release() guarantees that all the STORE operations specified *aft= er* - the barrier will appear to happen after all the LOAD or STORE operations - specified *before* the barrier with respect to the other components of - the system. - -- smp_mb() guarantees that all the LOAD and STORE operations specified - before the barrier will appear to happen before all the LOAD and - STORE operations specified after the barrier with respect to the other - components of the system. - - smp_mb() puts a partial ordering on both loads and stores. It is - stronger than both a read and a write memory barrier; it implies both - smp_mb_acquire() and smp_mb_release(), but it also prevents STOREs - coming before the barrier from overtaking LOADs coming after the - barrier and vice versa. - -- smp_read_barrier_depends() is a weaker kind of read barrier. On - most processors, whenever two loads are performed such that the - second depends on the result of the first (e.g., the first load - retrieves the address to which the second load will be directed), - the processor will guarantee that the first LOAD will appear to happen - before the second with respect to the other components of the system. - However, this is not always true---for example, it was not true on - Alpha processors. Whenever this kind of access happens to shared - memory (that is not protected by a lock), a read barrier is needed, - and smp_read_barrier_depends() can be used instead of smp_rmb(). - - Note that the first load really has to have a _data_ dependency and not - a control dependency. If the address for the second load is dependent - on the first load, but the dependency is through a conditional rather - than actually loading the address itself, then it's a _control_ - dependency and a full read barrier or better is required. - - -This is the set of barriers that is required *between* two atomic_read() -and atomic_set() operations to achieve sequential consistency: - - | 2nd operation | - |-----------------------------------------------| - 1st operation | (after last) | atomic_read | atomic_set | - ---------------+----------------+-------------+----------------| - (before first) | | none | smp_mb_release | - ---------------+----------------+-------------+----------------| - atomic_read | smp_mb_acquire | smp_rmb | ** | - ---------------+----------------+-------------+----------------| - atomic_set | none | smp_mb()*** | smp_wmb() | - ---------------+----------------+-------------+----------------| - - * Or smp_read_barrier_depends(). - - ** This requires a load-store barrier. This is achieved by - either smp_mb_acquire() or smp_mb_release(). - - *** This requires a store-load barrier. On most machines, the only - way to achieve this is a full barrier. - - -You can see that the two possible definitions of atomic_mb_read() -and atomic_mb_set() are the following: - - 1) atomic_mb_read(p) =3D atomic_read(p); smp_mb_acquire() - atomic_mb_set(p, v) =3D smp_mb_release(); atomic_set(p, v); smp_mb() - - 2) atomic_mb_read(p) =3D smp_mb() atomic_read(p); smp_mb_acquire() - atomic_mb_set(p, v) =3D smp_mb_release(); atomic_set(p, v); - -Usually the former is used, because smp_mb() is expensive and a program -normally has more reads than writes. Therefore it makes more sense to -make atomic_mb_set() the more expensive operation. - -There are two common cases in which atomic_mb_read and atomic_mb_set -generate too many memory barriers, and thus it can be useful to manually -place barriers, or use atomic_load_acquire/atomic_store_release instead: - -- when a data structure has one thread that is always a writer - and one thread that is always a reader, manual placement of - memory barriers makes the write side faster. Furthermore, - correctness is easy to check for in this case using the "pairing" - trick that is explained below: - - thread 1 thread 1 - ------------------------- ------------------------ - (other writes) - atomic_mb_set(&a, x) atomic_store_release(&a, x) - atomic_mb_set(&b, y) atomic_store_release(&b, y) - - =3D> - thread 2 thread 2 - ------------------------- ------------------------ - y =3D atomic_mb_read(&b) y =3D atomic_load_acquire(&= b) - x =3D atomic_mb_read(&a) x =3D atomic_load_acquire(&= a) - (other reads) - - Note that the barrier between the stores in thread 1, and between - the loads in thread 2, has been optimized here to a write or a - read memory barrier respectively. On some architectures, notably - ARMv7, smp_mb_acquire and smp_mb_release are just as expensive as - smp_mb, but smp_rmb and/or smp_wmb are more efficient. - -- sometimes, a thread is accessing many variables that are otherwise - unrelated to each other (for example because, apart from the current - thread, exactly one other thread will read or write each of these - variables). In this case, it is possible to "hoist" the implicit - barriers provided by atomic_mb_read() and atomic_mb_set() outside - a loop. For example, the above definition atomic_mb_read() gives - the following transformation: - - n =3D 0; n =3D 0; - for (i =3D 0; i < 10; i++) =3D> for (i =3D 0; i < 10; i++) - n +=3D atomic_mb_read(&a[i]); n +=3D atomic_read(&a[i]); - smp_mb_acquire(); - - Similarly, atomic_mb_set() can be transformed as follows: - - smp_mb_release(); - for (i =3D 0; i < 10; i++) =3D> for (i =3D 0; i < 10; i++) - atomic_mb_set(&a[i], false); atomic_set(&a[i], false); - smp_mb(); - - - The other thread can still use atomic_mb_read()/atomic_mb_set(). - -The two tricks can be combined. In this case, splitting a loop in -two lets you hoist the barriers out of the loops _and_ eliminate the -expensive smp_mb(): - - smp_mb_release(); - for (i =3D 0; i < 10; i++) { =3D> for (i =3D 0; i < 10; i++) - atomic_mb_set(&a[i], false); atomic_set(&a[i], false); - atomic_mb_set(&b[i], false); smb_wmb(); - } for (i =3D 0; i < 10; i++) - atomic_set(&a[i], false); - smp_mb(); - - -Memory barrier pairing ----------------------- - -A useful rule of thumb is that memory barriers should always, or almost -always, be paired with another barrier. In the case of QEMU, however, -note that the other barrier may actually be in a driver that runs in -the guest! - -For the purposes of pairing, smp_read_barrier_depends() and smp_rmb() -both count as read barriers. A read barrier shall pair with a write -barrier or a full barrier; a write barrier shall pair with a read -barrier or a full barrier. A full barrier can pair with anything. -For example: - - thread 1 thread 2 - =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D - a =3D 1; - smp_wmb(); - b =3D 2; x =3D b; - smp_rmb(); - y =3D a; - -Note that the "writing" thread is accessing the variables in the -opposite order as the "reading" thread. This is expected: stores -before the write barrier will normally match the loads after the -read barrier, and vice versa. The same is true for more than 2 -access and for data dependency barriers: - - thread 1 thread 2 - =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D - b[2] =3D 1; - smp_wmb(); - x->i =3D 2; - smp_wmb(); - a =3D x; x =3D a; - smp_read_barrier_depends(); - y =3D x->i; - smp_read_barrier_depends(); - z =3D b[y]; - -smp_wmb() also pairs with atomic_mb_read() and smp_mb_acquire(). -and smp_rmb() also pairs with atomic_mb_set() and smp_mb_release(). - - -COMPARISON WITH LINUX KERNEL MEMORY BARRIERS -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D - -Here is a list of differences between Linux kernel atomic operations -and memory barriers, and the equivalents in QEMU: - -- atomic operations in Linux are always on a 32-bit int type and - use a boxed atomic_t type; atomic operations in QEMU are polymorphic - and use normal C types. - -- Originally, atomic_read and atomic_set in Linux gave no guarantee - at all. Linux 4.1 updated them to implement volatile - semantics via ACCESS_ONCE (or the more recent READ/WRITE_ONCE). - - QEMU's atomic_read/set implement, if the compiler supports it, C11 - atomic relaxed semantics, and volatile semantics otherwise. - Both semantics prevent the compiler from doing certain transformations; - the difference is that atomic accesses are guaranteed to be atomic, - while volatile accesses aren't. Thus, in the volatile case we just cross - our fingers hoping that the compiler will generate atomic accesses, - since we assume the variables passed are machine-word sized and - properly aligned. - No barriers are implied by atomic_read/set in either Linux or QEMU. - -- atomic read-modify-write operations in Linux are of three kinds: - - atomic_OP returns void - atomic_OP_return returns new value of the variable - atomic_fetch_OP returns the old value of the variable - atomic_cmpxchg returns the old value of the variable - - In QEMU, the second kind does not exist. Currently Linux has - atomic_fetch_or only. QEMU provides and, or, inc, dec, add, sub. - -- different atomic read-modify-write operations in Linux imply - a different set of memory barriers; in QEMU, all of them enforce - sequential consistency, which means they imply full memory barriers - before and after the operation. - -- Linux does not have an equivalent of atomic_mb_set(). In particular, - note that smp_store_mb() is a little weaker than atomic_mb_set(). - atomic_mb_read() compiles to the same instructions as Linux's - smp_load_acquire(), but this should be treated as an implementation - detail. - -SOURCES -=3D=3D=3D=3D=3D=3D=3D - -* Documentation/memory-barriers.txt from the Linux kernel - -* "The JSR-133 Cookbook for Compiler Writers", available at - http://g.oswego.edu/dl/jmm/cookbook.html diff --git a/docs/devel/index.rst b/docs/devel/index.rst index b734ba4655..a9e1200dff 100644 --- a/docs/devel/index.rst +++ b/docs/devel/index.rst @@ -17,6 +17,7 @@ Contents: loads-stores memory migration + atomics stable-process testing decodetree --=20 2.18.2 From nobody Sun May 19 06:04:49 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1586268620; cv=none; d=zohomail.com; s=zohoarc; b=WBQlYE976SJKu+YIjrZAhg99m6CamgLBSotJq+qLUNU9G9htQojBmTuLffQxVkt5HlKiNHiNamcziqSmYaTOpF68adbJV52fY07lYhO3tHLbXcBRRn5gjeOyU7d0zjn9Fo0HMtE4R0YeAIWqRLOjpciol4jKYtow7QXWjSQGWsw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1586268620; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=d23dN8fiSuAwS50i3TRuFinurtR/I84rxZJYmzdtPIc=; b=L5VJ2u+wV5XqkH+NKuzoOi2UdFSYw4KhfkfpUVQNzOyUpzNQUFOIHDWU5MbMQKMcSgRBYHgQOLcCmVZ59LjV5Ns3cGjMQK8eXeTEp3KgMCFt/9U27C5zlZ54iQIWZvPVv9Ld8XEZ1SXGj0VBJ3wukRBPOTk2+EUiym3y0g0Zt7k= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1586268620989343.59278936541716; Tue, 7 Apr 2020 07:10:20 -0700 (PDT) Received: from localhost ([::1]:47892 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jLovj-0004Wl-Ik for importer@patchew.org; Tue, 07 Apr 2020 10:10:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51199) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jLotT-0000a6-Bp for qemu-devel@nongnu.org; Tue, 07 Apr 2020 10:08:02 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jLotQ-00013J-81 for qemu-devel@nongnu.org; Tue, 07 Apr 2020 10:07:59 -0400 Received: from us-smtp-2.mimecast.com ([207.211.31.81]:24730 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1jLotP-00012N-Uc for qemu-devel@nongnu.org; Tue, 07 Apr 2020 10:07:56 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-283-m8r81iZLO9OdLnfHa3aWEA-1; Tue, 07 Apr 2020 10:07:53 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 2F21F800D53; Tue, 7 Apr 2020 14:07:52 +0000 (UTC) Received: from virtlab701.virt.lab.eng.bos.redhat.com (virtlab701.virt.lab.eng.bos.redhat.com [10.19.152.228]) by smtp.corp.redhat.com (Postfix) with ESMTP id 832E85C1BB; Tue, 7 Apr 2020 14:07:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1586268475; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=d23dN8fiSuAwS50i3TRuFinurtR/I84rxZJYmzdtPIc=; b=dBvnPmvfRawsUFOLaf6ovMsXKBV6+4q/+vCGhqwWOsyziwx6U9SjGmzjaacv65NBxaM5/R nlZOKmhuiRTg/PKPMAYPbcdr0GXVNaB42dXG735Te0/lcpBzOpwCrBPh6/BtilSebhtclB zvRPbf7b0p20GXkNt8k+yFTi+BL+VCs= X-MC-Unique: m8r81iZLO9OdLnfHa3aWEA-1 From: Paolo Bonzini To: qemu-devel@nongnu.org Subject: [PATCH 2/5] atomics: update documentation Date: Tue, 7 Apr 2020 10:07:43 -0400 Message-Id: <20200407140746.8041-3-pbonzini@redhat.com> In-Reply-To: <20200407140746.8041-1-pbonzini@redhat.com> References: <20200407140746.8041-1-pbonzini@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 207.211.31.81 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fangying1@huawei.com, stefanha@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Some of the constraints on operand sizes have been relaxed, so adjust the documentation. Deprecate atomic_mb_read and atomic_mb_set; it is not really possible to use them correctly because they do not interoperate with sequentially-consi= stent RMW operations. Finally, extend the memory barrier pairing section to cover acquire and release semantics in general, roughly based on the KVM Forum 2016 talk, " weapons". Signed-off-by: Paolo Bonzini Reviewed-by: Richard Henderson --- docs/devel/atomics.rst | 467 +++++++++++++++++++++++------------------ 1 file changed, 261 insertions(+), 206 deletions(-) diff --git a/docs/devel/atomics.rst b/docs/devel/atomics.rst index 83ed3d6981..18bec11edd 100644 --- a/docs/devel/atomics.rst +++ b/docs/devel/atomics.rst @@ -11,10 +11,15 @@ that is consistent with the expectations of the program= mer. The most basic tool is locking. Mutexes, condition variables and semaphores are used in QEMU, and should be the default approach to synchronization. Anything else is considerably harder, but it's -also justified more often than one would like. The two tools that -are provided by ``qemu/atomic.h`` are memory barriers and atomic operation= s. +also justified more often than one would like; +the most performance-critical parts of QEMU in particular require +a very low level approach to concurrency, involving memory barriers +and atomic operations. The semantics of concurrent memory accesses are go= verned +by the C11 memory model. =20 -Macros defined by ``qemu/atomic.h`` fall in three camps: +QEMU provides a header, ``qemu/atomic.h``, which wraps C11 atomics to +provide better portability and a less verbose syntax. ``qemu/atomic.h`` +provides macros that fall in three camps: =20 - compiler barriers: ``barrier()``; =20 @@ -24,13 +29,21 @@ Macros defined by ``qemu/atomic.h`` fall in three camps: =20 - sequentially consistent atomic access: everything else. =20 +In general, use of ``qemu/atomic.h`` should be wrapped with more easily +used data structures (e.g. the lock-free singly-linked list operations +``QSLIST_INSERT_HEAD_ATOMIC`` and ``QSLIST_MOVE_ATOMIC``) or synchronizati= on +primitives (such as RCU, ``QemuEvent`` or ``QemuLockCnt``). Bare use of +atomic operations and memory barriers should be limited to inter-thread +checking of flags and documented thoroughly. + + =20 Compiler memory barrier =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 -``barrier()`` prevents the compiler from moving the memory accesses either -side of it to the other side. The compiler barrier has no direct effect -on the CPU, which may then reorder things however it wishes. +``barrier()`` prevents the compiler from moving the memory accesses on +either side of it to the other side. The compiler barrier has no direct +effect on the CPU, which may then reorder things however it wishes. =20 ``barrier()`` is mostly used within ``qemu/atomic.h`` itself. On some architectures, CPU guarantees are strong enough that blocking compiler @@ -73,7 +86,8 @@ operations:: typeof(*ptr) atomic_cmpxchg(ptr, old, new) =20 all of which return the old value of ``*ptr``. These operations are -polymorphic; they operate on any type that is as wide as a pointer. +polymorphic; they operate on any type that is as wide as a pointer or +smaller. =20 Similar operations return the new value of ``*ptr``:: =20 @@ -85,36 +99,28 @@ Similar operations return the new value of ``*ptr``:: typeof(*ptr) atomic_or_fetch(ptr, val) typeof(*ptr) atomic_xor_fetch(ptr, val) =20 -Sequentially consistent loads and stores can be done using:: - - atomic_fetch_add(ptr, 0) for loads - atomic_xchg(ptr, val) for stores - -However, they are quite expensive on some platforms, notably POWER and -Arm. Therefore, qemu/atomic.h provides two primitives with slightly -weaker constraints:: +``qemu/atomic.h`` also provides loads and stores that cannot be reordered +with each other:: =20 typeof(*ptr) atomic_mb_read(ptr) void atomic_mb_set(ptr, val) =20 -The semantics of these primitives map to Java volatile variables, -and are strongly related to memory barriers as used in the Linux -kernel (see below). +However these do not provide sequential consistency and, in particular, +they do not participate in the total ordering enforced by +sequentially-consistent operations. For this reason they are deprecated. +They should instead be replaced with any of the following (ordered from +easiest to hardest): =20 -As long as you use atomic_mb_read and atomic_mb_set, accesses cannot -be reordered with each other, and it is also not possible to reorder -"normal" accesses around them. +- accesses inside a mutex or spinlock =20 -However, and this is the important difference between -atomic_mb_read/atomic_mb_set and sequential consistency, it is important -for both threads to access the same volatile variable. It is not the -case that everything visible to thread A when it writes volatile field f -becomes visible to thread B after it reads volatile field g. The store -and load have to "match" (i.e., be performed on the same volatile -field) to achieve the right semantics. +- lightweight synchronization primitives such as ``QemuEvent`` =20 +- RCU operations (``atomic_rcu_read``, ``atomic_rcu_set``) when publishing + or accessing a new version of a data structure =20 -These operations operate on any type that is as wide as an int or smaller. +- other atomic accesses: ``atomic_read`` and ``atomic_load_acquire`` for + loads, ``atomic_set`` and ``atomic_store_release`` for stores, ``smp_mb`` + to forbid reordering subsequent loads before a store. =20 =20 Weak atomic access and manual memory barriers @@ -122,9 +128,24 @@ Weak atomic access and manual memory barriers =20 Compared to sequentially consistent atomic access, programming with weaker consistency models can be considerably more complicated. -In general, if the algorithm you are writing includes both writes -and reads on the same side, it is generally simpler to use sequentially -consistent primitives. +The only guarantees that you can rely upon in this case are: + +- atomic accesses will not cause data races (and hence undefined behavior); + ordinary accesses instead cause data races if they are concurrent with + other accesses and at least one access is a write. In order to ensure t= his, + the compiler will not optimize accesses out of existence, create unsolic= ited + accesses, or perform other similar optimzations. + +- acquire operations will appear to happen, with respect to the other + components of the system, before all the LOAD or STORE operations + specified afterwards. + +- release operations will appear to happen, with respect to the other + components of the system, after all the LOAD or STORE operations + specified afterwards. + +- release operations will *synchronize with* acquire operations; + see :ref:`acqrel` for a detailed explanation. =20 When using this model, variables are accessed with: =20 @@ -208,168 +229,182 @@ They come in six kinds: dependency and a full read barrier or better is required. =20 =20 -This is the set of barriers that is required *between* two ``atomic_read()= `` -and ``atomic_set()`` operations to achieve sequential consistency: - - +----------------+-----------------------------------------------------= --+ - | | 2nd operation = | - | +------------------+-----------------+----------------= --+ - | 1st operation | (after last) | atomic_read | atomic_set = | - +----------------+------------------+-----------------+----------------= --+ - | (before first) | .. | none | smp_mb_release(= ) | - +----------------+------------------+-----------------+----------------= --+ - | atomic_read | smp_mb_acquire() | smp_rmb() [1]_ | [2]_ = | - +----------------+------------------+-----------------+----------------= --+ - | atomic_set | none | smp_mb() [3]_ | smp_wmb() = | - +----------------+------------------+-----------------+----------------= --+ - - .. [1] Or smp_read_barrier_depends(). - - .. [2] This requires a load-store barrier. This is achieved by - either smp_mb_acquire() or smp_mb_release(). - - .. [3] This requires a store-load barrier. On most machines, the only - way to achieve this is a full barrier. - - -You can see that the two possible definitions of ``atomic_mb_read()`` -and ``atomic_mb_set()`` are the following: - - 1) | atomic_mb_read(p) =3D atomic_read(p); smp_mb_acquire() - | atomic_mb_set(p, v) =3D smp_mb_release(); atomic_set(p, v); smp_mb() - - 2) | atomic_mb_read(p) =3D smp_mb() atomic_read(p); smp_mb_acquire() - | atomic_mb_set(p, v) =3D smp_mb_release(); atomic_set(p, v); - -Usually the former is used, because ``smp_mb()`` is expensive and a program -normally has more reads than writes. Therefore it makes more sense to -make ``atomic_mb_set()`` the more expensive operation. - -There are two common cases in which atomic_mb_read and atomic_mb_set -generate too many memory barriers, and thus it can be useful to manually -place barriers, or use atomic_load_acquire/atomic_store_release instead: - -- when a data structure has one thread that is always a writer - and one thread that is always a reader, manual placement of - memory barriers makes the write side faster. Furthermore, - correctness is easy to check for in this case using the "pairing" - trick that is explained below: - - +---------------------------------------------------------------------= -+ - | thread 1 = | - +-----------------------------------+---------------------------------= -+ - | before | after = | - +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+ - | :: | :: = | - | | = | - | (other writes) | = | - | atomic_mb_set(&a, x) | atomic_store_release(&a, x) = | - | atomic_mb_set(&b, y) | atomic_store_release(&b, y) = | - +-----------------------------------+---------------------------------= -+ - - +---------------------------------------------------------------------= -+ - | thread 2 = | - +-----------------------------------+---------------------------------= -+ - | before | after = | - +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+ - | :: | :: = | - | | = | - | y =3D atomic_mb_read(&b) | y =3D atomic_load_acquire(&b= ) | - | x =3D atomic_mb_read(&a) | x =3D atomic_load_acquire(&a= ) | - | (other reads) | = | - +-----------------------------------+---------------------------------= -+ - - Note that the barrier between the stores in thread 1, and between - the loads in thread 2, has been optimized here to a write or a - read memory barrier respectively. On some architectures, notably - ARMv7, smp_mb_acquire and smp_mb_release are just as expensive as - smp_mb, but smp_rmb and/or smp_wmb are more efficient. - -- sometimes, a thread is accessing many variables that are otherwise - unrelated to each other (for example because, apart from the current - thread, exactly one other thread will read or write each of these - variables). In this case, it is possible to "hoist" the implicit - barriers provided by ``atomic_mb_read()`` and ``atomic_mb_set()`` outside - a loop. For example, the above definition ``atomic_mb_read()`` gives - the following transformation: - - +-----------------------------------+---------------------------------= -+ - | before | after = | - +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+ - | :: | :: = | - | | = | - | n =3D 0; | n =3D 0; = | - | for (i =3D 0; i < 10; i++) | for (i =3D 0; i < 10; i++) = | - | n +=3D atomic_mb_read(&a[i]); | n +=3D atomic_read(&a[i]);= | - | | smp_mb_acquire(); = | - +-----------------------------------+---------------------------------= -+ - - Similarly, atomic_mb_set() can be transformed as follows: - - +-----------------------------------+---------------------------------= -+ - | before | after = | - +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+ - | :: | :: = | - | | = | - | | smp_mb_release(); = | - | for (i =3D 0; i < 10; i++) | for (i =3D 0; i < 10; i++) = | - | atomic_mb_set(&a[i], false); | atomic_set(&a[i], false); = | - | | smp_mb(); = | - +-----------------------------------+---------------------------------= -+ - - - The other thread can still use ``atomic_mb_read()``/``atomic_mb_set()``. - -The two tricks can be combined. In this case, splitting a loop in -two lets you hoist the barriers out of the loops _and_ eliminate the -expensive ``smp_mb()``: - - +-----------------------------------+---------------------------------= -+ - | before | after = | - +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+ - | :: | :: = | - | | = | - | | smp_mb_release(); = | - | for (i =3D 0; i < 10; i++) { | for (i =3D 0; i < 10; i++)= | - | atomic_mb_set(&a[i], false); | atomic_set(&a[i], false); = | - | atomic_mb_set(&b[i], false); | smb_wmb(); = | - | } | for (i =3D 0; i < 10; i++) = | - | | atomic_set(&a[i], false); = | - | | smp_mb(); = | - +-----------------------------------+---------------------------------= -+ - - -Memory barrier pairing ----------------------- - -A useful rule of thumb is that memory barriers should always, or almost -always, be paired with another barrier. In the case of QEMU, however, -note that the other barrier may actually be in a driver that runs in -the guest! - -For the purposes of pairing, ``smp_read_barrier_depends()`` and ``smp_rmb(= )`` -both count as read barriers. A read barrier shall pair with a write -barrier or a full barrier; a write barrier shall pair with a read -barrier or a full barrier. A full barrier can pair with anything. -For example: - - +--------------------+------------------------------+ - | thread 1 | thread 2 | - +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D+ - | :: | :: | - | | | - | a =3D 1; | | - | smp_wmb(); | | - | b =3D 2; | x =3D b; | - | | smp_rmb(); | - | | y =3D a; | - +--------------------+------------------------------+ +Memory barriers and ``atomic_load_acquire``/``atomic_store_release`` are +mostly used when a data structure has one thread that is always a writer +and one thread that is always a reader: + + +----------------------------------+----------------------------------+ + | thread 1 | thread 2 | + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+ + | :: | :: | + | | | + | atomic_store_release(&a, x); | y =3D atomic_load_acquire(&b); = | + | atomic_store_release(&b, y); | x =3D atomic_load_acquire(&a); = | + +----------------------------------+----------------------------------+ + +In this case, correctness is easy to check for in this case using the +"pairing" trick that is explained below. + +Sometimes, a thread is accessing many variables that are otherwise +unrelated to each other (for example because, apart from the current +thread, exactly one other thread will read or write each of these +variables). In this case, it is possible to "hoist" the barriers +outside a loop. For example: + + +------------------------------------------+--------------------------= --------+ + | before | after = | + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D+ + | :: | :: = | + | | = | + | n =3D 0; | n =3D 0; = | + | for (i =3D 0; i < 10; i++) | for (i =3D 0; i < 10;= i++) | + | n +=3D atomic_load_acquire(&a[i]); | n +=3D atomic_read(= &a[i]); | + | | smp_mb_acquire(); = | + +------------------------------------------+--------------------------= --------+ + | :: | :: = | + | | = | + | | smp_mb_release(); = | + | for (i =3D 0; i < 10; i++) | for (i =3D 0; i < 10;= i++) | + | atomic_store_release(&a[i], false); | atomic_set(&a[i], fal= se); | + +------------------------------------------+--------------------------= --------+ + +Splitting a loop can also be useful to reduce the number of barriers: + + +------------------------------------------+--------------------------= --------+ + | before | after = | + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D+ + | :: | :: = | + | | = | + | n =3D 0; | smp_mb_release(); = | + | for (i =3D 0; i < 10; i++) { | for (i =3D 0; i < 1= 0; i++) | + | atomic_store_release(&a[i], false); | atomic_set(&a[i], f= alse); | + | smp_mb(); | smb_mb(); = | + | n +=3D atomic_read(&b[i]); | n =3D 0; = | + | } | for (i =3D 0; i < 10;= i++) | + | | n +=3D atomic_read(= &b[i]); | + +------------------------------------------+--------------------------= --------+ + +In this case, a ``smp_mb_release()`` is also replaced with a (possibly che= aper, and clearer +as well) ``smp_wmb()``: + + +------------------------------------------+--------------------------= --------+ + | before | after = | + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D+ + | :: | :: = | + | | = | + | | smp_mb_release(); = | + | for (i =3D 0; i < 10; i++) { | for (i =3D 0; i < 1= 0; i++) | + | atomic_store_release(&a[i], false); | atomic_set(&a[i], f= alse); | + | atomic_store_release(&b[i], false); | smb_wmb(); = | + | } | for (i =3D 0; i < 10;= i++) | + | | atomic_set(&b[i], f= alse); | + +------------------------------------------+--------------------------= --------+ + + +.. _acqrel: + +Acquire/release pairing and the *synchronizes-with* relation +------------------------------------------------------------ + +Atomic operations other than ``atomic_set()`` and ``atomic_read()`` have +either *acquire* or *release* semantics. This has two effects: + +- within a thread, they are ordered either after previous operations (for + acquire) or before subsequent operations (for release). + +- if a release operation in one thread *synchronizes with* an acquire oper= ation in another + thread, the ordering constraints propagates from the first to the + second thread. That is, everything before the release operation in the + first thread is guaranteed to *happen before* everything after the + acquire operation in the second thread. + +The concept of acquire and release semantics is not exclusive to atomic +operations; almost all higher-level synchronization primitives also have +acquire or release semantics. For example: + +- ``pthread_mutex_lock`` has acquire semantics, ``pthread_mutex_unlock`` h= as + release semantics and synchronizes with a ``pthread_mutex_lock`` for the + same mutex. + +- ``pthread_cond_broadcast`` has release semantics, ``pthread_cond_wait`` = has + both release semantics (for loads and stores before the wait) and acquire + semantics (for loads and stores after the wakeup) + +- ``pthread_create`` has release semantics and synchronizes with the start + of the new thread; ``pthread_join`` has acquire semantics and synchroniz= es + with the exiting of the thread. + +- ``qemu_event_set`` has release semantics, ``qemu_event_wait`` has + acquire semantics + +For example, in the following example there are no atomic accesses, but st= ill +thread 2 is relying on the *synchronizes-with* relation between ``pthread_= exit`` +(release) and ``pthread_join`` (acquire): + + +----------------------+-------------------------------+ + | thread 1 | thread 2 | + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D+ + | :: | :: | + | | | + | *a =3D 1; | | + | pthread_exit(a); | pthread_join(thread1, &a); | + | | x =3D *a; | + +----------------------+-------------------------------+ + +As a result, a useful rule of thumb is that atomic operations other than +``atomic_set()`` and ``atomic_read()`` will almost always be paired with a= nother +operation of the opposite kind: an acquire operation will pair with +a release operation and vice versa. In the case of QEMU, however, note +that the other barrier may actually be in a driver that runs in the guest! + +``smp_read_barrier_depends()``, ``smp_rmb()``, ``smp_mb_acquire()``, +``atomic_load_acquire()`` and ``atomic_rcu_read()`` all count +as acquire operations. ``smp_wmb()``, ``smp_mb_release()``, +``atomic_store_release()`` and ``atomic_rcu_set()`` all count as release +operations. ``smp_mb()`` counts as both acquire and release, therefore +it can pair with any other atomic operation. Here is an example: + + +----------------------+------------------------------+ + | thread 1 | thread 2 | + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D+ + | :: | :: | + | | | + | atomic_set(&a, 1); | | + | smp_wmb(); | | + | atomic_set(&b, 2); | x =3D atomic_read(&b); | + | | smp_rmb(); | + | | y =3D atomic_read(&a); | + +----------------------+------------------------------+ + +Note that a load-store pair only counts if the two operations access the +same variable: that is, a store-release on a variable ``x`` *synchronizes +with* a load-acquire on a variable ``x``, while a release barrier +synchronizes with any acquire operation. The following example shows +correct synchronization: + + +--------------------------------+--------------------------------+ + | thread 1 | thread 2 | + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+ + | :: | :: | + | | | + | atomic_set(&a, 1); | | + | atomic_store_release(&b, 2); | x =3D atomic_load_acquire(&b); | + | | y =3D atomic_read(&a); | + +--------------------------------+--------------------------------+ + +Acquire and release semantics of higher-level synchronization primitives +can also be relied upon for the purpose of pairing memory barriers or +acquire/release operations. =20 Note that the "writing" thread is accessing the variables in the opposite order as the "reading" thread. This is expected: stores -before the write barrier will normally match the loads after the -read barrier, and vice versa. The same is true for more than 2 -access and for data dependency barriers: +before a release operation or write barrier will normally match the loads +after the acquire operation or read barrier, and vice versa. The same +was true in the ``pthread_exit``/``pthread_join`` example above. + +Finally, this more complex example has more than two accesses and data +dependency barriers. It also does not use atomic accesses whenever there +cannot be a data race: =20 +----------------------+------------------------------+ | thread 1 | thread 2 | @@ -380,17 +415,13 @@ access and for data dependency barriers: | smp_wmb(); | | | x->i =3D 2; | | | smp_wmb(); | | - | a =3D x; | x =3D a; | + | atomic_set(&a, x); | x =3D atomic_read(&a); | | | smp_read_barrier_depends(); | | | y =3D x->i; | | | smp_read_barrier_depends(); | | | z =3D b[y]; | +----------------------+------------------------------+ =20 -``smp_wmb()`` also pairs with ``atomic_mb_read()`` and ``smp_mb_acquire()`= `. -and ``smp_rmb()`` also pairs with ``atomic_mb_set()`` and ``smp_mb_release= ()``. - - Comparison with Linux kernel memory barriers =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 @@ -426,19 +457,43 @@ and memory barriers, and the equivalents in QEMU: ``atomic_cmpxchg`` returns the old value of the variable =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 - In QEMU, the second kind does not exist. Currently Linux has - atomic_fetch_or only. QEMU provides and, or, inc, dec, add, sub. + In QEMU, the second kind is named ``atomic_OP_fetch``. =20 - different atomic read-modify-write operations in Linux imply a different set of memory barriers; in QEMU, all of them enforce - sequential consistency, which means they imply full memory barriers - before and after the operation. - -- Linux does not have an equivalent of ``atomic_mb_set()``. In particular, - note that ``smp_store_mb()`` is a little weaker than ``atomic_mb_set()``. - ``atomic_mb_read()`` compiles to the same instructions as Linux's - ``smp_load_acquire()``, but this should be treated as an implementation - detail. + sequential consistency. + +- in QEMU, ``atomic_read()`` and ``atomic_set()`` do not participate in + the total ordering enforced by sequentially-consistent operations. + This is because QEMU uses the C11 memory model. The following example + is correct in Linux but not in QEMU: + + +----------------------------------+--------------------------------+ + | Linux (correct) | QEMU (incorrect) | + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+ + | :: | :: | + | | | + | a =3D atomic_fetch_add(&x, 2); | a =3D atomic_fetch_add(&x, = 2); | + | b =3D READ_ONCE(&y); | b =3D atomic_read(&y); = | + +----------------------------------+--------------------------------+ + + because the read of ``y`` can be moved (by either the processor or the + compiler) before the write of ``x``. + + Fixing this requires an ``smp_mb()`` memory barrier between the write + of ``x`` and the read of ``y``. In the common case where only one thread + writes ``x``, it is also possible to write it like this: + + +--------------------------------+ + | QEMU (correct) | + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+ + | :: | + | | + | a =3D atomic_read(&x); | + | atomic_set(&x, a + 2); | + | smp_mb(); | + | b =3D atomic_read(&y); | + +--------------------------------+ =20 Sources =3D=3D=3D=3D=3D=3D=3D --=20 2.18.2 From nobody Sun May 19 06:04:49 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1586268536; cv=none; d=zohomail.com; s=zohoarc; b=DKmVT5yZD3wKrmfM1qMGmV1JmiHIZTwJLM8vILsZMWQ6QEkF2a2L7d+wOnC/P97pgSGaUuyPrXnuRTtre0eInBQk2I/AqYlSPj8FeexR0vT9HqCsgj95W+7BFWwRnemHTXtJeeA18dH2jhfDAsKcSPKLbgfGQy3gzwzpwymPX+M= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1586268536; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=XRLWXRytzTGVO0r/GudLCTGjNoCUKmXsKyZKPig2BiY=; b=oAvp+1oaQ2+wnfoaL2guvaVpXOy9vD2PXB89ZVQUVDPuLmCd6LlPctY3WpUq/ipvKvkyX4qEYYQbsg2t5T5HhiQiNTFWABqRELqr/FG+rZ/4khtOfVOxeGpUVDCsvMQQujAgI7ZPBr58rllHYWMKHZFcq9mR0fvBfju80VSoElE= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 158626853676918.494841318042518; Tue, 7 Apr 2020 07:08:56 -0700 (PDT) Received: from localhost ([::1]:47872 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jLouN-0002AM-77 for importer@patchew.org; Tue, 07 Apr 2020 10:08:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51174) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jLotR-0000XT-NS for qemu-devel@nongnu.org; Tue, 07 Apr 2020 10:07:58 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jLotQ-000137-6p for qemu-devel@nongnu.org; Tue, 07 Apr 2020 10:07:57 -0400 Received: from us-smtp-1.mimecast.com ([205.139.110.61]:31061 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1jLotP-00011V-Un for qemu-devel@nongnu.org; Tue, 07 Apr 2020 10:07:56 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-386-xn0n26WHNKC0s4Vj2PZjmA-1; Tue, 07 Apr 2020 10:07:53 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id DB7B618A8C94; Tue, 7 Apr 2020 14:07:52 +0000 (UTC) Received: from virtlab701.virt.lab.eng.bos.redhat.com (virtlab701.virt.lab.eng.bos.redhat.com [10.19.152.228]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5341A5C1BB; Tue, 7 Apr 2020 14:07:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1586268475; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XRLWXRytzTGVO0r/GudLCTGjNoCUKmXsKyZKPig2BiY=; b=MQ6u0D3Y/BJmx5f7Ahjl7hoJR1NzXtXq3nVmgJDIS2lJb7+h+5ct9Z2RUGxQopG95GM2Jx 3RzJl7Ke8jT1x88v5uDqoaI9jW/L3LWTMgCnly4b+Yy3x0P6k6EhKFyZvFdVkfXJCcbiDo DjQPewgkNhPETW5Ds5cAFx8dLuukKIQ= X-MC-Unique: xn0n26WHNKC0s4Vj2PZjmA-1 From: Paolo Bonzini To: qemu-devel@nongnu.org Subject: [PATCH 3/5] rcu: do not mention atomic_mb_read/set in documentation Date: Tue, 7 Apr 2020 10:07:44 -0400 Message-Id: <20200407140746.8041-4-pbonzini@redhat.com> In-Reply-To: <20200407140746.8041-1-pbonzini@redhat.com> References: <20200407140746.8041-1-pbonzini@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 205.139.110.61 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fangying1@huawei.com, stefanha@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Signed-off-by: Paolo Bonzini Reviewed-by: Richard Henderson --- docs/devel/rcu.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/devel/rcu.txt b/docs/devel/rcu.txt index d83fed2f79..0ce15ba198 100644 --- a/docs/devel/rcu.txt +++ b/docs/devel/rcu.txt @@ -132,7 +132,7 @@ The core RCU API is small: =20 typeof(*p) atomic_rcu_read(p); =20 - atomic_rcu_read() is similar to atomic_mb_read(), but it makes + atomic_rcu_read() is similar to atomic_load_acquire(), but it makes some assumptions on the code that calls it. This allows a more optimized implementation. =20 @@ -154,7 +154,7 @@ The core RCU API is small: =20 void atomic_rcu_set(p, typeof(*p) v); =20 - atomic_rcu_set() is also similar to atomic_mb_set(), and it also + atomic_rcu_set() is similar to atomic_store_release(), though it a= lso makes assumptions on the code that calls it in order to allow a mo= re optimized implementation. =20 --=20 2.18.2 From nobody Sun May 19 06:04:49 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1586268537; cv=none; d=zohomail.com; s=zohoarc; b=kTd7Im0KSF7U3yL5SA55HSXf68Im6UdUPB3g/elkwtbLCJpEsvRuLAwZjz+cxn3FprYbaH+lNb4OwXjXWLmL44/hXv0+vAu7EiH7JhMDigoBfOs05ObrSwmBwsAJIWgSBAvZzAqN1fqZ4aATEbJkje/x9C+mf5S56+L5GDkCYkY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1586268537; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=2NlGDHsxs6NB6PgDud3ZxG1DzThL5NOMW87BbQ97wik=; b=TQ8zu5p+7hinvwk9t3ZJOFbBAXMQQAGpBwXmYeRrJgJWuShzbW4SngI5oRTfjupzmW12d1ZUGqfCN19Degp7CQN78dw09KiHtU0TDSHE+2DEuskBjZ85pghpk79sNPKt0QqdIJ9aesAL89VqVsAwnl+Dga/Fw/iKITW8UBVIcDg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1586268537876505.72563573979323; Tue, 7 Apr 2020 07:08:57 -0700 (PDT) Received: from localhost ([::1]:47874 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jLouO-0002DU-KM for importer@patchew.org; Tue, 07 Apr 2020 10:08:56 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51190) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jLotS-0000Z2-He for qemu-devel@nongnu.org; Tue, 07 Apr 2020 10:07:59 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jLotQ-00013s-WC for qemu-devel@nongnu.org; Tue, 07 Apr 2020 10:07:58 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:39940 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1jLotQ-00013X-S8 for qemu-devel@nongnu.org; Tue, 07 Apr 2020 10:07:56 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-357-Q-8qwBxMMaC9j52qItvfGQ-1; Tue, 07 Apr 2020 10:07:54 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 99FCE1005509; Tue, 7 Apr 2020 14:07:53 +0000 (UTC) Received: from virtlab701.virt.lab.eng.bos.redhat.com (virtlab701.virt.lab.eng.bos.redhat.com [10.19.152.228]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0D5755C1BB; Tue, 7 Apr 2020 14:07:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1586268476; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2NlGDHsxs6NB6PgDud3ZxG1DzThL5NOMW87BbQ97wik=; b=e/EAs9Tb+kg/pay/IC4VvUFanwrSj230ZQ0+YZ6I6ccq2fZM8asOhxoBfvK3j55VwRi52d X/7j11rTDmXJnIOP29f63x1a9aLilPM1PurY3ZJwVH3DNEqkulLWIWD5I2jV7GEDF6qvFz EZz5AmCIFFCJqzKwMcKh1igXp9kVoJA= X-MC-Unique: Q-8qwBxMMaC9j52qItvfGQ-1 From: Paolo Bonzini To: qemu-devel@nongnu.org Subject: [PATCH 4/5] aio-wait: delegate polling of main AioContext if BQL not held Date: Tue, 7 Apr 2020 10:07:45 -0400 Message-Id: <20200407140746.8041-5-pbonzini@redhat.com> In-Reply-To: <20200407140746.8041-1-pbonzini@redhat.com> References: <20200407140746.8041-1-pbonzini@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 207.211.31.120 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fangying1@huawei.com, stefanha@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Any thread that is not a iothread returns NULL for qemu_get_current_aio_con= text(). As a result, it would also return true for in_aio_context_home_thread(qemu_get_aio_context()), causing AIO_WAIT_WHILE to invoke aio_poll() directly. This is incorrect if the BQL is not held, because aio_poll() does not expect to run concurrently from multiple threads, and it can actually happen when savevm writes to the vmstate file from the migration thread. Therefore, restrict in_aio_context_home_thread to return true for the main AioContext only if the BQL is held. The function is moved to aio-wait.h because it is mostly used there and to avoid a circular reference between main-loop.h and block/aio.h. Signed-off-by: Paolo Bonzini --- include/block/aio-wait.h | 22 ++++++++++++++++++++++ include/block/aio.h | 29 ++++++++++------------------- 2 files changed, 32 insertions(+), 19 deletions(-) diff --git a/include/block/aio-wait.h b/include/block/aio-wait.h index afeeb18f95..716d2639df 100644 --- a/include/block/aio-wait.h +++ b/include/block/aio-wait.h @@ -26,6 +26,7 @@ #define QEMU_AIO_WAIT_H =20 #include "block/aio.h" +#include "qemu/main-loop.h" =20 /** * AioWait: @@ -124,4 +125,25 @@ void aio_wait_kick(void); */ void aio_wait_bh_oneshot(AioContext *ctx, QEMUBHFunc *cb, void *opaque); =20 +/** + * in_aio_context_home_thread: + * @ctx: the aio context + * + * Return whether we are running in the thread that normally runs @ctx. N= ote + * that acquiring/releasing ctx does not affect the outcome, each AioConte= xt + * still only has one home thread that is responsible for running it. + */ +static inline bool in_aio_context_home_thread(AioContext *ctx) +{ + if (ctx =3D=3D qemu_get_current_aio_context()) { + return true; + } + + if (ctx =3D=3D qemu_get_aio_context()) { + return qemu_mutex_iothread_locked(); + } else { + return false; + } +} + #endif /* QEMU_AIO_WAIT_H */ diff --git a/include/block/aio.h b/include/block/aio.h index cb1989105a..62ed954344 100644 --- a/include/block/aio.h +++ b/include/block/aio.h @@ -133,12 +133,16 @@ struct AioContext { AioHandlerList deleted_aio_handlers; =20 /* Used to avoid unnecessary event_notifier_set calls in aio_notify; - * accessed with atomic primitives. If this field is 0, everything - * (file descriptors, bottom halves, timers) will be re-evaluated - * before the next blocking poll(), thus the event_notifier_set call - * can be skipped. If it is non-zero, you may need to wake up a - * concurrent aio_poll or the glib main event loop, making - * event_notifier_set necessary. + * only written from the AioContext home thread, or under the BQL in + * the case of the main AioContext. However, it is read from any + * thread so it is still accessed with atomic primitives. + * + * If this field is 0, everything (file descriptors, bottom halves, + * timers) will be re-evaluated before the next blocking poll() or + * io_uring wait; therefore, the event_notifier_set call can be + * skipped. If it is non-zero, you may need to wake up a concurrent + * aio_poll or the glib main event loop, making event_notifier_set + * necessary. * * Bit 0 is reserved for GSource usage of the AioContext, and is 1 * between a call to aio_ctx_prepare and the next call to aio_ctx_chec= k. @@ -681,19 +685,6 @@ void aio_co_enter(AioContext *ctx, struct Coroutine *c= o); */ AioContext *qemu_get_current_aio_context(void); =20 -/** - * in_aio_context_home_thread: - * @ctx: the aio context - * - * Return whether we are running in the thread that normally runs @ctx. N= ote - * that acquiring/releasing ctx does not affect the outcome, each AioConte= xt - * still only has one home thread that is responsible for running it. - */ -static inline bool in_aio_context_home_thread(AioContext *ctx) -{ - return ctx =3D=3D qemu_get_current_aio_context(); -} - /** * aio_context_setup: * @ctx: the aio context --=20 2.18.2 From nobody Sun May 19 06:04:49 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1586268958; cv=none; d=zohomail.com; s=zohoarc; b=ZqBtoS7fKQEQpJXLMKdvCzH4jy/i7WbdqQh9c5qvJ1Dx4GKvntrLhYzvmsvbxFsjesyujwsiFcWrk6s9eTV1vvQ/45g9A3I35PUnIdyDSiosMi26EAabO+vfMWCDyjmciB+2nUbMN57HPZ5unKq1GnBLchyprQwe+QcF4QMUo+Y= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1586268958; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=20aiP+/dA+EblNQZt/yrEm2C26ewENAgFUQO4Z58Jv0=; b=AbfrsLgZcZBrR+/kFx3hrwq6pIsM+wQQSutPE0FeVEVRJ4lBFHD7KoQkZdi3vSDU620BguBpW3LvE47G7zyaZxk2bwFBFEjzZ+YS5FzedLcPO0UK/PpidmCdSTePr/sqEtUnO7MwfjrkoWOLfVNGJK/6GZyDuaW/Rlvho8DFOD4= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1586268958438989.0309970104242; Tue, 7 Apr 2020 07:15:58 -0700 (PDT) Received: from localhost ([::1]:47960 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jLp17-0006sP-Oc for importer@patchew.org; Tue, 07 Apr 2020 10:15:53 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52696) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jLozX-0006Kb-8i for qemu-devel@nongnu.org; Tue, 07 Apr 2020 10:14:17 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jLozV-0005cj-OF for qemu-devel@nongnu.org; Tue, 07 Apr 2020 10:14:14 -0400 Received: from us-smtp-1.mimecast.com ([205.139.110.61]:50366 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1jLozV-0005bu-L5 for qemu-devel@nongnu.org; Tue, 07 Apr 2020 10:14:13 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-428-LeA1f9ieNpWWN8KL4EM4Rw-1; Tue, 07 Apr 2020 10:07:55 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 579C2107B7D4; Tue, 7 Apr 2020 14:07:54 +0000 (UTC) Received: from virtlab701.virt.lab.eng.bos.redhat.com (virtlab701.virt.lab.eng.bos.redhat.com [10.19.152.228]) by smtp.corp.redhat.com (Postfix) with ESMTP id BE91F5C1BB; Tue, 7 Apr 2020 14:07:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1586268852; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=20aiP+/dA+EblNQZt/yrEm2C26ewENAgFUQO4Z58Jv0=; b=NQcYtyinQ+/0ZJ3jrPCVKuKF3V5uuPmETKAzG2/jilLpo1Mn7i4JZk0ihzxkmKe7nSGfnV /iaH0VKU8w0WmpNEigmBtI3TIQYi0mcdHFv63P2x5PR8xLRP2soFPQpBSvvgknVCASsU+c qNk08ru0eeGuMkoTZg6ovvPBWb4A88U= X-MC-Unique: LeA1f9ieNpWWN8KL4EM4Rw-1 From: Paolo Bonzini To: qemu-devel@nongnu.org Subject: [PATCH 5/5] async: use explicit memory barriers Date: Tue, 7 Apr 2020 10:07:46 -0400 Message-Id: <20200407140746.8041-6-pbonzini@redhat.com> In-Reply-To: <20200407140746.8041-1-pbonzini@redhat.com> References: <20200407140746.8041-1-pbonzini@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 205.139.110.61 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fangying1@huawei.com, stefanha@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" When using C11 atomics, non-seqcst reads and writes do not participate in the total order of seqcst operations. In util/async.c and util/aio-posi= x.c, in particular, the pattern that we use write ctx->notify_me write bh->scheduled read bh->scheduled read ctx->notify_me if !bh->scheduled, sleep if ctx->notify_me, notify needs to use seqcst operations for both the write and the read. In general this is something that we do not want, because there can be many sources that are polled in addition to bottom halves. The alternative is to place a seqcst memory barrier between the write and the read. This also comes with a disadvantage, in that the memory barrier is implicit on strongly-ordered architectures and it wastes a few dozen clock cycles. Fortunately, ctx->notify_me is never written concurrently by two threads, so we can assert that and relax the writes to ctx->notify_me. The resulting solution works and performs well on both aarch64 and x86. Note that the atomic_set/atomic_read combination is not an atomic read-modify-write, and therefore it is even weaker than C11 ATOMIC_RELAXED; on x86, ATOMIC_RELAXED compiles to a locked operation. Analyzed-by: Ying Fang Signed-off-by: Paolo Bonzini Tested-by: Ying Fang --- util/aio-posix.c | 16 ++++++++++++++-- util/aio-win32.c | 17 ++++++++++++++--- util/async.c | 16 ++++++++++++---- 3 files changed, 40 insertions(+), 9 deletions(-) diff --git a/util/aio-posix.c b/util/aio-posix.c index cd6cf0a4a9..c3613d299e 100644 --- a/util/aio-posix.c +++ b/util/aio-posix.c @@ -559,6 +559,11 @@ bool aio_poll(AioContext *ctx, bool blocking) int64_t timeout; int64_t start =3D 0; =20 + /* + * There cannot be two concurrent aio_poll calls for the same AioConte= xt (or + * an aio_poll concurrent with a GSource prepare/check/dispatch callba= ck). + * We rely on this below to avoid slow locked accesses to ctx->notify_= me. + */ assert(in_aio_context_home_thread(ctx)); =20 /* aio_notify can avoid the expensive event_notifier_set if @@ -569,7 +574,13 @@ bool aio_poll(AioContext *ctx, bool blocking) * so disable the optimization now. */ if (blocking) { - atomic_add(&ctx->notify_me, 2); + atomic_set(&ctx->notify_me, atomic_read(&ctx->notify_me) + 2); + /* + * Write ctx->notify_me before computing the timeout + * (reading bottom half flags, etc.). Pairs with + * smp_mb in aio_notify(). + */ + smp_mb(); } =20 qemu_lockcnt_inc(&ctx->list_lock); @@ -590,7 +601,8 @@ bool aio_poll(AioContext *ctx, bool blocking) } =20 if (blocking) { - atomic_sub(&ctx->notify_me, 2); + /* Finish the poll before clearing the flag. */ + atomic_store_release(&ctx->notify_me, atomic_read(&ctx->notify_me)= - 2); aio_notify_accept(ctx); } =20 diff --git a/util/aio-win32.c b/util/aio-win32.c index a23b9c364d..729d533faf 100644 --- a/util/aio-win32.c +++ b/util/aio-win32.c @@ -321,6 +321,12 @@ bool aio_poll(AioContext *ctx, bool blocking) int count; int timeout; =20 + /* + * There cannot be two concurrent aio_poll calls for the same AioConte= xt (or + * an aio_poll concurrent with a GSource prepare/check/dispatch callba= ck). + * We rely on this below to avoid slow locked accesses to ctx->notify_= me. + */ + assert(in_aio_context_home_thread(ctx)); progress =3D false; =20 /* aio_notify can avoid the expensive event_notifier_set if @@ -331,7 +337,13 @@ bool aio_poll(AioContext *ctx, bool blocking) * so disable the optimization now. */ if (blocking) { - atomic_add(&ctx->notify_me, 2); + atomic_set(&ctx->notify_me, atomic_read(&ctx->notify_me) + 2); + /* + * Write ctx->notify_me before computing the timeout + * (reading bottom half flags, etc.). Pairs with + * smp_mb in aio_notify(). + */ + smp_mb(); } =20 qemu_lockcnt_inc(&ctx->list_lock); @@ -364,8 +376,7 @@ bool aio_poll(AioContext *ctx, bool blocking) ret =3D WaitForMultipleObjects(count, events, FALSE, timeout); if (blocking) { assert(first); - assert(in_aio_context_home_thread(ctx)); - atomic_sub(&ctx->notify_me, 2); + atomic_store_release(&ctx->notify_me, atomic_read(&ctx->notify= _me) - 2); aio_notify_accept(ctx); } =20 diff --git a/util/async.c b/util/async.c index b94518b948..3165a28f2f 100644 --- a/util/async.c +++ b/util/async.c @@ -249,7 +249,14 @@ aio_ctx_prepare(GSource *source, gint *timeout) { AioContext *ctx =3D (AioContext *) source; =20 - atomic_or(&ctx->notify_me, 1); + atomic_set(&ctx->notify_me, atomic_read(&ctx->notify_me) | 1); + + /* + * Write ctx->notify_me before computing the timeout + * (reading bottom half flags, etc.). Pairs with + * smp_mb in aio_notify(). + */ + smp_mb(); =20 /* We assume there is no timeout already supplied */ *timeout =3D qemu_timeout_ns_to_ms(aio_compute_timeout(ctx)); @@ -268,7 +275,8 @@ aio_ctx_check(GSource *source) QEMUBH *bh; BHListSlice *s; =20 - atomic_and(&ctx->notify_me, ~1); + /* Finish computing the timeout before clearing the flag. */ + atomic_store_release(&ctx->notify_me, atomic_read(&ctx->notify_me) & ~= 1); aio_notify_accept(ctx); =20 QSLIST_FOREACH_RCU(bh, &ctx->bh_list, next) { @@ -411,10 +419,10 @@ LuringState *aio_get_linux_io_uring(AioContext *ctx) void aio_notify(AioContext *ctx) { /* Write e.g. bh->scheduled before reading ctx->notify_me. Pairs - * with atomic_or in aio_ctx_prepare or atomic_add in aio_poll. + * with smp_mb in aio_ctx_prepare or aio_poll. */ smp_mb(); - if (ctx->notify_me) { + if (atomic_read(&ctx->notify_me)) { event_notifier_set(&ctx->notifier); atomic_mb_set(&ctx->notified, true); } --=20 2.18.2