From nobody Sat Feb 14 07:06:11 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9F92C1422B9 for ; Wed, 15 May 2024 14:31:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715783489; cv=none; b=BYdsna/ZIhG6nD5s+dPonnhdE0IFE+vXBAKJ5FQ6EtjWToRdnfI+N+zuFw6siVTn2edxX2ZJ892br60wJVTqxokIUYVRNb2ghAcKx1mpoqlk49udnHR4hKZT7OkIQGjCQjAxGrdWetDJntR4g1/gBwyE0vo4k0E3WIsH0ad7E/A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715783489; c=relaxed/simple; bh=knddn969BjEL1cGIM/hhNec75ASOhQ8q9F478YWF4D0=; h=From:To:Cc:Subject:Date:Message-Id:MIME-Version; b=psaGMC1iAkduBMHt2yGuKnZ9QM6dNYOEVj6NrR+pXb0fJZe/PNrS+JUzU4tTBr82SCStmblwcPz8MgIROSryR5NmTZkiuQ7QZ3nPnE0WiOFR1tfMx7i4Nsstt0TtxdFgHfzFv9poWo1Adw8QV6kDggklP/bDPQfZ673EyhVDdJI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=JLQbEqrK; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="JLQbEqrK" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1715783485; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=6D0kM1W2IVqoV/ykSNuolWPIzDpABZdm4eW8Ou5+aG0=; b=JLQbEqrKiAj0WZ/uqSyyVfxzkXZ0crbitbdapuTnZ+XvzcE5INLCOjPBAqSRBG+YuGmAo3 rJR2AcbvuObQgdAwc6wXLiD8OZbt/JJ9i4h0NDh2J0wGQnnmRfqIgnExNgTxqFJ39vrLYZ +tRmrnsHgOWO1YC5IBcMQee9RSQi8Lw= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-135-kQSitauyNaqCbg0FNlS48w-1; Wed, 15 May 2024 10:31:24 -0400 X-MC-Unique: kQSitauyNaqCbg0FNlS48w-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8110B8016FA; Wed, 15 May 2024 14:31:23 +0000 (UTC) Received: from llong.com (unknown [10.22.33.50]) by smtp.corp.redhat.com (Postfix) with ESMTP id 91A67400057; Wed, 15 May 2024 14:31:22 +0000 (UTC) From: Waiman Long To: Jens Axboe , Tejun Heo , Josef Bacik Cc: cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Dan Schatzberg , Ming Lei , Waiman Long Subject: [PATCH] blk-cgroup: Properly propagate the iostat update up the hierarchy Date: Wed, 15 May 2024 10:30:59 -0400 Message-Id: <20240515143059.276677-1-longman@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.9 Content-Type: text/plain; charset="utf-8" During a cgroup_rstat_flush() call, the lowest level of nodes are flushed first before their parents. Since commit 3b8cc6298724 ("blk-cgroup: Optimize blkcg_rstat_flush()"), iostat propagation was still done to the parent. Grandparent, however, may not get the iostat update if the parent has no blkg_iostat_set queued in its lhead lockless list. Fix this iostat propagation problem by queuing the parent's global blkg->iostat into one of its percpu lockless lists to make sure that the delta will always be propagated up to the grandparent and so on toward the root blkcg. Note that successive calls to __blkcg_rstat_flush() are serialized by the cgroup_rstat_lock. So no special barrier is used in the reading and writing of blkg->iostat.lqueued. Fixes: 3b8cc6298724 ("blk-cgroup: Optimize blkcg_rstat_flush()") Reported-by: Dan Schatzberg Closes: https://lore.kernel.org/lkml/ZkO6l%2FODzadSgdhC@dschatzberg-fedora-= PF3DHTBV/ Signed-off-by: Waiman Long Acked-by: Tejun Heo Reviewed-by: Ming Lei --- block/blk-cgroup.c | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index 059467086b13..2a7624c32a1a 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -323,6 +323,7 @@ static struct blkcg_gq *blkg_alloc(struct blkcg *blkcg,= struct gendisk *disk, blkg->q =3D disk->queue; INIT_LIST_HEAD(&blkg->q_node); blkg->blkcg =3D blkcg; + blkg->iostat.blkg =3D blkg; #ifdef CONFIG_BLK_CGROUP_PUNT_BIO spin_lock_init(&blkg->async_bio_lock); bio_list_init(&blkg->async_bios); @@ -1025,6 +1026,8 @@ static void __blkcg_rstat_flush(struct blkcg *blkcg, = int cpu) unsigned int seq; =20 WRITE_ONCE(bisc->lqueued, false); + if (bisc =3D=3D &blkg->iostat) + goto propagate_up; /* propagate up to parent only */ =20 /* fetch the current per-cpu values */ do { @@ -1034,10 +1037,24 @@ static void __blkcg_rstat_flush(struct blkcg *blkcg= , int cpu) =20 blkcg_iostat_update(blkg, &cur, &bisc->last); =20 +propagate_up: /* propagate global delta to parent (unless that's root) */ - if (parent && parent->parent) + if (parent && parent->parent) { blkcg_iostat_update(parent, &blkg->iostat.cur, &blkg->iostat.last); + /* + * Queue parent->iostat to its blkcg's lockless + * list to propagate up to the grandparent if the + * iostat hasn't been queued yet. + */ + if (!parent->iostat.lqueued) { + struct llist_head *plhead; + + plhead =3D per_cpu_ptr(parent->blkcg->lhead, cpu); + llist_add(&parent->iostat.lnode, plhead); + parent->iostat.lqueued =3D true; + } + } } raw_spin_unlock_irqrestore(&blkg_stat_lock, flags); out: --=20 2.39.3