From nobody Fri Aug 21 21:26:59 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=openvz.org ARC-Seal: i=1; a=rsa-sha256; t=1787226801; cv=none; d=zohomail.com; s=zohoarc; b=GHwXkLIfhxQ2gWaw1c/T9L6LK3csAfLvJ3q5A4Q3pEryvzJ1FXeSVw33dXlMR2dSsnIgjQOa0MNQbCQA6UwBTn7RPmjvafZKlDHJsFeZHcA5GQ+b9dHSYeKVHwHurM9nKMqKeMDyt2o4xC253hGtvz8TdNcOSFvpzFPXrlxInFQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1787226801; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=lJGFRZT4xBgbHoDeIFX0KDibwMy8Oj0WwFErBWi550o=; b=OoO2iEjMMS55KA27PmIYFlqCm8PVSKJOPDprMygQjat6euPCm6S9agueIU823IYw7SYNp3s5l5TlXptiV/Mn2YQ7+7Bn5xSqRTNorgkNye7APthLjjYo4woZz+EgXek9k38djzvjCD9i7zx6JuXB+v7ILIBj7lFJ7z5rmUYYj6U= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1787226801296640.6284839836653; Thu, 20 Aug 2026 04:53:21 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wx1K2-00087q-AM; Thu, 20 Aug 2026 07:52:38 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wx1K0-00086m-9l for qemu-devel@nongnu.org; Thu, 20 Aug 2026 07:52:36 -0400 Received: from mail-wr1-x42c.google.com ([2a00:1450:4864:20::42c]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wx1Jy-0005p0-9y for qemu-devel@nongnu.org; Thu, 20 Aug 2026 07:52:36 -0400 Received: by mail-wr1-x42c.google.com with SMTP id ffacd0b85a97d-47f93b2fe4cso1229224f8f.0 for ; Thu, 20 Aug 2026 04:52:33 -0700 (PDT) Received: from athena.sw.ru ([2a06:5b06:b600:300:a123:7b43:afd8:8b47]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-482b10fcebdsm13069627f8f.0.2026.08.20.04.52.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Aug 2026 04:52:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=openvz.org; s=google; t=1787226753; x=1787831553; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=lJGFRZT4xBgbHoDeIFX0KDibwMy8Oj0WwFErBWi550o=; b=oj6kco/7STE/U3TOOQSnvXJp8K7OtlXApQispR7fkz4N4PXVfnBPigZ2WSkx3m61Xc vIH+5S55Koy9qrFZFE7Onrp2OCX6MKAARzYZWNLEblh/enbkrfY4VucSK6JsLw4JcmcH ZEPVBFi3MqsAEA7P8lIUEsqROBlwVsSfg9IPKW3D3w/NDUxpegfPUIGjcJGMHdYJ7w4q nzrrLVR6lJYvayrpajBRJBPpmmZvnjEXRt6UWTcOUzq4rFDzonWtijUvpIhIGdaw8m73 NJtnSJ4eFuxNWJ8JOSvh9khlxkVitMhGD420/jQY6DGZiJYnJuZuvOlyv7n6CQ+hH7Tw nIMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787226753; x=1787831553; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=lJGFRZT4xBgbHoDeIFX0KDibwMy8Oj0WwFErBWi550o=; b=rwLmbaRMvsW+cTscxp7fR7VjwOszNin4cgjBSl0/JaVhvM8PUzCYl4ro/4NS7JgYTF V+OWPWdvydhdQjgJosLDDRbc3qprfyLAEKsfAw1DCaETG50dP/3wf/NljjQctQ2W6fRe mewPk/LjUnxwJ7PfARobeuG3rVQqHrSp1LTQBYYLahpcaeSflEs9lIVK+nPK+lFNlPUw YoPL2fwu5I/l8GcblEDyaL97RwubStZueDgMiIUYkdCd++4g3KLUAowJaMaUSnAgfeGu dsvLrZ82GjxO+Paewg2S3lbCV4xYpvn6gOiDQNkXagwFDNzzRwen0iIfQ4TE+2gxuj7P CMpA== X-Gm-Message-State: AFuF++m3v7+KBDb9MxIKNiPj7A75m562LIHskMZWL4TtK2xeCylguRmq gdOlNc4TswjfQZ2TTda3mWvLe++5pioQ3UP15Je9hb1NdaAwejaHgW0MlRlrTvt5o+llM0FGT0M XTlYe X-Gm-Gg: AR+sD11vYh+EUL6eJM95XCKOD+pkNTisLgLD4t2RmCZMJBjZW0dlHVRkPhtp0hA3dqr j6d6Loh4Dz4NxyO8pZtaLJWyONhJ+2Uy/cvhIa6eOKKcBgfV/sBsvn8d2v7J8na9CewszKtU8ey 6NtyHQiQ0Qoh8V7+p/iPGs43tfLBEXl4dvsRNZ8fFPXHtxrdQ1DEYcrbWDkftjVAw9h8/Hr1s+m b5COAeNxAMyFvNfIym08qBtGAPQtLitUVrqgQQ6C1t0WNfYbYC9iLYVC4hri3b++M6I9GHADeBb d2kQRLaln9y4YorNoL0jRLT9P7dZlPuvmbzAn1qdGUemSPzWaiT7yZIpjOlL92SXttNpm2bRtsh X928dW5IjF689qaFbuK7r37ihtgJ7pP3BZJpzpGTzqeyFts1Tz8PAuyZE/dQxLFAQIpk0/7xmTP GqirUrEOV0b8yvHsA0Ha7Viy6vNjNAh/IyCOofSme0zbap0vOqFNnlLFEaQXrT2xVnv8tg X-Received: by 2002:a05:6000:298e:20b0:482:b887:f033 with SMTP id ffacd0b85a97d-482b887f05dmr6991049f8f.16.1787226752888; Thu, 20 Aug 2026 04:52:32 -0700 (PDT) From: "Denis V. Lunev" To: qemu-devel@nongnu.org Cc: qemu-block@nongnu.org, "Denis V. Lunev" , Eric Blake , Vladimir Sementsov-Ogievskiy Subject: [PATCH v2 1/3] block/nbd: clear reply.cookie when the reply is rejected Date: Thu, 20 Aug 2026 13:52:26 +0200 Message-ID: <20260820115228.587427-2-den@openvz.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260820115228.587427-1-den@openvz.org> References: <20260820115228.587427-1-den@openvz.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::42c; envelope-from=den@openvz.org; helo=mail-wr1-x42c.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @openvz.org) X-ZM-MESSAGEID: 1787226802622158500 Content-Type: text/plain; charset="utf-8" From: Denis V. Lunev nbd_receive_replies() reads a reply header into s->reply and, when the header turns out to be unusable, reports a channel error and returns without touching it. The cookie stays there until the request which owns the reply clears it, and until then the waiters are explicitly allowed to look at a cookie which is not theirs: if (s->reply.cookie !=3D 0) { ind2 =3D COOKIE_TO_INDEX(s->reply.cookie); assert(!s->requests[ind2].receiving); Two of the error paths leave a value chosen by the server behind: one returns before the cookie is validated at all, the other returns because that validation has failed. A waiter which picks such a cookie up turns it into an index which is not in requests[] and accesses the array out of bounds, at an offset the server controls. The reply is of no use to anybody at this point, so clear the cookie before the mutex is released and keep the invariant that a non-zero s->reply.cookie is always an index of a live request. Observing the stale cookie takes a second thread, which a multiqueue configuration provides. Within one AioContext there is no yield point between the failed read and the clearing done by the owner in nbd_co_receive_one_chunk(), so nothing else of this node runs in between. The parked waiters cannot see it either, as they are woken only after the cookie has been cleared. What can get in is a request entering nbd_receive_replies() afresh, one just sent or one back for its next reply chunk, because that path takes the mutex without looking at the state. Cc: Eric Blake Cc: Vladimir Sementsov-Ogievskiy Signed-off-by: Denis V. Lunev Reviewed-by: Vladimir Sementsov-Ogievskiy --- block/nbd.c | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/block/nbd.c b/block/nbd.c index 5d231d5c4e..d9b776283f 100644 --- a/block/nbd.c +++ b/block/nbd.c @@ -466,20 +466,19 @@ static coroutine_fn int nbd_receive_replies(BDRVNBDSt= ate *s, uint64_t cookie, error_setg(errp, "server dropped connection"); } if (ret < 0) { - nbd_channel_error(s, ret); - return ret; + goto err; } if (nbd_reply_is_structured(&s->reply) && s->info.mode < NBD_MODE_STRUCTURED) { - nbd_channel_error(s, -EINVAL); + ret =3D -EINVAL; error_setg(errp, "unexpected structured reply"); - return -EINVAL; + goto err; } ind2 =3D COOKIE_TO_INDEX(s->reply.cookie); if (ind2 >=3D MAX_NBD_REQUESTS || !s->requests[ind2].coroutine) { - nbd_channel_error(s, -EINVAL); + ret =3D -EINVAL; error_setg(errp, "unexpected cookie value"); - return -EINVAL; + goto err; } if (s->reply.cookie =3D=3D cookie) { /* We are done */ @@ -487,6 +486,13 @@ static coroutine_fn int nbd_receive_replies(BDRVNBDSta= te *s, uint64_t cookie, } nbd_recv_coroutine_wake_one(&s->requests[ind2]); } + +err: + /* Waiters look at this cookie, so do not leave a rejected one behind.= */ + s->reply.cookie =3D 0; + nbd_channel_error(s, ret); + + return ret; } =20 static int coroutine_fn GRAPH_RDLOCK --=20 2.53.0 From nobody Fri Aug 21 21:26:59 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=openvz.org ARC-Seal: i=1; a=rsa-sha256; t=1787226801; cv=none; d=zohomail.com; s=zohoarc; b=ckKy+cAQUbtXUFnzJ40fTXU1ycFK+Paw3cgf9FHBg/h1XjeacXbwE1HjdMtlRlenC38OqUGhBEHojXfx//x34YGukz+iRAcsj3KOB1OGkMJd1iq7c7OcOFB56gV0asdeIXRwHH1UPT7Zia3/KudnSnlDRv3T68CZYcB+sxO9ZQc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1787226801; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=st0WgwSEv4Yi/cZ0YrOYJwHq5izpY4g4nImWv1PkfRw=; b=WmI7i+f6NMzuyKEhZCNML0LjiOWpRILv50oiSZFYgcCKPNaPyLowZN25L1bVc+Q8lmi63Nt7hV907/Phdq3nrIl8r8x0+MKOyzy2685crIR/NuXjerYps/Xb5jY01N2DJigAahiO2unnvcPmeJiGoNjhi+cv7whWP+tdyckHEWg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1787226801173198.54411251808335; Thu, 20 Aug 2026 04:53:21 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wx1K2-00088K-SL; Thu, 20 Aug 2026 07:52:38 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wx1K1-00087A-A4 for qemu-devel@nongnu.org; Thu, 20 Aug 2026 07:52:37 -0400 Received: from mail-wr1-x436.google.com ([2a00:1450:4864:20::436]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wx1Jz-0005pR-Fj for qemu-devel@nongnu.org; Thu, 20 Aug 2026 07:52:37 -0400 Received: by mail-wr1-x436.google.com with SMTP id ffacd0b85a97d-47f59f25ec4so1123627f8f.2 for ; Thu, 20 Aug 2026 04:52:35 -0700 (PDT) Received: from athena.sw.ru ([2a06:5b06:b600:300:a123:7b43:afd8:8b47]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-482b10fcebdsm13069627f8f.0.2026.08.20.04.52.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Aug 2026 04:52:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=openvz.org; s=google; t=1787226754; x=1787831554; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=st0WgwSEv4Yi/cZ0YrOYJwHq5izpY4g4nImWv1PkfRw=; b=VapBiy6yLxM24f+Ibh/qS+4KHY9LiPWCfAWzQRcIOmaPFBvSybB+LSGUqJ41EJlhek pf+9todxRm2pvpQCaIlSqmd/gfP0fEGCtNccoA7J3F4HZZGO9aCiyqo25Jzjv3QH4xjX duWuniuwERrCKypwMBvDfsohUWmJGug9TcgNOfMG1A85IftKD1m3PSt4FhRVCl+jlE4/ 1AUW8ILd+G+VxwsWfydG2l1B5W3Tum01cC0Bk/QXJJc6o/y42cH8E+mHHe/Lu93Y/svB 3JscUQjauRWp/AHOg1D+Aa9LMreLAhYgZWTK5r0NMss98sG96P9U9caPf3h1yG7ZWgKG QVUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787226754; x=1787831554; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=st0WgwSEv4Yi/cZ0YrOYJwHq5izpY4g4nImWv1PkfRw=; b=PkODc99pnJxaeGsIniEmacYFdsYSqpH1esLygPzgZXSa2cigX9YdWtiraU1XH87cgy nHHmdeN+6ao56o3ENQqyJc5Bc7+wcHVWD0Z4QMMPahCzrjeCZKNSdCOD94tNZo5C3xFg jmiswm/4yVi79BvGDMTtwhIoH7H/e8qizUFM+Ht7DVMMtgtYF7oyNup1KktPb+NfPqC4 747ZxynsPviJ5Zrb2k5V4IylhMHDUqz9NUVhdFPE3nJU0lyaLvBMWsu481CfRabIwmaE LWc78M9Wi0MPOn3pWQQ1bBoN5u6cufPeVpy8oUBnVh0Uvi0z5ljgrJZSBq8AgpJEKCEX xsKg== X-Gm-Message-State: AOJu0YxbSaY9BKXFxOiAsh7kUpixEuP2uUmZ3Hn7fNnVhv6kB3dvGVMs 53kiSwMVahHa59BUYrHfCn3g2QmcFU2JguEf71ctJudtWOSrKu8LSZR7zcBnTkXvIJWjAVRhmGt 3ErPL X-Gm-Gg: AR+sD10NLm6CxxXaYwFfqE10/w3DVo6NULAZXAILC0AVkLV+rO/sipf/xO43iKj0GPZ HMkL3kMTNrR8NrTKqB6pLNeFwGmFEGS8MyWKou09nZrOziVrvhjBdCIlQQAPSNP8jVdgNcIUmJB Sv33/SMrl9odL78wVMADAV/hE2gAWjGgGPxHFDghhGn8zNXWaj/i+VH6jHK2jc+cTSiLtj+fsae BbhLEBY+hZRLF1OwtbImpf/Jkos+k3c8it+paH1556GJLeUpzMtkuTiWvSbhaR0xTMQOwcYL6i0 2zbS7xvCBq/M3DIMF3P1pci7ZhQ1+UIrUejiokpqkvoDts6Rdy4hApP9NaKA/9REOM54LErQb4c fz4+7pLg9LhqfM8FY61GjWVafGK7Brud5PgL8Dn2+ykrR5+gcjR9jqen5XvLTOsKn+wdDaqJSWA L9xOx1Nynx3pfuo+gQC//svLwVEdttaEMqKFhFAHMqssgNEkIdBPfTiJh26Q== X-Received: by 2002:a05:600c:19c7:b0:495:7a04:b006 with SMTP id 5b1f17b1804b1-499aa192422mr181214115e9.8.1787226754015; Thu, 20 Aug 2026 04:52:34 -0700 (PDT) From: "Denis V. Lunev" To: qemu-devel@nongnu.org Cc: qemu-block@nongnu.org, "Denis V. Lunev" , Eric Blake , Vladimir Sementsov-Ogievskiy Subject: [PATCH v2 2/3] block/nbd: never index requests[] with an unchecked cookie Date: Thu, 20 Aug 2026 13:52:27 +0200 Message-ID: <20260820115228.587427-3-den@openvz.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260820115228.587427-1-den@openvz.org> References: <20260820115228.587427-1-den@openvz.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::436; envelope-from=den@openvz.org; helo=mail-wr1-x436.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @openvz.org) X-ZM-MESSAGEID: 1787226802635158500 Content-Type: text/plain; charset="utf-8" From: Denis V. Lunev Cookies are converted into indices of s->requests[] in several places and the result is used right away, without any check: int i =3D COOKIE_TO_INDEX(cookie); ... return nbd_co_receive_offset_data_payload(s, s->requests[i].offset, COOKIE_TO_INDEX() subtracts one, so a zero cookie becomes an index of -1 and the access lands in front of the array. This is undefined and, depending on the type of the index and on what the compiler has put there, it can as well pass silently: at the site above i is signed, so even a plain i < MAX_NBD_REQUESTS check would happily let -1 through. Route every conversion through a helper which hands out the slot only for a cookie in range and owned by a request in flight, so that the check can not be forgotten again. The helper reports through an Error, as the one cookie which is not ours to trust, the one taken from the wire in nbd_receive_replies(), is a protocol error rather than an internal inconsistency and has to stay a channel error. Every other cookie is one we have issued and still own, so a bad value there is a bug in this file, which is what &error_abort spells out. The cookie of the reply in flight goes through the helper as well. It is not an unchecked value either: it has passed the check on the wire path, or it has been cleared by the previous commit. Cc: Eric Blake Cc: Vladimir Sementsov-Ogievskiy Signed-off-by: Denis V. Lunev Reviewed-by: Vladimir Sementsov-Ogievskiy --- block/nbd.c | 39 ++++++++++++++++++++++++++------------- 1 file changed, 26 insertions(+), 13 deletions(-) diff --git a/block/nbd.c b/block/nbd.c index d9b776283f..d0a7097034 100644 --- a/block/nbd.c +++ b/block/nbd.c @@ -136,6 +136,19 @@ static void nbd_clear_bdrvstate(BlockDriverState *bs) s->x_dirty_bitmap =3D NULL; } =20 +static NBDClientRequest *nbd_request_by_cookie(BDRVNBDState *s, uint64_t c= ookie, + Error **errp) +{ + uint64_t ind =3D COOKIE_TO_INDEX(cookie); + + if (ind >=3D MAX_NBD_REQUESTS || !s->requests[ind].coroutine) { + error_setg(errp, "unexpected cookie value"); + return NULL; + } + + return &s->requests[ind]; +} + /* Called with s->receive_mutex taken. */ static bool coroutine_fn nbd_recv_coroutine_wake_one(NBDClientRequest *req) { @@ -422,7 +435,8 @@ static coroutine_fn int nbd_receive_replies(BDRVNBDStat= e *s, uint64_t cookie, Error **errp) { int ret; - uint64_t ind =3D COOKIE_TO_INDEX(cookie), ind2; + NBDClientRequest *req =3D nbd_request_by_cookie(s, cookie, &error_abor= t); + NBDClientRequest *owner; QEMU_LOCK_GUARD(&s->receive_mutex); =20 while (true) { @@ -437,10 +451,10 @@ static coroutine_fn int nbd_receive_replies(BDRVNBDSt= ate *s, uint64_t cookie, * woken by whoever set s->reply.cookie (or never wait in this * yield). So, we should not wake it here. */ - ind2 =3D COOKIE_TO_INDEX(s->reply.cookie); - assert(!s->requests[ind2].receiving); + owner =3D nbd_request_by_cookie(s, s->reply.cookie, &error_abo= rt); + assert(!owner->receiving); =20 - s->requests[ind].receiving =3D true; + req->receiving =3D true; qemu_co_mutex_unlock(&s->receive_mutex); =20 qemu_coroutine_yield(); @@ -454,7 +468,7 @@ static coroutine_fn int nbd_receive_replies(BDRVNBDStat= e *s, uint64_t cookie, */ =20 qemu_co_mutex_lock(&s->receive_mutex); - assert(!s->requests[ind].receiving); + assert(!req->receiving); continue; } =20 @@ -474,17 +488,16 @@ static coroutine_fn int nbd_receive_replies(BDRVNBDSt= ate *s, uint64_t cookie, error_setg(errp, "unexpected structured reply"); goto err; } - ind2 =3D COOKIE_TO_INDEX(s->reply.cookie); - if (ind2 >=3D MAX_NBD_REQUESTS || !s->requests[ind2].coroutine) { + owner =3D nbd_request_by_cookie(s, s->reply.cookie, errp); + if (!owner) { ret =3D -EINVAL; - error_setg(errp, "unexpected cookie value"); goto err; } if (s->reply.cookie =3D=3D cookie) { /* We are done */ return 0; } - nbd_recv_coroutine_wake_one(&s->requests[ind2]); + nbd_recv_coroutine_wake_one(owner); } =20 err: @@ -861,7 +874,6 @@ static coroutine_fn int nbd_co_do_receive_one_chunk( { ERRP_GUARD(); int ret; - int i =3D COOKIE_TO_INDEX(cookie); void *local_payload =3D NULL; NBDStructuredReplyChunk *chunk; =20 @@ -919,8 +931,9 @@ static coroutine_fn int nbd_co_do_receive_one_chunk( return -EINVAL; } =20 - return nbd_co_receive_offset_data_payload(s, s->requests[i].offset, - qiov, errp); + return nbd_co_receive_offset_data_payload( + s, nbd_request_by_cookie(s, cookie, &error_abort)->offset, + qiov, errp); } =20 if (nbd_reply_type_is_error(chunk->type)) { @@ -1067,7 +1080,7 @@ static bool coroutine_fn nbd_reply_chunk_iter_receive= (BDRVNBDState *s, =20 break_loop: qemu_mutex_lock(&s->requests_lock); - s->requests[COOKIE_TO_INDEX(cookie)].coroutine =3D NULL; + nbd_request_by_cookie(s, cookie, &error_abort)->coroutine =3D NULL; s->in_flight--; qemu_co_queue_next(&s->free_sema); qemu_mutex_unlock(&s->requests_lock); --=20 2.53.0 From nobody Fri Aug 21 21:26:59 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=openvz.org ARC-Seal: i=1; a=rsa-sha256; t=1787226829; cv=none; d=zohomail.com; s=zohoarc; b=dS+/NH74kUM5DxKxO54IauJq4tVGRsMNn5HqnSyO0Y+PjfeQqVD9Pg6L/pj+BDjSEgO+xXdMxk1YKhBYR1NTvbXxlBjACrrmQNmoNkYbZ7a25Fs2p5B0ZDYvO8xnqS37jxlRQAISgMdTtNkCHOMVg7d4px7ynhwmG4ZQ0vK+cgc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1787226829; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=GPlAeeOjoY4SZFtVFtu8z+mQmq26G3t8GLazxFHp+fg=; b=VRIvHgHn+mlqJZXP0Wgv1/0WM46IR2ZbSoPO+e+RuBpBVneCLGE9+k+Qhr8dyOZOk7XJCfjKGHU1MEUgSNP1AW539lwa3/Qo3pKvqFTV81pZ+HuY3wo+gSBOir4QQZEULujOBE4POVSUVZ5GgTUqcYzChItTq6O1PwMT0ilOKBs= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1787226829370789.738322722136; Thu, 20 Aug 2026 04:53:49 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wx1K3-000896-Vp; Thu, 20 Aug 2026 07:52:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wx1K2-000883-Kt for qemu-devel@nongnu.org; Thu, 20 Aug 2026 07:52:38 -0400 Received: from mail-wr1-x433.google.com ([2a00:1450:4864:20::433]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wx1K0-0005pz-Uz for qemu-devel@nongnu.org; Thu, 20 Aug 2026 07:52:38 -0400 Received: by mail-wr1-x433.google.com with SMTP id ffacd0b85a97d-47f7027ca11so1318165f8f.3 for ; Thu, 20 Aug 2026 04:52:36 -0700 (PDT) Received: from athena.sw.ru ([2a06:5b06:b600:300:a123:7b43:afd8:8b47]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-482b10fcebdsm13069627f8f.0.2026.08.20.04.52.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Aug 2026 04:52:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=openvz.org; s=google; t=1787226755; x=1787831555; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=GPlAeeOjoY4SZFtVFtu8z+mQmq26G3t8GLazxFHp+fg=; b=n6mEE4D9YMhlAkAcqgPvRHEy+SdJ661YuLiM9Cp5/Ly5UOBIMqwRDjfT8iYp7Nznr9 RUU067Hoxdo7+ckvrNCZenmgUVg+mOIfb3sUBzyPdtzeWIm6aynDO/Vp1UcO3FgJbs9q M3hGjlXkeb5nST3JYO0XUCF/cUpRsOhSYPN9laBxpSkp4rx0ttod+3J6yKz+hwXmUefw ZgO8FV0ZZ4Qxu4POSR74YTglhOkfRbVrvvX2PSH1zN61ouOIYuHo8KGZsenMeiNPCLDj m/uta16V3xn1bizyRlYyIoSkn0bIzsj26m2ium2uM4b0lUfARxkcAmtyOKR4Cug3aVqx okpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787226755; x=1787831555; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=GPlAeeOjoY4SZFtVFtu8z+mQmq26G3t8GLazxFHp+fg=; b=qBjhn0QBnn2HP2gLmzaywotc+SbStvrY2+YJjBfT0EQXL7fcoJD96tBZqF1zMofPlm wd3dp6RDxZ+RJlKQ7i0Uf7ucaIZo5ORQnlpTu1cUu3Cmt2FCfz663GriMP03N9Ca/NBs rOeKypBpHVvvFBuJfYkBXsKq53pIjZlt8iFNkSJ8S6ZLR17mDbRDbpmW8GyFa7Yr/EWj pttkb67w8Ou1E+4sj5EfmAVF9Rl2HhbV7RPesivFSNPdXKa67UpXx8xw/vCwVhCdgRxL oK+YUYRqHXZ0u+qcTqesGvuw+fbnxXsySxTP+6U42B4h1yfHVYXgJRx2n6zYG+WrB4MR 3efA== X-Gm-Message-State: AFuF++kwYGWo5FaCPf/bIRsbBgleiFg3Xw9/iHXs+SIL7CARwrlvZ0Qf cEeCVyQlV7k0ixEV3I6HarrCjUpO7sq5M0dIwrVoLUnhEM1eoK1Rz2+Wu3vQZsFfAENmNtpdcRa G/RwG X-Gm-Gg: AR+sD11JEr8X7YdBL0WMhSIFcOX7qAg6WHAZwSDZAgCmN11VUKFZckQnSUuszF93eEZ g8B1YMc6N8z9saHNiZWPkWalsqBXoMVwtGcpLyv/NVOY0dgl9L9vmpcEB/JjzkvgsVu5b/G6wD4 TkWKsd9J1XYToZNVrUUty9nx94YXdp+Aa9T7bDqN22phUj94lIpOEcGxf1eZthZWoKUimJk1IsS E5yUof8s30mJHpL/iBCuG5xMhn8jNgwIW56UNvJg06s1z6kaOEzh/xPU6xPSLZTLKykgu8U+dib mdtnPhk1rPQkSm22rw6Ux0bw2eu/bwcOOJ+veI/xaECiX/nO/WYWeZP/tI4uDFOUgd5Oq13IC9c DZVGCmZcpD2+04OhD5J5eyA6MqwUb6JRwckW4d/gk3fzcm+X48GELISNLaSKe1p5UGgWGlrx6Fq JLT9OfWjaNBe66CCS+O2GArDZb7WSiUD2ttXWL0OF8/iB/BTxhjixB8is81A== X-Received: by 2002:a05:6000:1884:b0:47f:9662:85fe with SMTP id ffacd0b85a97d-482b1fe8a3dmr19862231f8f.16.1787226755218; Thu, 20 Aug 2026 04:52:35 -0700 (PDT) From: "Denis V. Lunev" To: qemu-devel@nongnu.org Cc: qemu-block@nongnu.org, "Denis V. Lunev" , Eric Blake , Vladimir Sementsov-Ogievskiy Subject: [PATCH v2 3/3] block/nbd: clear reply.cookie under receive_mutex Date: Thu, 20 Aug 2026 13:52:28 +0200 Message-ID: <20260820115228.587427-4-den@openvz.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260820115228.587427-1-den@openvz.org> References: <20260820115228.587427-1-den@openvz.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::433; envelope-from=den@openvz.org; helo=mail-wr1-x433.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @openvz.org) X-ZM-MESSAGEID: 1787226830168158500 Content-Type: text/plain; charset="utf-8" From: Denis V. Lunev s->reply is documented as protected by s->receive_mutex, but the cookie is cleared without it once the owning request has consumed its chunk. A waiter in nbd_receive_replies() inspects the very same field under the mutex, and does so with two separate loads: if (s->reply.cookie !=3D 0) { ind2 =3D COOKIE_TO_INDEX(s->reply.cookie); assert(!s->requests[ind2].receiving); Nothing keeps those two loads consistent. If the owner clears the cookie in between, the second one reads 0, COOKIE_TO_INDEX() turns it into an index of -1, and s->requests[] is accessed out of bounds: Assertion `!s->requests[ind2].receiving' failed. (gdb) p cookie $1 =3D 8 (gdb) p s->reply.cookie $2 =3D 0 (gdb) p &((NBDClientRequest *)s->requests)[-1].receiving $3 =3D (_Bool *) 0x5555558416c0 (gdb) p &s->in_flight $4 =3D (unsigned int *) 0x5555558416c0 (gdb) p s->in_flight $5 =3D 8 The cookie we wait for is 8, yet reply.cookie reads 0 one line after it was found non-zero, so the index is -1. requests[-1].receiving lands on in_flight, which is non-zero while requests are outstanding, and that is what the assertion trips over. Hitting this requires two coroutines of one NBD node to run in different threads, as there is no yield point between the two loads for the owner to squeeze into. A multiqueue configuration provides exactly that, with the virtqueues of one disk spread over several iothreads. Note that a compiler is free to merge the two loads into one, in which case the race is invisible, so builds with reduced optimization are much more likely to trip over it. Accessing s->reply without the mutex is fine for the coroutine that owns the reply: a non-zero cookie makes the field private to it. Releasing that ownership is not, as it races with the waiters which are explicitly allowed to look at the cookie. Clear it under the mutex, in the same critical section as the wakeup, and make nbd_recv_coroutines_wake() caller-locked, as CoMutex is not recursive. It has a single caller. The added acquisition cannot block behind the header read in nbd_receive_replies(), because that path is only reachable with reply.cookie =3D=3D 0 while we still own a non-zero cookie. Merging the clear with the wakeup also keeps a newcomer from starting a header read in between, which would stall this already completed request for the duration of that read. There is no cookie to own when we get here after an error, and then a newcomer can indeed be inside that read. It does not hold us for long either, as the channel has been shut down before the error was reported, so the read it sits in returns right away. Fixes: 4ddb5d2fde ("block/nbd: drop connection_co") Cc: Eric Blake Cc: Vladimir Sementsov-Ogievskiy Signed-off-by: Denis V. Lunev Reviewed-by: Vladimir Sementsov-Ogievskiy --- block/nbd.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/block/nbd.c b/block/nbd.c index d0a7097034..e5e16722ba 100644 --- a/block/nbd.c +++ b/block/nbd.c @@ -161,11 +161,11 @@ static bool coroutine_fn nbd_recv_coroutine_wake_one(= NBDClientRequest *req) return false; } =20 +/* Called with s->receive_mutex taken. */ static void coroutine_fn nbd_recv_coroutines_wake(BDRVNBDState *s) { int i; =20 - QEMU_LOCK_GUARD(&s->receive_mutex); for (i =3D 0; i < MAX_NBD_REQUESTS; i++) { if (nbd_recv_coroutine_wake_one(&s->requests[i])) { return; @@ -974,9 +974,11 @@ static coroutine_fn int nbd_co_receive_one_chunk( /* For assert at loop start in nbd_connection_entry */ *reply =3D s->reply; } - s->reply.cookie =3D 0; =20 - nbd_recv_coroutines_wake(s); + WITH_QEMU_LOCK_GUARD(&s->receive_mutex) { + s->reply.cookie =3D 0; + nbd_recv_coroutines_wake(s); + } =20 return ret; } --=20 2.53.0