From nobody Fri May 17 06:54:17 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=yandex-team.ru ARC-Seal: i=1; a=rsa-sha256; t=1628506266; cv=none; d=zohomail.com; s=zohoarc; b=MSnLVm5Ch81KeUCvNsjGS3GQ4vAoHL5edrAO/aBzq7W1duyrPOKaVFPvq5mCeNj/RhYckeBwErVo/Pjvth1DINlT7waDrpwSLpRq4EPeP1rdJPtWBunXwXlIBBItwQSnWoQS7VUNJiy74sJePy6V3qsiFBOUyjwbFxXYBm7oTfo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1628506266; h=Content-Transfer-Encoding:Cc:Date:From:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:To; bh=E+AIi5dcAT/D+Nw4sUXYYV5dpnn8ts74ugbXUphWCsw=; b=mMQit/AqAEhtJjOaM+pbtNAJ/aT93Rvt2hwQvSEU10ZVtHBNojjyCLLUtey3JGreXQd3Ivp0T6Lxir7k6sw1WS+cjxaDBI4SOz/g3h8mXJmphFPqv/rfgD8ZadGqRdcjqjf/MOhggjffkb6es41kw3L1jMSximMLkQro+DES1ZU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1628506265899370.0571135677028; Mon, 9 Aug 2021 03:51:05 -0700 (PDT) Received: from localhost ([::1]:47438 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mD2s4-0003Lv-IL for importer@patchew.org; Mon, 09 Aug 2021 06:51:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42282) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mD2pj-0007SX-Jn for qemu-devel@nongnu.org; Mon, 09 Aug 2021 06:48:40 -0400 Received: from forwardcorp1j.mail.yandex.net ([5.45.199.163]:48930) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mD2pf-0002iN-3V for qemu-devel@nongnu.org; Mon, 09 Aug 2021 06:48:37 -0400 Received: from vla1-fdfb804fb3f3.qloud-c.yandex.net (vla1-fdfb804fb3f3.qloud-c.yandex.net [IPv6:2a02:6b8:c0d:3199:0:640:fdfb:804f]) by forwardcorp1j.mail.yandex.net (Yandex) with ESMTP id A22642E167B; Mon, 9 Aug 2021 13:48:30 +0300 (MSK) Received: from vla5-d6d5ce7a4718.qloud-c.yandex.net (vla5-d6d5ce7a4718.qloud-c.yandex.net [2a02:6b8:c18:341e:0:640:d6d5:ce7a]) by vla1-fdfb804fb3f3.qloud-c.yandex.net (mxbackcorp/Yandex) with ESMTP id 4TP2V7RTEF-mU0aJ7Bp; Mon, 09 Aug 2021 13:48:30 +0300 Received: from dynamic-vpn.dhcp.yndx.net (dynamic-vpn.dhcp.yndx.net [2a02:6b8:b081:8008::1:24]) by vla5-d6d5ce7a4718.qloud-c.yandex.net (smtpcorp/Yandex) with ESMTPSA id EsY8JDDQcV-mU24Xhv0; Mon, 09 Aug 2021 13:48:30 +0300 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client certificate not present) Precedence: bulk DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.ru; s=default; t=1628506110; bh=E+AIi5dcAT/D+Nw4sUXYYV5dpnn8ts74ugbXUphWCsw=; h=Message-Id:Date:Subject:To:From:Cc; b=b41hQF1JBaqpFz9lto80BQcjAch+LE0B5qK934Pdm8fSIV6ca5O1SG1wZ029P0DJX smXCHV5URC/3yz57UHWlKebxKA4PVV1J4rS1NUsx+ldBHokGqxr/FpKr17r46AMXFF TlF7rSsjlLs3NDzsl51gaP7PZ+QGu/6DsSL42m0c= Authentication-Results: vla1-fdfb804fb3f3.qloud-c.yandex.net; dkim=pass header.i=@yandex-team.ru From: Denis Plotnikov To: qemu-devel@nongnu.org Subject: [PATCH v4] vhost: make SET_VRING_ADDR, SET_FEATURES send replies Date: Mon, 9 Aug 2021 13:48:24 +0300 Message-Id: <20210809104824.78830-1-den-plotnikov@yandex-team.ru> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=5.45.199.163; envelope-from=den-plotnikov@yandex-team.ru; helo=forwardcorp1j.mail.yandex.net X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yc-core@yandex-team.ru, mst@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @yandex-team.ru) X-ZM-MESSAGEID: 1628506268577100001 Content-Type: text/plain; charset="utf-8" On vhost-user-blk migration, qemu normally sends a number of commands to enable logging if VHOST_USER_PROTOCOL_F_LOG_SHMFD is negotiated. Qemu sends VHOST_USER_SET_FEATURES to enable buffers logging and VHOST_USER_SET_VRING_ADDR per each started ring to enable "used ring" data logging. The issue is that qemu doesn't wait for reply from the vhost daemon for these commands which may result in races between qemu expectation of logging starting and actual login starting in vhost daemon. The race can appear as follows: on migration setup, qemu enables dirty page logging by sending VHOST_USER_SET_FEATURES. The command doesn't arrive to a vhost-user-blk daemon immediately and the daemon needs some time to turn the logging on internally. If qemu doesn't wait for reply, after sending the command, qemu may start migrateing memory pages to a destination. At this t= ime, the logging may not be actually turned on in the daemon but some guest page= s, which the daemon is about to write to, may have already been transferred without logging to the destination. Since the logging wasn't turned on, those pages won't be transferred again as dirty. So we may end up with corrupted data on the destination. The same scenario is applicable for "used ring" data logging, which is turned on with VHOST_USER_SET_VRING_ADDR command. To resolve this issue, this patch makes qemu wait for the command result explicitly if VHOST_USER_PROTOCOL_F_REPLY_ACK is negotiated and logging ena= bled. Signed-off-by: Denis Plotnikov --- v3 -> v4: * join acked reply and get_features in enforce_reply [mst] * typos, rewording, cosmetic changes [mst] v2 -> v3: * send VHOST_USER_GET_FEATURES to flush out outstanding messages [mst] v1 -> v2: * send reply only when logging is enabled [mst] v0 -> v1: * send reply for SET_VRING_ADDR, SET_FEATURES only [mst] --- hw/virtio/vhost-user.c | 139 +++++++++++++++++++++++++++++------------ 1 file changed, 98 insertions(+), 41 deletions(-) diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index ee57abe04526..5bb9254acd21 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -1095,23 +1095,6 @@ static int vhost_user_set_mem_table(struct vhost_dev= *dev, return 0; } =20 -static int vhost_user_set_vring_addr(struct vhost_dev *dev, - struct vhost_vring_addr *addr) -{ - VhostUserMsg msg =3D { - .hdr.request =3D VHOST_USER_SET_VRING_ADDR, - .hdr.flags =3D VHOST_USER_VERSION, - .payload.addr =3D *addr, - .hdr.size =3D sizeof(msg.payload.addr), - }; - - if (vhost_user_write(dev, &msg, NULL, 0) < 0) { - return -1; - } - - return 0; -} - static int vhost_user_set_vring_endian(struct vhost_dev *dev, struct vhost_vring_state *ring) { @@ -1288,72 +1271,146 @@ static int vhost_user_set_vring_call(struct vhost_= dev *dev, return vhost_set_vring_file(dev, VHOST_USER_SET_VRING_CALL, file); } =20 -static int vhost_user_set_u64(struct vhost_dev *dev, int request, uint64_t= u64) + +static int vhost_user_get_u64(struct vhost_dev *dev, int request, uint64_t= *u64) { VhostUserMsg msg =3D { .hdr.request =3D request, .hdr.flags =3D VHOST_USER_VERSION, - .payload.u64 =3D u64, - .hdr.size =3D sizeof(msg.payload.u64), }; =20 + if (vhost_user_one_time_request(request) && dev->vq_index !=3D 0) { + return 0; + } + if (vhost_user_write(dev, &msg, NULL, 0) < 0) { return -1; } =20 + if (vhost_user_read(dev, &msg) < 0) { + return -1; + } + + if (msg.hdr.request !=3D request) { + error_report("Received unexpected msg type. Expected %d received %= d", + request, msg.hdr.request); + return -1; + } + + if (msg.hdr.size !=3D sizeof(msg.payload.u64)) { + error_report("Received bad msg size."); + return -1; + } + + *u64 =3D msg.payload.u64; + return 0; } =20 -static int vhost_user_set_features(struct vhost_dev *dev, - uint64_t features) +static int vhost_user_get_features(struct vhost_dev *dev, uint64_t *featur= es) { - return vhost_user_set_u64(dev, VHOST_USER_SET_FEATURES, features); + return vhost_user_get_u64(dev, VHOST_USER_GET_FEATURES, features); } =20 -static int vhost_user_set_protocol_features(struct vhost_dev *dev, - uint64_t features) +static int enforce_reply(struct vhost_dev *dev, + const VhostUserMsg *msg) { - return vhost_user_set_u64(dev, VHOST_USER_SET_PROTOCOL_FEATURES, featu= res); + uint64_t dummy; + + if (msg->hdr.flags & VHOST_USER_NEED_REPLY_MASK) { + return process_message_reply(dev, msg); + } + + /* + * We need to wait for a reply but the backend does not + * support replies for the command we just sent. + * Send VHOST_USER_GET_FEATURES which makes all backends + * send a reply. + */ + return vhost_user_get_features(dev, &dummy); } =20 -static int vhost_user_get_u64(struct vhost_dev *dev, int request, uint64_t= *u64) +static int vhost_user_set_vring_addr(struct vhost_dev *dev, + struct vhost_vring_addr *addr) { VhostUserMsg msg =3D { - .hdr.request =3D request, + .hdr.request =3D VHOST_USER_SET_VRING_ADDR, .hdr.flags =3D VHOST_USER_VERSION, + .payload.addr =3D *addr, + .hdr.size =3D sizeof(msg.payload.addr), }; =20 - if (vhost_user_one_time_request(request) && dev->vq_index !=3D 0) { - return 0; + bool reply_supported =3D virtio_has_feature(dev->protocol_features, + VHOST_USER_PROTOCOL_F_REPLY_= ACK); + + /* + * wait for a reply if logging is enabled to make sure + * backend is actually logging changes + */ + bool wait_for_reply =3D addr->flags & (1 << VHOST_VRING_F_LOG); + + if (reply_supported && wait_for_reply) { + msg.hdr.flags |=3D VHOST_USER_NEED_REPLY_MASK; } =20 if (vhost_user_write(dev, &msg, NULL, 0) < 0) { return -1; } =20 - if (vhost_user_read(dev, &msg) < 0) { - return -1; + if (wait_for_reply) { + return enforce_reply(dev, &msg); } =20 - if (msg.hdr.request !=3D request) { - error_report("Received unexpected msg type. Expected %d received %= d", - request, msg.hdr.request); - return -1; + return 0; +} + +static int vhost_user_set_u64(struct vhost_dev *dev, int request, uint64_t= u64, + bool wait_for_reply) +{ + VhostUserMsg msg =3D { + .hdr.request =3D request, + .hdr.flags =3D VHOST_USER_VERSION, + .payload.u64 =3D u64, + .hdr.size =3D sizeof(msg.payload.u64), + }; + + if (wait_for_reply) { + bool reply_supported =3D virtio_has_feature(dev->protocol_features, + VHOST_USER_PROTOCOL_F_REPLY_ACK); + if (reply_supported) { + msg.hdr.flags |=3D VHOST_USER_NEED_REPLY_MASK; + } } =20 - if (msg.hdr.size !=3D sizeof(msg.payload.u64)) { - error_report("Received bad msg size."); + if (vhost_user_write(dev, &msg, NULL, 0) < 0) { return -1; } =20 - *u64 =3D msg.payload.u64; + if (wait_for_reply) { + return enforce_reply(dev, &msg); + } =20 return 0; } =20 -static int vhost_user_get_features(struct vhost_dev *dev, uint64_t *featur= es) +static int vhost_user_set_features(struct vhost_dev *dev, + uint64_t features) { - return vhost_user_get_u64(dev, VHOST_USER_GET_FEATURES, features); + /* + * wait for a reply if logging is enabled to make sure + * backend is actually logging changes + */ + bool log_enabled =3D features & (0x1ULL << VHOST_F_LOG_ALL); + + return vhost_user_set_u64(dev, VHOST_USER_SET_FEATURES, features, + log_enabled); +} + +static int vhost_user_set_protocol_features(struct vhost_dev *dev, + uint64_t features) +{ + return vhost_user_set_u64(dev, VHOST_USER_SET_PROTOCOL_FEATURES, featu= res, + false); } =20 static int vhost_user_set_owner(struct vhost_dev *dev) --=20 2.25.1