From nobody Wed Dec 17 03:50:56 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B685926D4E2 for ; Mon, 16 Jun 2025 08:25:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062339; cv=none; b=nq1tDQzpp6zOsaDxVr+DvaVtWibN86z2RIHkkrpYzSEiVBw6nHSctBN6eLU7bRdoOZHYk8TfZguERWWbF/sWG36B+bcnKnIGlz07T2ScmWXItix6utd0mha2hPHF2/dH64H8hTF8KttXZ3u6Z+B7ONwe/F8Iy8mXWD5jK54/3Vw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062339; c=relaxed/simple; bh=l9yYXpVqdAXJijcXpRhmrPySzy023mVOc1RDh5s/M9Q=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nV+JsFtV8yrzsP5txUzLlMJicXa9ckOjy7yoasF8BcnbvyhXceDqk0ZhFuOI+f4Qrh1ojrBMeC2Jkil5kwgoHGAVK+uLWFN8Twuf0ihzf64SCm0VSNdRaLRjQur4Y88dssw5ZDzfo9tJ6VVR3W9TlK6m5eHq8rLJflCQRLESAJM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=eSyZHPDX; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="eSyZHPDX" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1750062334; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=anOeUruLXgalb93hmRoUy51mNb2XkK4WZHKRllxvw5o=; b=eSyZHPDXcAY0GA94Y6xgTKvyY2L4yJ3zyHrBltTusZJ9khjj8It06/5S5pKElt9OhuGhvc 7zC8cYyMH4jyJjdKmQzrkJJrh6rcOnsYjJrRscX1cnXJvYNiNUjUUw12E8wE7s96V6qDLp 6Jz+WRWv+z1EKoPN+vQopIxrpFozBqY= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-288--l9R9-IvPyiFFRR6NfHQgA-1; Mon, 16 Jun 2025 04:25:31 -0400 X-MC-Unique: -l9R9-IvPyiFFRR6NfHQgA-1 X-Mimecast-MFC-AGG-ID: -l9R9-IvPyiFFRR6NfHQgA_1750062330 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 108D8180028A; Mon, 16 Jun 2025 08:25:30 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.72]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 8D47A1800288; Mon, 16 Jun 2025 08:25:26 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com Cc: xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V3 01/19] virtio_ring: rename virtqueue_reinit_xxx to virtqueue_reset_xxx() Date: Mon, 16 Jun 2025 16:24:59 +0800 Message-ID: <20250616082518.10411-2-jasowang@redhat.com> In-Reply-To: <20250616082518.10411-1-jasowang@redhat.com> References: <20250616082518.10411-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" To be consistent with virtqueue_reset(). Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang Acked-by: Eugenio P=C3=A9rez Tested-by: Lei Yang --- drivers/virtio/virtio_ring.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index b784aab66867..afdd51fc3c9c 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -1005,7 +1005,7 @@ static void virtqueue_vring_init_split(struct vring_v= irtqueue_split *vring_split } } =20 -static void virtqueue_reinit_split(struct vring_virtqueue *vq) +static void virtqueue_reset_split(struct vring_virtqueue *vq) { int num; =20 @@ -1248,7 +1248,7 @@ static int virtqueue_resize_split(struct virtqueue *_= vq, u32 num) err_state_extra: vring_free_split(&vring_split, vdev, vring_dma_dev(vq)); err: - virtqueue_reinit_split(vq); + virtqueue_reset_split(vq); return -ENOMEM; } =20 @@ -2092,7 +2092,7 @@ static void virtqueue_vring_attach_packed(struct vrin= g_virtqueue *vq, vq->free_head =3D 0; } =20 -static void virtqueue_reinit_packed(struct vring_virtqueue *vq) +static void virtqueue_reset_packed(struct vring_virtqueue *vq) { memset(vq->packed.vring.device, 0, vq->packed.event_size_in_bytes); memset(vq->packed.vring.driver, 0, vq->packed.event_size_in_bytes); @@ -2219,7 +2219,7 @@ static int virtqueue_resize_packed(struct virtqueue *= _vq, u32 num) err_state_extra: vring_free_packed(&vring_packed, vdev, vring_dma_dev(vq)); err_ring: - virtqueue_reinit_packed(vq); + virtqueue_reset_packed(vq); return -ENOMEM; } =20 @@ -2852,9 +2852,9 @@ int virtqueue_reset(struct virtqueue *_vq, recycle_done(_vq); =20 if (vq->packed_ring) - virtqueue_reinit_packed(vq); + virtqueue_reset_packed(vq); else - virtqueue_reinit_split(vq); + virtqueue_reset_split(vq); =20 return virtqueue_enable_after_reset(_vq); } --=20 2.34.1 From nobody Wed Dec 17 03:50:56 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2C5F026D4E3 for ; Mon, 16 Jun 2025 08:25:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062341; cv=none; b=WAYkQ5w5aJNJH0adWWKTMn3BKjXuwElPu9S9XKuI3rFOFudPMPHY3T9aoveF+XZkPSpGtC2OGzjG/U60xV9b3dOiqKl9chnJ4KvRQARL2yKBJJ4sH9yycx+SuYwcKby7uMJxQfrjVPccPgMjBogCP2tjjt/SdBvOpQqKMowocgM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062341; c=relaxed/simple; bh=ltRl+emWPBu7wipuJOfYwLvpwcQ4/vWxCz1pEW5n0cU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=k+wVKnJl/eq3K6X7835IeEzs6QViPvKBD0Q4ohwiweDG2IkIbX0DXdV78ZrKiEFMme3u1res/+G/D8sDimsFrmQM9QYITTLMf0VMACaXjNAMKpjeTQXeeEOdUx78JpCHnYiSG7cTS4Q0cuMkcJ2U5q0IJC+sU7Nd1L9/B5w9sMo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=bSioArqr; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="bSioArqr" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1750062339; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CwNLhInba3gY2zDxjXczl0IeLlj5vXy4C2mIZqPhwy8=; b=bSioArqrpcpftOE5Diar63l/HuV8csMQFQCa2wpKxD6NJXODEjJTL/ADWcVKDzn0u8p5DO Rg43sBdGkuLnnQJ9ln4JD2GNg4DWyGY/90hHZVYlkAIPP5GL4uQV9+sSzJFZQ367Btof5Q 4ps+w5qZ7ARFNPhFclRo22P1Q8bjeyk= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-455-1b6hN2JFOMSGA8mu4ixtow-1; Mon, 16 Jun 2025 04:25:35 -0400 X-MC-Unique: 1b6hN2JFOMSGA8mu4ixtow-1 X-Mimecast-MFC-AGG-ID: 1b6hN2JFOMSGA8mu4ixtow_1750062334 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 33E3E19560A6; Mon, 16 Jun 2025 08:25:34 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.72]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id DCB5818003FC; Mon, 16 Jun 2025 08:25:30 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com Cc: xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V3 02/19] virtio_ring: switch to use vring_virtqueue in virtqueue_poll variants Date: Mon, 16 Jun 2025 16:25:00 +0800 Message-ID: <20250616082518.10411-3-jasowang@redhat.com> In-Reply-To: <20250616082518.10411-1-jasowang@redhat.com> References: <20250616082518.10411-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" Those variants are used internally so let's switch to use vring_virtqueue as parameter to be consistent with other internal virtqueue helpers. Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang Acked-by: Eugenio P=C3=A9rez --- drivers/virtio/virtio_ring.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index afdd51fc3c9c..9bc6c30458b5 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -915,11 +915,10 @@ static unsigned int virtqueue_enable_cb_prepare_split= (struct virtqueue *_vq) return last_used_idx; } =20 -static bool virtqueue_poll_split(struct virtqueue *_vq, unsigned int last_= used_idx) +static bool virtqueue_poll_split(struct vring_virtqueue *vq, + unsigned int last_used_idx) { - struct vring_virtqueue *vq =3D to_vvq(_vq); - - return (u16)last_used_idx !=3D virtio16_to_cpu(_vq->vdev, + return (u16)last_used_idx !=3D virtio16_to_cpu(vq->vq.vdev, vq->split.vring.used->idx); } =20 @@ -1845,9 +1844,8 @@ static unsigned int virtqueue_enable_cb_prepare_packe= d(struct virtqueue *_vq) return vq->last_used_idx; } =20 -static bool virtqueue_poll_packed(struct virtqueue *_vq, u16 off_wrap) +static bool virtqueue_poll_packed(struct vring_virtqueue *vq, u16 off_wrap) { - struct vring_virtqueue *vq =3D to_vvq(_vq); bool wrap_counter; u16 used_idx; =20 @@ -2608,8 +2606,8 @@ bool virtqueue_poll(struct virtqueue *_vq, unsigned i= nt last_used_idx) return false; =20 virtio_mb(vq->weak_barriers); - return vq->packed_ring ? virtqueue_poll_packed(_vq, last_used_idx) : - virtqueue_poll_split(_vq, last_used_idx); + return vq->packed_ring ? virtqueue_poll_packed(vq, last_used_idx) : + virtqueue_poll_split(vq, last_used_idx); } EXPORT_SYMBOL_GPL(virtqueue_poll); =20 --=20 2.34.1 From nobody Wed Dec 17 03:50:56 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3628A26E17D for ; Mon, 16 Jun 2025 08:25:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062345; cv=none; b=OIrJVEHLVOpcXivCUQOgUoPq642QPreuqtecjmjCrmMF+cT+7iVvbnVOgNPRUgqltteTYE7lMAMqBHGlWuT4cFUuPWVC5HMBh1zFTZNN8iMCD6VnF2RxSqmTZl5QHXfdHNWbaX45JldP2EC4a39+EtPPssV8T5UxMy7Q/FwRa2Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062345; c=relaxed/simple; bh=GCD93KUV7iRSk4N/ygyNxHVcWGVx3ZbfX6C9AsGuHv4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NJvtRpKODbTu8oeyWxd02R546JCQE9RLN4tvyuAdiXCTjScsedJeVsILV/kDpEWMoKy84KSXQcokjryS138H8chewgA6wD+SjJCzqpRrwiUBQEE10DE9v1bOzJhS+iB+9608D3drtbx5TXuJw6O8fDsDjle1Sd7H7QILJaApoiA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=fju8lxj9; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="fju8lxj9" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1750062342; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Z0G5RoIUxIjlNRXmAj3Y/oAlyW4NGdkkfJMJZNnBgMU=; b=fju8lxj9HlTCfjKGqrf+rVF+MoVSCDQ8gPhe68/miAMnZ8Q3V8jrKip/jqMRyvOlsoSKkG DYnd4bHkzlpthHGC6Qlg9BvBCnBU3S1DxYfSlYnCqLfTXXzw+Q9wMPykePo1WMBwHeR1+s 1tMsxRCeVI1sTDuXlEKNhPke+skeeDI= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-146-KDQyAwrtM1ajmbUJqyzF0Q-1; Mon, 16 Jun 2025 04:25:39 -0400 X-MC-Unique: KDQyAwrtM1ajmbUJqyzF0Q-1 X-Mimecast-MFC-AGG-ID: KDQyAwrtM1ajmbUJqyzF0Q_1750062338 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 4319D1800281; Mon, 16 Jun 2025 08:25:38 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.72]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id DB60118003FC; Mon, 16 Jun 2025 08:25:34 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com Cc: xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V3 03/19] virtio_ring: unify logic of virtqueue_poll() and more_used() Date: Mon, 16 Jun 2025 16:25:01 +0800 Message-ID: <20250616082518.10411-4-jasowang@redhat.com> In-Reply-To: <20250616082518.10411-1-jasowang@redhat.com> References: <20250616082518.10411-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" This patch unifies the logic of virtqueue_poll() and more_used() for better code reusing and ease the future in order implementation. Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang Acked-by: Eugenio P=C3=A9rez --- drivers/virtio/virtio_ring.c | 48 +++++++++++++++--------------------- 1 file changed, 20 insertions(+), 28 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 9bc6c30458b5..633379b2dc42 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -802,12 +802,18 @@ static void detach_buf_split(struct vring_virtqueue *= vq, unsigned int head, } } =20 -static bool more_used_split(const struct vring_virtqueue *vq) +static bool virtqueue_poll_split(const struct vring_virtqueue *vq, + unsigned int last_used_idx) { - return vq->last_used_idx !=3D virtio16_to_cpu(vq->vq.vdev, + return (u16)last_used_idx !=3D virtio16_to_cpu(vq->vq.vdev, vq->split.vring.used->idx); } =20 +static bool more_used_split(const struct vring_virtqueue *vq) +{ + return virtqueue_poll_split(vq, vq->last_used_idx); +} + static void *virtqueue_get_buf_ctx_split(struct virtqueue *_vq, unsigned int *len, void **ctx) @@ -915,13 +921,6 @@ static unsigned int virtqueue_enable_cb_prepare_split(= struct virtqueue *_vq) return last_used_idx; } =20 -static bool virtqueue_poll_split(struct vring_virtqueue *vq, - unsigned int last_used_idx) -{ - return (u16)last_used_idx !=3D virtio16_to_cpu(vq->vq.vdev, - vq->split.vring.used->idx); -} - static bool virtqueue_enable_cb_delayed_split(struct virtqueue *_vq) { struct vring_virtqueue *vq =3D to_vvq(_vq); @@ -1711,16 +1710,20 @@ static inline bool is_used_desc_packed(const struct= vring_virtqueue *vq, return avail =3D=3D used && used =3D=3D used_wrap_counter; } =20 -static bool more_used_packed(const struct vring_virtqueue *vq) +static bool virtqueue_poll_packed(const struct vring_virtqueue *vq, u16 of= f_wrap) { - u16 last_used; - u16 last_used_idx; - bool used_wrap_counter; + bool wrap_counter; + u16 used_idx; =20 - last_used_idx =3D READ_ONCE(vq->last_used_idx); - last_used =3D packed_last_used(last_used_idx); - used_wrap_counter =3D packed_used_wrap_counter(last_used_idx); - return is_used_desc_packed(vq, last_used, used_wrap_counter); + wrap_counter =3D off_wrap >> VRING_PACKED_EVENT_F_WRAP_CTR; + used_idx =3D off_wrap & ~(1 << VRING_PACKED_EVENT_F_WRAP_CTR); + + return is_used_desc_packed(vq, used_idx, wrap_counter); +} + +static bool more_used_packed(const struct vring_virtqueue *vq) +{ + return virtqueue_poll_packed(vq, READ_ONCE(vq->last_used_idx)); } =20 static void *virtqueue_get_buf_ctx_packed(struct virtqueue *_vq, @@ -1844,17 +1847,6 @@ static unsigned int virtqueue_enable_cb_prepare_pack= ed(struct virtqueue *_vq) return vq->last_used_idx; } =20 -static bool virtqueue_poll_packed(struct vring_virtqueue *vq, u16 off_wrap) -{ - bool wrap_counter; - u16 used_idx; - - wrap_counter =3D off_wrap >> VRING_PACKED_EVENT_F_WRAP_CTR; - used_idx =3D off_wrap & ~(1 << VRING_PACKED_EVENT_F_WRAP_CTR); - - return is_used_desc_packed(vq, used_idx, wrap_counter); -} - static bool virtqueue_enable_cb_delayed_packed(struct virtqueue *_vq) { struct vring_virtqueue *vq =3D to_vvq(_vq); --=20 2.34.1 From nobody Wed Dec 17 03:50:56 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8BD0426D4C3 for ; Mon, 16 Jun 2025 08:25:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062348; cv=none; b=PF0V47uGH/K96mR24liocRmCjUoc/OoVTsPDFtfqgiOfXUDjwuUT389X3obiv2N6AA740RC+/wrTR0ytS/LTqSXJdbMKavXSLzLk6d7Xc/91EajjQLaI84/qKuYdv7juobBg0WjZimYeL3lzzeVmSRtednp1TJUVspn8RQbQfFA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062348; c=relaxed/simple; bh=btrWKaG8/rzeBsz6IdVxk16L4Rb/ZH0842aOqeUc3vs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MvB9UXCtVI8b/xuVFXQbITQtS7gqtBmdA3rk4T35UYPNBDJ1Z8+bmtWTcByYDHsMP30chdn0wKiS3PYBMEmNi7f7Lk9VF8m79AESTkUQlxl/XHBbH5ZDoaVCrpW23TVWlvZDQ13zp0s0LnDg/ZshuY1x3ZGOq3G8xAe/yZ4pSBs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=IbXYHX7v; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="IbXYHX7v" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1750062345; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iPs4Pk+ZGeTi5sQMll9sHQqrUEkat8mM6CgdpH2mZwY=; b=IbXYHX7vWgTtw09aHU1q2LqTHLuEh8hlTqhSwfvx13qeZW6FPsJBh80OVQk5N2Up23RQU/ IFqx20VZuYaL37Fmwk1L9Wth+zkOsWxitUr23vHChZsNIdBRWMSBozJZGbabWGq5qmhVCB hkjjK+Ixkr0dQVj5eoJvOfb7+VI/WMA= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-522-9Ak6MBoBNBeMEk4WGFebNA-1; Mon, 16 Jun 2025 04:25:43 -0400 X-MC-Unique: 9Ak6MBoBNBeMEk4WGFebNA-1 X-Mimecast-MFC-AGG-ID: 9Ak6MBoBNBeMEk4WGFebNA_1750062342 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 76F661800282; Mon, 16 Jun 2025 08:25:42 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.72]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 5411818003FC; Mon, 16 Jun 2025 08:25:38 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com Cc: xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V3 04/19] virtio_ring: switch to use vring_virtqueue for virtqueue resize variants Date: Mon, 16 Jun 2025 16:25:02 +0800 Message-ID: <20250616082518.10411-5-jasowang@redhat.com> In-Reply-To: <20250616082518.10411-1-jasowang@redhat.com> References: <20250616082518.10411-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" Those variants are used internally so let's switch to use vring_virtqueue as parameter to be consistent with other internal virtqueue helpers. Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang Acked-by: Eugenio P=C3=A9rez --- drivers/virtio/virtio_ring.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 633379b2dc42..e7be480bee67 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -1216,11 +1216,10 @@ static struct virtqueue *vring_create_virtqueue_spl= it( return vq; } =20 -static int virtqueue_resize_split(struct virtqueue *_vq, u32 num) +static int virtqueue_resize_split(struct vring_virtqueue *vq, u32 num) { struct vring_virtqueue_split vring_split =3D {}; - struct vring_virtqueue *vq =3D to_vvq(_vq); - struct virtio_device *vdev =3D _vq->vdev; + struct virtio_device *vdev =3D vq->vq.vdev; int err; =20 err =3D vring_alloc_queue_split(&vring_split, vdev, num, @@ -2183,11 +2182,10 @@ static struct virtqueue *vring_create_virtqueue_pac= ked( return vq; } =20 -static int virtqueue_resize_packed(struct virtqueue *_vq, u32 num) +static int virtqueue_resize_packed(struct vring_virtqueue *vq, u32 num) { struct vring_virtqueue_packed vring_packed =3D {}; - struct vring_virtqueue *vq =3D to_vvq(_vq); - struct virtio_device *vdev =3D _vq->vdev; + struct virtio_device *vdev =3D vq->vq.vdev; int err; =20 if (vring_alloc_queue_packed(&vring_packed, vdev, num, vring_dma_dev(vq))) @@ -2805,9 +2803,9 @@ int virtqueue_resize(struct virtqueue *_vq, u32 num, recycle_done(_vq); =20 if (vq->packed_ring) - err =3D virtqueue_resize_packed(_vq, num); + err =3D virtqueue_resize_packed(vq, num); else - err =3D virtqueue_resize_split(_vq, num); + err =3D virtqueue_resize_split(vq, num); =20 return virtqueue_enable_after_reset(_vq); } --=20 2.34.1 From nobody Wed Dec 17 03:50:56 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EE9ED26E717 for ; Mon, 16 Jun 2025 08:25:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062351; cv=none; b=iZF5cLEwYAhzUbIOHHj2u0UWp0HIPbHDcrbXg7rMVHe7cpOCLGBD2GbETVYsXsEKzG6iZwzlIUi/Zc7NAlKioHZV5fr2yxvKnnvgwGAZQWWrFsfr6vfLWT1sLEj7gtCR5bJfLZuXSD1TUyQVwg+9dbxQ4FLQdyHNmFB2LYcIwf8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062351; c=relaxed/simple; bh=R2g4TwySJDJJocU4xK4nOTDTSRAr1GmJ2/lN3ltr1qg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=q2trK+fu3/YNceNl+8fsUZvMkv1CqoAvXU8eRZ3WJ/Pq3tYAZYPb03oWPmZ7EG9fgUF4vSONWaoAHyvGNLgHf9HeZrAi1/AXnOeKmgaPSV80VxGBXEmzvOhnalDSnZN6fBNlDk3x41C9Gzv03tQLKnpN8KaVSjrHyKsel6UtFyo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=EZIWwnCN; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="EZIWwnCN" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1750062349; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5oGRzY001stMjDiZhS80KlkkuWn5cfO4rLHANW0IMQ4=; b=EZIWwnCN/F+7MAWbD+6DiOj7Mz3EEg0P5b+SRR3M0QyOf9yWESdud+kwGMjildIx0Egoxo oss026bGMECUJIg/6RYa4qkKuOaaZenZmYqGFYMZDTWCrXktiZ1o1gI42JRHvTiWiBoNbd lhZT9x15TkUxYFLJrZlNPUb+KNPxMrk= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-131-QXeykfAKOvCloURUsEpyzw-1; Mon, 16 Jun 2025 04:25:47 -0400 X-MC-Unique: QXeykfAKOvCloURUsEpyzw-1 X-Mimecast-MFC-AGG-ID: QXeykfAKOvCloURUsEpyzw_1750062346 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 5F555195608D; Mon, 16 Jun 2025 08:25:46 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.72]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 379C9180045B; Mon, 16 Jun 2025 08:25:42 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com Cc: xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V3 05/19] virtio_ring: switch to use vring_virtqueue for virtqueue_kick_prepare variants Date: Mon, 16 Jun 2025 16:25:03 +0800 Message-ID: <20250616082518.10411-6-jasowang@redhat.com> In-Reply-To: <20250616082518.10411-1-jasowang@redhat.com> References: <20250616082518.10411-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" Those variants are used internally so let's switch to use vring_virtqueue as parameter to be consistent with other internal virtqueue helpers. Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang Acked-by: Eugenio P=C3=A9rez --- drivers/virtio/virtio_ring.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index e7be480bee67..8ee5393803f0 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -713,9 +713,8 @@ static inline int virtqueue_add_split(struct virtqueue = *_vq, return -ENOMEM; } =20 -static bool virtqueue_kick_prepare_split(struct virtqueue *_vq) +static bool virtqueue_kick_prepare_split(struct vring_virtqueue *vq) { - struct vring_virtqueue *vq =3D to_vvq(_vq); u16 new, old; bool needs_kick; =20 @@ -732,12 +731,12 @@ static bool virtqueue_kick_prepare_split(struct virtq= ueue *_vq) LAST_ADD_TIME_INVALID(vq); =20 if (vq->event) { - needs_kick =3D vring_need_event(virtio16_to_cpu(_vq->vdev, + needs_kick =3D vring_need_event(virtio16_to_cpu(vq->vq.vdev, vring_avail_event(&vq->split.vring)), new, old); } else { needs_kick =3D !(vq->split.vring.used->flags & - cpu_to_virtio16(_vq->vdev, + cpu_to_virtio16(vq->vq.vdev, VRING_USED_F_NO_NOTIFY)); } END_USE(vq); @@ -1597,9 +1596,8 @@ static inline int virtqueue_add_packed(struct virtque= ue *_vq, return -EIO; } =20 -static bool virtqueue_kick_prepare_packed(struct virtqueue *_vq) +static bool virtqueue_kick_prepare_packed(struct vring_virtqueue *vq) { - struct vring_virtqueue *vq =3D to_vvq(_vq); u16 new, old, off_wrap, flags, wrap_counter, event_idx; bool needs_kick; union { @@ -2454,8 +2452,8 @@ bool virtqueue_kick_prepare(struct virtqueue *_vq) { struct vring_virtqueue *vq =3D to_vvq(_vq); =20 - return vq->packed_ring ? virtqueue_kick_prepare_packed(_vq) : - virtqueue_kick_prepare_split(_vq); + return vq->packed_ring ? virtqueue_kick_prepare_packed(vq) : + virtqueue_kick_prepare_split(vq); } EXPORT_SYMBOL_GPL(virtqueue_kick_prepare); =20 --=20 2.34.1 From nobody Wed Dec 17 03:50:56 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 382F726E140 for ; Mon, 16 Jun 2025 08:25:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062357; cv=none; b=CTgn1WADQE/Ynk3Jyl0kQXU956It6LtUjkomHVq0T9FoeGXXlKqr/y0MJlHHIoc+P1Nz0gMeqhGOj3It/cuxCI0HpTBLQYMmNikgJTInS/vwtRJUcwu+IAfSDKtIuFuXlmhn2LkYJOn6Z+Ozbu7SaudQTOhI8h4Y17o/dXis1/c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062357; c=relaxed/simple; bh=nt3B7DEzQx6N5VdwnNvqsubVH9rR1Vk908e1fj/eLNw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ucXhNNnp6MM1Qh8Gssh3vMSBdpD885873ExvR+f0oovCQmzDhf7NRU13S+nPWCUEtKRShQFq6cVfXzBgNoTCBuvQPGm7KDPiMP2XmjdqZRRBzGBabnjWzdslL+4cSTqNhY7tEVJlbf4rFjEjhQPd8nFpS/2RPXKs63RZhrugleI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=T8ApM2Qh; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="T8ApM2Qh" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1750062355; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tyId8yI7nUURoGxQPFfWH8F2CiQwnAczJlVkGrGxoi0=; b=T8ApM2QhdBgXJ7JgHDyaXN9PSdBlZqrgdco6bpl73HzhkV3UqKBonQ0KIfZmMC9rucbg+Q UWo0ffqozbQHSIks4UiZwYc9pHXJ2XUCgEjqBdBnzENuPunWI6xUtAjpc5gPkd+OQf+GRU vrPOgVamsZbtNjcHgbRAfeqMTMQxHxA= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-611-68e2uuOiPjuLOAeRkawPPw-1; Mon, 16 Jun 2025 04:25:51 -0400 X-MC-Unique: 68e2uuOiPjuLOAeRkawPPw-1 X-Mimecast-MFC-AGG-ID: 68e2uuOiPjuLOAeRkawPPw_1750062350 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 3C58818089B8; Mon, 16 Jun 2025 08:25:50 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.72]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 268F7180045B; Mon, 16 Jun 2025 08:25:46 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com Cc: xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V3 06/19] virtio_ring: switch to use vring_virtqueue for virtqueue_add variants Date: Mon, 16 Jun 2025 16:25:04 +0800 Message-ID: <20250616082518.10411-7-jasowang@redhat.com> In-Reply-To: <20250616082518.10411-1-jasowang@redhat.com> References: <20250616082518.10411-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" Those variants are used internally so let's switch to use vring_virtqueue as parameter to be consistent with other internal virtqueue helpers. Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang Acked-by: Eugenio P=C3=A9rez --- drivers/virtio/virtio_ring.c | 40 +++++++++++++++++------------------- 1 file changed, 19 insertions(+), 21 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 8ee5393803f0..a808792ca86b 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -472,7 +472,7 @@ static unsigned int vring_unmap_one_split(const struct = vring_virtqueue *vq, return extra->next; } =20 -static struct vring_desc *alloc_indirect_split(struct virtqueue *_vq, +static struct vring_desc *alloc_indirect_split(struct vring_virtqueue *vq, unsigned int total_sg, gfp_t gfp) { @@ -501,7 +501,7 @@ static struct vring_desc *alloc_indirect_split(struct v= irtqueue *_vq, return desc; } =20 -static inline unsigned int virtqueue_add_desc_split(struct virtqueue *vq, +static inline unsigned int virtqueue_add_desc_split(struct vring_virtqueue= *vq, struct vring_desc *desc, struct vring_desc_extra *extra, unsigned int i, @@ -509,11 +509,12 @@ static inline unsigned int virtqueue_add_desc_split(s= truct virtqueue *vq, unsigned int len, u16 flags, bool premapped) { + struct virtio_device *vdev =3D vq->vq.vdev; u16 next; =20 - desc[i].flags =3D cpu_to_virtio16(vq->vdev, flags); - desc[i].addr =3D cpu_to_virtio64(vq->vdev, addr); - desc[i].len =3D cpu_to_virtio32(vq->vdev, len); + desc[i].flags =3D cpu_to_virtio16(vdev, flags); + desc[i].addr =3D cpu_to_virtio64(vdev, addr); + desc[i].len =3D cpu_to_virtio32(vdev, len); =20 extra[i].addr =3D premapped ? DMA_MAPPING_ERROR : addr; extra[i].len =3D len; @@ -521,12 +522,12 @@ static inline unsigned int virtqueue_add_desc_split(s= truct virtqueue *vq, =20 next =3D extra[i].next; =20 - desc[i].next =3D cpu_to_virtio16(vq->vdev, next); + desc[i].next =3D cpu_to_virtio16(vdev, next); =20 return next; } =20 -static inline int virtqueue_add_split(struct virtqueue *_vq, +static inline int virtqueue_add_split(struct vring_virtqueue *vq, struct scatterlist *sgs[], unsigned int total_sg, unsigned int out_sgs, @@ -536,7 +537,6 @@ static inline int virtqueue_add_split(struct virtqueue = *_vq, bool premapped, gfp_t gfp) { - struct vring_virtqueue *vq =3D to_vvq(_vq); struct vring_desc_extra *extra; struct scatterlist *sg; struct vring_desc *desc; @@ -561,7 +561,7 @@ static inline int virtqueue_add_split(struct virtqueue = *_vq, head =3D vq->free_head; =20 if (virtqueue_use_indirect(vq, total_sg)) - desc =3D alloc_indirect_split(_vq, total_sg, gfp); + desc =3D alloc_indirect_split(vq, total_sg, gfp); else { desc =3D NULL; WARN_ON_ONCE(total_sg > vq->split.vring.num && !vq->indirect); @@ -608,7 +608,7 @@ static inline int virtqueue_add_split(struct virtqueue = *_vq, /* Note that we trust indirect descriptor * table since it use stream DMA mapping. */ - i =3D virtqueue_add_desc_split(_vq, desc, extra, i, addr, len, + i =3D virtqueue_add_desc_split(vq, desc, extra, i, addr, len, VRING_DESC_F_NEXT, premapped); } @@ -625,14 +625,14 @@ static inline int virtqueue_add_split(struct virtqueu= e *_vq, /* Note that we trust indirect descriptor * table since it use stream DMA mapping. */ - i =3D virtqueue_add_desc_split(_vq, desc, extra, i, addr, len, + i =3D virtqueue_add_desc_split(vq, desc, extra, i, addr, len, VRING_DESC_F_NEXT | VRING_DESC_F_WRITE, premapped); } } /* Last one doesn't continue. */ - desc[prev].flags &=3D cpu_to_virtio16(_vq->vdev, ~VRING_DESC_F_NEXT); + desc[prev].flags &=3D cpu_to_virtio16(vq->vq.vdev, ~VRING_DESC_F_NEXT); if (!indirect && vring_need_unmap_buffer(vq, &extra[prev])) vq->split.desc_extra[prev & (vq->split.vring.num - 1)].flags &=3D ~VRING_DESC_F_NEXT; @@ -645,7 +645,7 @@ static inline int virtqueue_add_split(struct virtqueue = *_vq, if (vring_mapping_error(vq, addr)) goto unmap_release; =20 - virtqueue_add_desc_split(_vq, vq->split.vring.desc, + virtqueue_add_desc_split(vq, vq->split.vring.desc, vq->split.desc_extra, head, addr, total_sg * sizeof(struct vring_desc), @@ -671,13 +671,13 @@ static inline int virtqueue_add_split(struct virtqueu= e *_vq, /* Put entry in available array (but don't update avail->idx until they * do sync). */ avail =3D vq->split.avail_idx_shadow & (vq->split.vring.num - 1); - vq->split.vring.avail->ring[avail] =3D cpu_to_virtio16(_vq->vdev, head); + vq->split.vring.avail->ring[avail] =3D cpu_to_virtio16(vq->vq.vdev, head); =20 /* Descriptors and available array need to be set before we expose the * new available array entries. */ virtio_wmb(vq->weak_barriers); vq->split.avail_idx_shadow++; - vq->split.vring.avail->idx =3D cpu_to_virtio16(_vq->vdev, + vq->split.vring.avail->idx =3D cpu_to_virtio16(vq->vq.vdev, vq->split.avail_idx_shadow); vq->num_added++; =20 @@ -687,7 +687,7 @@ static inline int virtqueue_add_split(struct virtqueue = *_vq, /* This is very unlikely, but theoretically possible. Kick * just in case. */ if (unlikely(vq->num_added =3D=3D (1 << 16) - 1)) - virtqueue_kick(_vq); + virtqueue_kick(&vq->vq); =20 return 0; =20 @@ -702,7 +702,6 @@ static inline int virtqueue_add_split(struct virtqueue = *_vq, for (n =3D 0; n < total_sg; n++) { if (i =3D=3D err_idx) break; - i =3D vring_unmap_one_split(vq, &extra[i]); } =20 @@ -1441,7 +1440,7 @@ static int virtqueue_add_indirect_packed(struct vring= _virtqueue *vq, return -ENOMEM; } =20 -static inline int virtqueue_add_packed(struct virtqueue *_vq, +static inline int virtqueue_add_packed(struct vring_virtqueue *vq, struct scatterlist *sgs[], unsigned int total_sg, unsigned int out_sgs, @@ -1451,7 +1450,6 @@ static inline int virtqueue_add_packed(struct virtque= ue *_vq, bool premapped, gfp_t gfp) { - struct vring_virtqueue *vq =3D to_vvq(_vq); struct vring_packed_desc *desc; struct scatterlist *sg; unsigned int i, n, c, descs_used, err_idx, len; @@ -2263,9 +2261,9 @@ static inline int virtqueue_add(struct virtqueue *_vq, { struct vring_virtqueue *vq =3D to_vvq(_vq); =20 - return vq->packed_ring ? virtqueue_add_packed(_vq, sgs, total_sg, + return vq->packed_ring ? virtqueue_add_packed(vq, sgs, total_sg, out_sgs, in_sgs, data, ctx, premapped, gfp) : - virtqueue_add_split(_vq, sgs, total_sg, + virtqueue_add_split(vq, sgs, total_sg, out_sgs, in_sgs, data, ctx, premapped, gfp); } =20 --=20 2.34.1 From nobody Wed Dec 17 03:50:56 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A765426D4FB for ; Mon, 16 Jun 2025 08:25:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062359; cv=none; b=kf73DT23Fl+GwqV4wN/0WbCjsIcFs9rVkGj7iw1iSJ72M0T8jNh49n4+B9TuV1SNsSJemJ+SXNWCGLUvVb3knkpTiQeV/70En9Ggnkm30kOyvXmTvc2b5ankaTK8LisNkn/Ve8xheksjPO0kf8WFPe74OyziXTizmatlBQGuNf8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062359; c=relaxed/simple; bh=006uCEyOIcGfx7NjBeZmXVBAuYLOY6/grsXDLBpyKNk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rjMC0r2UYP6UOfglmRJ22dfVp2/Iq0sdJM04hzL/eL8QteNWMHkzx0nkXiU98rsM7N2PE7xB8gmKGJaJCHuIJHwmKYvI+nL5SYF2kOt5xRfp1QupNEUC6y8ebv9sUc1OVvifiWMZvNfBkmscYLZNyiuN5+yknQSPvqEyB1JkQVU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=PVTDGjBw; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="PVTDGjBw" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1750062356; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xzvVr9ECZrNVmwWPiSq/eJF3/msAjqsreZZOmA+V+j8=; b=PVTDGjBw3FhNbSt1NwjRx8HeoGIdDfDS+K4tD+MVWe+7yGCyOVctVL9KofZGTNzE2xqyyk u/aCOodVNwhVh06a6rRHuzIpkb2xhd63aygeth7U1RFzZqzfrK9AS3xuoOBfoRWAaun/at K/szyzXiFwArOGJM/OqJYFybIbjZ/g8= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-625-I5YG2QPiOVW1YcR0KpfL6Q-1; Mon, 16 Jun 2025 04:25:55 -0400 X-MC-Unique: I5YG2QPiOVW1YcR0KpfL6Q-1 X-Mimecast-MFC-AGG-ID: I5YG2QPiOVW1YcR0KpfL6Q_1750062354 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 370C719560B6; Mon, 16 Jun 2025 08:25:54 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.72]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id EFD2B180045B; Mon, 16 Jun 2025 08:25:50 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com Cc: xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V3 07/19] virtio: switch to use vring_virtqueue for virtqueue_add variants Date: Mon, 16 Jun 2025 16:25:05 +0800 Message-ID: <20250616082518.10411-8-jasowang@redhat.com> In-Reply-To: <20250616082518.10411-1-jasowang@redhat.com> References: <20250616082518.10411-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" Those variants are used internally so let's switch to use vring_virtqueue as parameter to be consistent with other internal virtqueue helpers. Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang Acked-by: Eugenio P=C3=A9rez --- drivers/virtio/virtio_ring.c | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index a808792ca86b..4e70b7eb127f 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -812,11 +812,10 @@ static bool more_used_split(const struct vring_virtqu= eue *vq) return virtqueue_poll_split(vq, vq->last_used_idx); } =20 -static void *virtqueue_get_buf_ctx_split(struct virtqueue *_vq, +static void *virtqueue_get_buf_ctx_split(struct vring_virtqueue *vq, unsigned int *len, void **ctx) { - struct vring_virtqueue *vq =3D to_vvq(_vq); void *ret; unsigned int i; u16 last_used; @@ -838,9 +837,9 @@ static void *virtqueue_get_buf_ctx_split(struct virtque= ue *_vq, virtio_rmb(vq->weak_barriers); =20 last_used =3D (vq->last_used_idx & (vq->split.vring.num - 1)); - i =3D virtio32_to_cpu(_vq->vdev, + i =3D virtio32_to_cpu(vq->vq.vdev, vq->split.vring.used->ring[last_used].id); - *len =3D virtio32_to_cpu(_vq->vdev, + *len =3D virtio32_to_cpu(vq->vq.vdev, vq->split.vring.used->ring[last_used].len); =20 if (unlikely(i >=3D vq->split.vring.num)) { @@ -862,7 +861,7 @@ static void *virtqueue_get_buf_ctx_split(struct virtque= ue *_vq, if (!(vq->split.avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT)) virtio_store_mb(vq->weak_barriers, &vring_used_event(&vq->split.vring), - cpu_to_virtio16(_vq->vdev, vq->last_used_idx)); + cpu_to_virtio16(vq->vq.vdev, vq->last_used_idx)); =20 LAST_ADD_TIME_INVALID(vq); =20 @@ -1721,11 +1720,10 @@ static bool more_used_packed(const struct vring_vir= tqueue *vq) return virtqueue_poll_packed(vq, READ_ONCE(vq->last_used_idx)); } =20 -static void *virtqueue_get_buf_ctx_packed(struct virtqueue *_vq, +static void *virtqueue_get_buf_ctx_packed(struct vring_virtqueue *vq, unsigned int *len, void **ctx) { - struct vring_virtqueue *vq =3D to_vvq(_vq); u16 last_used, id, last_used_idx; bool used_wrap_counter; void *ret; @@ -2521,8 +2519,8 @@ void *virtqueue_get_buf_ctx(struct virtqueue *_vq, un= signed int *len, { struct vring_virtqueue *vq =3D to_vvq(_vq); =20 - return vq->packed_ring ? virtqueue_get_buf_ctx_packed(_vq, len, ctx) : - virtqueue_get_buf_ctx_split(_vq, len, ctx); + return vq->packed_ring ? virtqueue_get_buf_ctx_packed(vq, len, ctx) : + virtqueue_get_buf_ctx_split(vq, len, ctx); } EXPORT_SYMBOL_GPL(virtqueue_get_buf_ctx); =20 --=20 2.34.1 From nobody Wed Dec 17 03:50:56 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 40A9D26E140 for ; Mon, 16 Jun 2025 08:26:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062363; cv=none; b=Dw/soqDzvXu1o5rx5w3xnf4qvWfh090p2s5uSCJT3+subQZjkT1RQ2itMwwpSGNV39wl8dyOAmPx/SWqz13B6LyZeDnzrW+ct5cdV+3AYrrWjEOURai6CVqql3bz4861YXGyT0bck+oMzVCKUNjuRM9ZQCrDdCInnmjZucEnwCk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062363; c=relaxed/simple; bh=u1xTQqhC8xumQrThpUKU3cx+qP8/giKcy8EzKKSLLpQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=THka6Dj7dlcdAyr4jJd1dBgJbbD6gtOQ6j5v12js5BfFpixcJ8RUqauv/Xn1yCqtmJe2DzQq9itZjfYUj25JGEk3v3ZMNRIi06vozR2G4ktQ6IoR72S17HK1hVt4qX0U0dVDx9d1LMgDsU5B0IHktNc13IGCKjiJJ2pqI495WMk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=NU3g7q+s; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="NU3g7q+s" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1750062361; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yQmbPB9KOXUudyA4Np5TsxEyOaQPjCMAjSAANoPhTis=; b=NU3g7q+sJVP01lLwWMJ4XYxHkiRGzfozCbbp7vNqnFhNra46gAC1vR7QYDqotIxLFaL8T4 q4aNQO8TkZK8EWRjN4/IOvUZ8nTDM+PH2fy3Eu0yAxC0q7d7hMIPLyLriVFVbYLVkRqFVY GrZ3sLvoyEdd1wxMcEJMgnDIhn0aAPA= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-684-MS0jTb7jMamvxWN6UXUCDQ-1; Mon, 16 Jun 2025 04:25:59 -0400 X-MC-Unique: MS0jTb7jMamvxWN6UXUCDQ-1 X-Mimecast-MFC-AGG-ID: MS0jTb7jMamvxWN6UXUCDQ_1750062358 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 05C991956095; Mon, 16 Jun 2025 08:25:58 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.72]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id EA9FA180045C; Mon, 16 Jun 2025 08:25:54 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com Cc: xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V3 08/19] virtio_ring: switch to use vring_virtqueue for enable_cb_prepare variants Date: Mon, 16 Jun 2025 16:25:06 +0800 Message-ID: <20250616082518.10411-9-jasowang@redhat.com> In-Reply-To: <20250616082518.10411-1-jasowang@redhat.com> References: <20250616082518.10411-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" Those variants are used internally so let's switch to use vring_virtqueue as parameter to be consistent with other internal virtqueue helpers. Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang Acked-by: Eugenio P=C3=A9rez --- drivers/virtio/virtio_ring.c | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 4e70b7eb127f..8c18f3edda00 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -893,9 +893,8 @@ static void virtqueue_disable_cb_split(struct virtqueue= *_vq) } } =20 -static unsigned int virtqueue_enable_cb_prepare_split(struct virtqueue *_v= q) +static unsigned int virtqueue_enable_cb_prepare_split(struct vring_virtque= ue *vq) { - struct vring_virtqueue *vq =3D to_vvq(_vq); u16 last_used_idx; =20 START_USE(vq); @@ -909,10 +908,10 @@ static unsigned int virtqueue_enable_cb_prepare_split= (struct virtqueue *_vq) vq->split.avail_flags_shadow &=3D ~VRING_AVAIL_F_NO_INTERRUPT; if (!vq->event) vq->split.vring.avail->flags =3D - cpu_to_virtio16(_vq->vdev, + cpu_to_virtio16(vq->vq.vdev, vq->split.avail_flags_shadow); } - vring_used_event(&vq->split.vring) =3D cpu_to_virtio16(_vq->vdev, + vring_used_event(&vq->split.vring) =3D cpu_to_virtio16(vq->vq.vdev, last_used_idx =3D vq->last_used_idx); END_USE(vq); return last_used_idx; @@ -1807,10 +1806,8 @@ static void virtqueue_disable_cb_packed(struct virtq= ueue *_vq) } } =20 -static unsigned int virtqueue_enable_cb_prepare_packed(struct virtqueue *_= vq) +static unsigned int virtqueue_enable_cb_prepare_packed(struct vring_virtqu= eue *vq) { - struct vring_virtqueue *vq =3D to_vvq(_vq); - START_USE(vq); =20 /* @@ -2568,8 +2565,8 @@ unsigned int virtqueue_enable_cb_prepare(struct virtq= ueue *_vq) if (vq->event_triggered) vq->event_triggered =3D false; =20 - return vq->packed_ring ? virtqueue_enable_cb_prepare_packed(_vq) : - virtqueue_enable_cb_prepare_split(_vq); + return vq->packed_ring ? virtqueue_enable_cb_prepare_packed(vq) : + virtqueue_enable_cb_prepare_split(vq); } EXPORT_SYMBOL_GPL(virtqueue_enable_cb_prepare); =20 --=20 2.34.1 From nobody Wed Dec 17 03:50:56 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BBE312701A1 for ; Mon, 16 Jun 2025 08:26:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062368; cv=none; b=qfks20IbkDDfKRbDTUl9sVVUSp39ZZQApeqEnMmJjxMlhj5qKyUJ6PBFNRhy0USwr7Z5FZ5ccUwjcK3jWetOxiGNu3kCFdiYyXTSPnAsUXLmYygatubcanXVbNicSh7WLinOmJ8l7UPJ+e2RpalJMZWDcKsdqw+bDZnCBLRa8L0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062368; c=relaxed/simple; bh=nIFwCL3LvvMf1hnbugVDczRpMa79tK9rhUWMUPxuzCg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Hhrp2ZhZpKME9W3wY4zRUzm/x0QxQI6Y8sOznGbtZNcVuGJX7sz1X7uowEL+y32qDkmnmDnNiZwgwrvKJQh+Fb7pe7FRS04or0oYwhnGA8Ab7nbJEcddQ+J+0qamsYZQxhc1YWmG2IyfvZk81K3hzedi/5dt0Acqr+EVg4TTlHw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Tf1acYZl; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Tf1acYZl" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1750062365; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JxybsHFSbwGIVra5UuwiFVDVlrGXPEIskHhrIt4Gepc=; b=Tf1acYZlhjs8gHqAevb+cl/jXNK/GNRyj2YY3TlVvqgPLvJTN5Zy4t52GJlQoHlU15U8ah 50MUNV3dVIi5SB9BYyTyZrr8WsTCJ8K2+KO37AT1dpNPQletkvBY+XC5mcUOCUBhuZ1SYK tqqGLJmaK8So7xyxLVFbXkoxDoqPCd4= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-467-OiTGQ_V9O5KXnL9I-ppH4g-1; Mon, 16 Jun 2025 04:26:03 -0400 X-MC-Unique: OiTGQ_V9O5KXnL9I-ppH4g-1 X-Mimecast-MFC-AGG-ID: OiTGQ_V9O5KXnL9I-ppH4g_1750062362 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 172B81809C80; Mon, 16 Jun 2025 08:26:02 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.72]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 025C218003FC; Mon, 16 Jun 2025 08:25:58 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com Cc: xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V3 09/19] virtio_ring: use vring_virtqueue for enable_cb_delayed variants Date: Mon, 16 Jun 2025 16:25:07 +0800 Message-ID: <20250616082518.10411-10-jasowang@redhat.com> In-Reply-To: <20250616082518.10411-1-jasowang@redhat.com> References: <20250616082518.10411-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" Those variants are used internally so let's switch to use vring_virtqueue as parameter to be consistent with other internal virtqueue helpers. Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang Acked-by: Eugenio P=C3=A9rez --- drivers/virtio/virtio_ring.c | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 8c18f3edda00..a85db8c8cb24 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -917,9 +917,8 @@ static unsigned int virtqueue_enable_cb_prepare_split(s= truct vring_virtqueue *vq return last_used_idx; } =20 -static bool virtqueue_enable_cb_delayed_split(struct virtqueue *_vq) +static bool virtqueue_enable_cb_delayed_split(struct vring_virtqueue *vq) { - struct vring_virtqueue *vq =3D to_vvq(_vq); u16 bufs; =20 START_USE(vq); @@ -933,7 +932,7 @@ static bool virtqueue_enable_cb_delayed_split(struct vi= rtqueue *_vq) vq->split.avail_flags_shadow &=3D ~VRING_AVAIL_F_NO_INTERRUPT; if (!vq->event) vq->split.vring.avail->flags =3D - cpu_to_virtio16(_vq->vdev, + cpu_to_virtio16(vq->vq.vdev, vq->split.avail_flags_shadow); } /* TODO: tune this threshold */ @@ -941,9 +940,9 @@ static bool virtqueue_enable_cb_delayed_split(struct vi= rtqueue *_vq) =20 virtio_store_mb(vq->weak_barriers, &vring_used_event(&vq->split.vring), - cpu_to_virtio16(_vq->vdev, vq->last_used_idx + bufs)); + cpu_to_virtio16(vq->vq.vdev, vq->last_used_idx + bufs)); =20 - if (unlikely((u16)(virtio16_to_cpu(_vq->vdev, vq->split.vring.used->idx) + if (unlikely((u16)(virtio16_to_cpu(vq->vq.vdev, vq->split.vring.used->idx) - vq->last_used_idx) > bufs)) { END_USE(vq); return false; @@ -1837,9 +1836,8 @@ static unsigned int virtqueue_enable_cb_prepare_packe= d(struct vring_virtqueue *v return vq->last_used_idx; } =20 -static bool virtqueue_enable_cb_delayed_packed(struct virtqueue *_vq) +static bool virtqueue_enable_cb_delayed_packed(struct vring_virtqueue *vq) { - struct vring_virtqueue *vq =3D to_vvq(_vq); u16 used_idx, wrap_counter, last_used_idx; u16 bufs; =20 @@ -2631,8 +2629,8 @@ bool virtqueue_enable_cb_delayed(struct virtqueue *_v= q) if (vq->event_triggered) data_race(vq->event_triggered =3D false); =20 - return vq->packed_ring ? virtqueue_enable_cb_delayed_packed(_vq) : - virtqueue_enable_cb_delayed_split(_vq); + return vq->packed_ring ? virtqueue_enable_cb_delayed_packed(vq) : + virtqueue_enable_cb_delayed_split(vq); } EXPORT_SYMBOL_GPL(virtqueue_enable_cb_delayed); =20 --=20 2.34.1 From nobody Wed Dec 17 03:50:56 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AFB6D26E70B for ; Mon, 16 Jun 2025 08:26:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062380; cv=none; b=OFXNKFpwJzC/9SDCQYvHPXk9HLYef1YNQbvh1VbUsxlt/k3kSMe9U0UeY0XOsjvQdYrIpk7q32StCDfY3ssPs43T7ycED1Nn2UIDaOjFCg/iMoK1quNSfMdhlGuID2BS2QvgM/4WFc7zVCj4N2m3IuaVUl7sWifZyuhcck7Bi1w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062380; c=relaxed/simple; bh=pQ/qBfAP/YbFsSYZtTQm4Ikle06B8+fuTgKHV4RDXvo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tYFloGO5lDHZfxqaDILk+ad9jDVW8USvy+bP60Z0sk3kK1KF5j+Df1EuoW7XYAIT7+/Q21CEcasbDFwwjnaMKn5eN0NcN2yyyW2lnxZmTclwLBS/zhaHD0qr19HK6+iTEL8Xq1OHP4s6ju1Q5ff4hqu4N9r9CKRG+7FqY5wLjZs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=dhaD8uxE; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="dhaD8uxE" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1750062375; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hvMUw2OR66zDeWl5WPW7OL4jldXV0p/bHVpq4+2s510=; b=dhaD8uxEwbOSkcXKTh+4aomRjSH1UanKVhkGy4yfTXuJy+eehyvkgWLrUB3P0nc2XCbJ+Y r7apbqJGDb7Z6+JFqjVtJCCI6jJCTgrLSCZ+muUHXyY0dHbhYjm8zK3WSfCAaLHn1Kk//3 WQlIJ3nSABxYYO1wQjg5czAZQMguLUs= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-315-gayUxoA2M7WPYk809Sj3Hg-1; Mon, 16 Jun 2025 04:26:07 -0400 X-MC-Unique: gayUxoA2M7WPYk809Sj3Hg-1 X-Mimecast-MFC-AGG-ID: gayUxoA2M7WPYk809Sj3Hg_1750062366 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 23CEC180ABFF; Mon, 16 Jun 2025 08:26:06 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.72]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id CF3E418002B9; Mon, 16 Jun 2025 08:26:02 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com Cc: xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V3 10/19] virtio_ring: switch to use vring_virtqueue for disable_cb variants Date: Mon, 16 Jun 2025 16:25:08 +0800 Message-ID: <20250616082518.10411-11-jasowang@redhat.com> In-Reply-To: <20250616082518.10411-1-jasowang@redhat.com> References: <20250616082518.10411-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" Those variants are used internally so let's switch to use vring_virtqueue as parameter to be consistent with other internal virtqueue helpers. Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang Acked-by: Eugenio P=C3=A9rez --- drivers/virtio/virtio_ring.c | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index a85db8c8cb24..4af29a6e15e8 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -869,10 +869,8 @@ static void *virtqueue_get_buf_ctx_split(struct vring_= virtqueue *vq, return ret; } =20 -static void virtqueue_disable_cb_split(struct virtqueue *_vq) +static void virtqueue_disable_cb_split(struct vring_virtqueue *vq) { - struct vring_virtqueue *vq =3D to_vvq(_vq); - if (!(vq->split.avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT)) { vq->split.avail_flags_shadow |=3D VRING_AVAIL_F_NO_INTERRUPT; =20 @@ -888,7 +886,7 @@ static void virtqueue_disable_cb_split(struct virtqueue= *_vq) vring_used_event(&vq->split.vring) =3D 0x0; else vq->split.vring.avail->flags =3D - cpu_to_virtio16(_vq->vdev, + cpu_to_virtio16(vq->vq.vdev, vq->split.avail_flags_shadow); } } @@ -1786,10 +1784,8 @@ static void *virtqueue_get_buf_ctx_packed(struct vri= ng_virtqueue *vq, return ret; } =20 -static void virtqueue_disable_cb_packed(struct virtqueue *_vq) +static void virtqueue_disable_cb_packed(struct vring_virtqueue *vq) { - struct vring_virtqueue *vq =3D to_vvq(_vq); - if (vq->packed.event_flags_shadow !=3D VRING_PACKED_EVENT_FLAG_DISABLE) { vq->packed.event_flags_shadow =3D VRING_PACKED_EVENT_FLAG_DISABLE; =20 @@ -2538,9 +2534,9 @@ void virtqueue_disable_cb(struct virtqueue *_vq) struct vring_virtqueue *vq =3D to_vvq(_vq); =20 if (vq->packed_ring) - virtqueue_disable_cb_packed(_vq); + virtqueue_disable_cb_packed(vq); else - virtqueue_disable_cb_split(_vq); + virtqueue_disable_cb_split(vq); } EXPORT_SYMBOL_GPL(virtqueue_disable_cb); =20 --=20 2.34.1 From nobody Wed Dec 17 03:50:56 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A668B26E704 for ; Mon, 16 Jun 2025 08:26:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062378; cv=none; b=uao58m4XNVj+RZeiLFywigm6xF6rgY9QhAnVFWPY8XurQnWa0k1tIa4ECuXO/Lj06xeW5EtsmbnmfDjzEELUQ+zHWlKd3Z3c29FP5wlPo3yWt8TPLzFvFGHK9I1MHk7NTRL1ePVHQ2EQ02o0ofAyP0wJ5uH9YUB9DvXJ+3bhASc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062378; c=relaxed/simple; bh=pwqlNXqiThwXkwZ8WdEDyRZWZBWmhdcncW6BgL9BrcQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dg/RplT4U38UUbM5lBNcZYi8rwNEixtRB7qDghuykJSikVtwseRASef6+oAr7WvkM8ZtpyLnLDXk/a6ur6BHVWZESumX7y9AK2BF5fX1JSmkh21iLM/c1gZKYh6amxF1HxOwjbPRUIfTxhjxY+2COWd8z+93z27UH1AA9VK+P6M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=M94ZoSKo; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="M94ZoSKo" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1750062374; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cIjPOYFvozYojOMPnuFJOvuLP8xgwC21YPpzEJOgj1Y=; b=M94ZoSKoSbrxyFJHO8KcJqpqqYzM089nLwLVffNgioJAJ6RxK8mgNDb1bSrqOiDUUVaBfZ f2QHOEVnhAqmckle0hdNG71GSLLCWyF2P3fYeV6392KBqQFpmho0EJc8ZIMnQTFBsTOf0E S0dTazcY3IdindoXGK3yyQDKMoqXmrA= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-690-GNC6ExIaNXCAsj2VrGTR0w-1; Mon, 16 Jun 2025 04:26:11 -0400 X-MC-Unique: GNC6ExIaNXCAsj2VrGTR0w-1 X-Mimecast-MFC-AGG-ID: GNC6ExIaNXCAsj2VrGTR0w_1750062370 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 3ED1318001C6; Mon, 16 Jun 2025 08:26:10 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.72]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id C522618002B6; Mon, 16 Jun 2025 08:26:06 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com Cc: xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V3 11/19] virtio_ring: switch to use vring_virtqueue for detach_unused_buf variants Date: Mon, 16 Jun 2025 16:25:09 +0800 Message-ID: <20250616082518.10411-12-jasowang@redhat.com> In-Reply-To: <20250616082518.10411-1-jasowang@redhat.com> References: <20250616082518.10411-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" Those variants are used internally so let's switch to use vring_virtqueue as parameter to be consistent with other internal virtqueue helpers. Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang Acked-by: Eugenio P=C3=A9rez --- drivers/virtio/virtio_ring.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 4af29a6e15e8..03b17bc275dd 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -950,9 +950,8 @@ static bool virtqueue_enable_cb_delayed_split(struct vr= ing_virtqueue *vq) return true; } =20 -static void *virtqueue_detach_unused_buf_split(struct virtqueue *_vq) +static void *virtqueue_detach_unused_buf_split(struct vring_virtqueue *vq) { - struct vring_virtqueue *vq =3D to_vvq(_vq); unsigned int i; void *buf; =20 @@ -965,7 +964,7 @@ static void *virtqueue_detach_unused_buf_split(struct v= irtqueue *_vq) buf =3D vq->split.desc_state[i].data; detach_buf_split(vq, i, NULL); vq->split.avail_idx_shadow--; - vq->split.vring.avail->idx =3D cpu_to_virtio16(_vq->vdev, + vq->split.vring.avail->idx =3D cpu_to_virtio16(vq->vq.vdev, vq->split.avail_idx_shadow); END_USE(vq); return buf; @@ -1892,9 +1891,8 @@ static bool virtqueue_enable_cb_delayed_packed(struct= vring_virtqueue *vq) return true; } =20 -static void *virtqueue_detach_unused_buf_packed(struct virtqueue *_vq) +static void *virtqueue_detach_unused_buf_packed(struct vring_virtqueue *vq) { - struct vring_virtqueue *vq =3D to_vvq(_vq); unsigned int i; void *buf; =20 @@ -2642,8 +2640,8 @@ void *virtqueue_detach_unused_buf(struct virtqueue *_= vq) { struct vring_virtqueue *vq =3D to_vvq(_vq); =20 - return vq->packed_ring ? virtqueue_detach_unused_buf_packed(_vq) : - virtqueue_detach_unused_buf_split(_vq); + return vq->packed_ring ? virtqueue_detach_unused_buf_packed(vq) : + virtqueue_detach_unused_buf_split(vq); } EXPORT_SYMBOL_GPL(virtqueue_detach_unused_buf); =20 --=20 2.34.1 From nobody Wed Dec 17 03:50:56 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D50DE2701A1 for ; Mon, 16 Jun 2025 08:26:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062380; cv=none; b=bLJeVxXEYSO29jfG5bVav+45RXN80F6sn7yk6V99qqCmkzJSnklzoq4dgPpbV2JbL/Ii9P5aonKuwDHbAGbkaQH7l1K094JuOz/Nv7grOgEGX0huX4KIzfDgkmriE1rJHU77SSX3E7y7WFeQJHeOgogDk+h3I+1r3tLPUZ5C/Nk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062380; c=relaxed/simple; bh=k/j1JWrfmON7u1bdWCZW/FUEJTJ4hJKFDMpYrNv/D+M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Qyde7WAREopYMqjnf6YZ0QR2+XcAVtl5URdXdDQEi0qzyxzC5e1vuJaf+60ikBPQdSIxrDler7G4aGgwFJ4gDGRik6XLU/PCOf88/iHblw9GSOOeM9g3/H4iK2gs2tytwp0Wi2dARzLqhakiU7CkajCvIjqKVPy5LpcpGnr5qAU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=dr1Fmd9k; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="dr1Fmd9k" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1750062376; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+mjucDLGVtGHMuWA+LOyGgBQQBmFJEsmCYtz1e+yPSU=; b=dr1Fmd9kOmsVb410rItHJG/mz24wpUGjtkTGP2BKF8dX+OHGJARC5rxxcAU4bT3NZjXokW LK+DUrfO1OyBwHEL9aM92lt0TCrcwL64VS7K4wOw+ox1nRM0SZhMkL8G9YFTVl3xuAIs7T ivQfaiLuswXu0jWs0F/fuS3CAY2OYP8= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-635-qHmnmFFzMTaM8t_Cj_1V9Q-1; Mon, 16 Jun 2025 04:26:15 -0400 X-MC-Unique: qHmnmFFzMTaM8t_Cj_1V9Q-1 X-Mimecast-MFC-AGG-ID: qHmnmFFzMTaM8t_Cj_1V9Q_1750062374 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 38B881808985; Mon, 16 Jun 2025 08:26:14 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.72]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 0BA3A180045B; Mon, 16 Jun 2025 08:26:10 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com Cc: xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V3 12/19] virtio_ring: use u16 for last_used_idx in virtqueue_poll_split() Date: Mon, 16 Jun 2025 16:25:10 +0800 Message-ID: <20250616082518.10411-13-jasowang@redhat.com> In-Reply-To: <20250616082518.10411-1-jasowang@redhat.com> References: <20250616082518.10411-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" Use u16 for last_used_idx in virtqueue_poll_split() to align with the spec. Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang Acked-by: Eugenio P=C3=A9rez --- drivers/virtio/virtio_ring.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 03b17bc275dd..b14bbb4d6713 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -801,7 +801,7 @@ static void detach_buf_split(struct vring_virtqueue *vq= , unsigned int head, } =20 static bool virtqueue_poll_split(const struct vring_virtqueue *vq, - unsigned int last_used_idx) + u16 last_used_idx) { return (u16)last_used_idx !=3D virtio16_to_cpu(vq->vq.vdev, vq->split.vring.used->idx); --=20 2.34.1 From nobody Wed Dec 17 03:50:56 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C490C270EA9 for ; Mon, 16 Jun 2025 08:26:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062385; cv=none; b=gFn/ws0sB8eOTzSZCFnU6ATkydXdRf91vk6UC2g45V29xyJr+y9DaN7a+4b/i9RhvYOz7RnhJuVA4RHJonYSDaxGCGvuI0TnVbBXgHovJhcnVJrAbT+qSe4xXdgdBJoFUFdo9vuQo1RqnpSdEOSx8tjh82U1Z10e3BR3ermhlCQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062385; c=relaxed/simple; bh=Ra8Cw8qZAU7d2GcRkLqsSC5/JvQUeW8rl8TIy78VIog=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=c/KDpTqe274sQHwoIGHT75yOHOeW3040MWDFAuJYnLu2vfj5xagSJI3NLvkq7T2wQz5XeLK0FhwKtJyzZhGFrfZzWVg4RL7wz2vHkFRJnZDb5wgiCnh3iA5Gj9PxhS5B5WV/cPMagasQkf25ZJYYu/M9RKUNyPqgpr+VBJO7W+E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Xau7L+wE; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Xau7L+wE" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1750062383; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=crXqaWat/tyK8J/ggb8HTsCRrc2y2f8yDg30nEWbI7Q=; b=Xau7L+wEE6ik8nb0H1J0Mi5PTacgtgaYe4L/UQbX8PEQts1aqqJGgEuq0ebxVAImHsqMwl DFq84BPccNH+jWnYvxpzivJG3of4rMncBXsdxmoJVJwy2mBNGiEP87fG0VOOO/Q0mIYXjr w7HpRgW7LtuSkMEVtHjGT9UV2ijDIsM= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-478-xy0KaYADOiWFCTgHC0OfpQ-1; Mon, 16 Jun 2025 04:26:19 -0400 X-MC-Unique: xy0KaYADOiWFCTgHC0OfpQ-1 X-Mimecast-MFC-AGG-ID: xy0KaYADOiWFCTgHC0OfpQ_1750062378 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id D149619560B5; Mon, 16 Jun 2025 08:26:18 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.72]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 3682E18003FC; Mon, 16 Jun 2025 08:26:14 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com Cc: xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V3 13/19] virtio_ring: introduce virtqueue ops Date: Mon, 16 Jun 2025 16:25:11 +0800 Message-ID: <20250616082518.10411-14-jasowang@redhat.com> In-Reply-To: <20250616082518.10411-1-jasowang@redhat.com> References: <20250616082518.10411-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" This patch introduces virtqueue ops which is a set of the callbacks that will be called for different queue layout or features. This would help to avoid branches for split/packed and will ease the future implementation like in order. Note that in order to eliminate the indirect calls this patch uses global array of const ops to allow compiler to avoid indirect branches. Tested with CONFIG_MITIGATION_RETPOLINE, no performance differences were noticed. Suggested-by: Michael S. Tsirkin Signed-off-by: Jason Wang Acked-by: Eugenio P=C3=A9rez --- drivers/virtio/virtio_ring.c | 172 ++++++++++++++++++++++++++--------- 1 file changed, 129 insertions(+), 43 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index b14bbb4d6713..af32d1a1a1db 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -67,6 +67,12 @@ #define LAST_ADD_TIME_INVALID(vq) #endif =20 +enum vq_layout { + SPLIT =3D 0, + PACKED, + VQ_TYPE_MAX, +}; + struct vring_desc_state_split { void *data; /* Data for callback. */ =20 @@ -159,12 +165,28 @@ struct vring_virtqueue_packed { size_t event_size_in_bytes; }; =20 +struct vring_virtqueue; + +struct virtqueue_ops { + int (*add)(struct vring_virtqueue *_vq, struct scatterlist *sgs[], + unsigned int total_sg, unsigned int out_sgs, + unsigned int in_sgs, void *data, + void *ctx, bool premapped, gfp_t gfp); + void *(*get)(struct vring_virtqueue *vq, unsigned int *len, void **ctx); + bool (*kick_prepare)(struct vring_virtqueue *vq); + void (*disable_cb)(struct vring_virtqueue *vq); + bool (*enable_cb_delayed)(struct vring_virtqueue *vq); + unsigned int (*enable_cb_prepare)(struct vring_virtqueue *vq); + bool (*poll)(const struct vring_virtqueue *vq, u16 last_used_idx); + void *(*detach_unused_buf)(struct vring_virtqueue *vq); + bool (*more_used)(const struct vring_virtqueue *vq); + int (*resize)(struct vring_virtqueue *vq, u32 num); + void (*reset)(struct vring_virtqueue *vq); +}; + struct vring_virtqueue { struct virtqueue vq; =20 - /* Is this a packed ring? */ - bool packed_ring; - /* Is DMA API used? */ bool use_dma_api; =20 @@ -180,6 +202,8 @@ struct vring_virtqueue { /* Host publishes avail event idx */ bool event; =20 + enum vq_layout layout; + /* Head of free buffer list. */ unsigned int free_head; /* Number we've added since last sync. */ @@ -232,6 +256,12 @@ static void vring_free(struct virtqueue *_vq); =20 #define to_vvq(_vq) container_of_const(_vq, struct vring_virtqueue, vq) =20 + +static inline bool virtqueue_is_packed(const struct vring_virtqueue *vq) +{ + return vq->layout =3D=3D PACKED; +} + static bool virtqueue_use_indirect(const struct vring_virtqueue *vq, unsigned int total_sg) { @@ -422,7 +452,7 @@ static void virtqueue_init(struct vring_virtqueue *vq, = u32 num) { vq->vq.num_free =3D num; =20 - if (vq->packed_ring) + if (virtqueue_is_packed(vq)) vq->last_used_idx =3D 0 | (1 << VRING_PACKED_EVENT_F_WRAP_CTR); else vq->last_used_idx =3D 0; @@ -1116,6 +1146,8 @@ static int vring_alloc_queue_split(struct vring_virtq= ueue_split *vring_split, return 0; } =20 +static const struct virtqueue_ops split_ops; + static struct virtqueue *__vring_new_virtqueue_split(unsigned int index, struct vring_virtqueue_split *vring_split, struct virtio_device *vdev, @@ -1133,7 +1165,7 @@ static struct virtqueue *__vring_new_virtqueue_split(= unsigned int index, if (!vq) return NULL; =20 - vq->packed_ring =3D false; + vq->layout =3D SPLIT; vq->vq.callback =3D callback; vq->vq.vdev =3D vdev; vq->vq.name =3D name; @@ -2076,6 +2108,8 @@ static void virtqueue_reset_packed(struct vring_virtq= ueue *vq) virtqueue_vring_init_packed(&vq->packed, !!vq->vq.callback); } =20 +static const struct virtqueue_ops packed_ops; + static struct virtqueue *__vring_new_virtqueue_packed(unsigned int index, struct vring_virtqueue_packed *vring_packed, struct virtio_device *vdev, @@ -2106,7 +2140,7 @@ static struct virtqueue *__vring_new_virtqueue_packed= (unsigned int index, #else vq->broken =3D false; #endif - vq->packed_ring =3D true; + vq->layout =3D PACKED; vq->dma_dev =3D dma_dev; vq->use_dma_api =3D vring_use_dma_api(vdev); =20 @@ -2194,6 +2228,39 @@ static int virtqueue_resize_packed(struct vring_virt= queue *vq, u32 num) return -ENOMEM; } =20 +static const struct virtqueue_ops split_ops =3D { + .add =3D virtqueue_add_split, + .get =3D virtqueue_get_buf_ctx_split, + .kick_prepare =3D virtqueue_kick_prepare_split, + .disable_cb =3D virtqueue_disable_cb_split, + .enable_cb_delayed =3D virtqueue_enable_cb_delayed_split, + .enable_cb_prepare =3D virtqueue_enable_cb_prepare_split, + .poll =3D virtqueue_poll_split, + .detach_unused_buf =3D virtqueue_detach_unused_buf_split, + .more_used =3D more_used_split, + .resize =3D virtqueue_resize_split, + .reset =3D virtqueue_reset_split, +}; + +static const struct virtqueue_ops packed_ops =3D { + .add =3D virtqueue_add_packed, + .get =3D virtqueue_get_buf_ctx_packed, + .kick_prepare =3D virtqueue_kick_prepare_packed, + .disable_cb =3D virtqueue_disable_cb_packed, + .enable_cb_delayed =3D virtqueue_enable_cb_delayed_packed, + .enable_cb_prepare =3D virtqueue_enable_cb_prepare_packed, + .poll =3D virtqueue_poll_packed, + .detach_unused_buf =3D virtqueue_detach_unused_buf_packed, + .more_used =3D more_used_packed, + .resize =3D virtqueue_resize_packed, + .reset =3D virtqueue_reset_packed, +}; + +static const struct virtqueue_ops *const all_ops[VQ_TYPE_MAX] =3D { + [SPLIT] =3D &split_ops, + [PACKED] =3D &packed_ops +}; + static int virtqueue_disable_and_recycle(struct virtqueue *_vq, void (*recycle)(struct virtqueue *vq, void *buf)) { @@ -2236,6 +2303,38 @@ static int virtqueue_enable_after_reset(struct virtq= ueue *_vq) * Generic functions and exported symbols. */ =20 +#define VIRTQUEUE_CALL(vq, op, ...) \ + ({ \ + typeof(all_ops[SPLIT]->op(vq, ##__VA_ARGS__)) ret; \ + switch (vq->layout) { \ + case SPLIT: \ + ret =3D all_ops[SPLIT]->op(vq, ##__VA_ARGS__); \ + break; \ + case PACKED: \ + ret =3D all_ops[PACKED]->op(vq, ##__VA_ARGS__); \ + break; \ + default: \ + BUG(); \ + break; \ + } \ + ret; \ +}) + +#define VOID_VIRTQUEUE_CALL(vq, op, ...) \ + ({ \ + switch ((vq)->layout) { \ + case SPLIT: \ + all_ops[SPLIT]->op(vq, ##__VA_ARGS__); \ + break; \ + case PACKED: \ + all_ops[PACKED]->op(vq, ##__VA_ARGS__); \ + break; \ + default: \ + BUG(); \ + break; \ + } \ +}) + static inline int virtqueue_add(struct virtqueue *_vq, struct scatterlist *sgs[], unsigned int total_sg, @@ -2248,10 +2347,9 @@ static inline int virtqueue_add(struct virtqueue *_v= q, { struct vring_virtqueue *vq =3D to_vvq(_vq); =20 - return vq->packed_ring ? virtqueue_add_packed(vq, sgs, total_sg, - out_sgs, in_sgs, data, ctx, premapped, gfp) : - virtqueue_add_split(vq, sgs, total_sg, - out_sgs, in_sgs, data, ctx, premapped, gfp); + return VIRTQUEUE_CALL(vq, add, sgs, total_sg, + out_sgs, in_sgs, data, + ctx, premapped, gfp); } =20 /** @@ -2437,8 +2535,7 @@ bool virtqueue_kick_prepare(struct virtqueue *_vq) { struct vring_virtqueue *vq =3D to_vvq(_vq); =20 - return vq->packed_ring ? virtqueue_kick_prepare_packed(vq) : - virtqueue_kick_prepare_split(vq); + return VIRTQUEUE_CALL(vq, kick_prepare); } EXPORT_SYMBOL_GPL(virtqueue_kick_prepare); =20 @@ -2508,8 +2605,7 @@ void *virtqueue_get_buf_ctx(struct virtqueue *_vq, un= signed int *len, { struct vring_virtqueue *vq =3D to_vvq(_vq); =20 - return vq->packed_ring ? virtqueue_get_buf_ctx_packed(vq, len, ctx) : - virtqueue_get_buf_ctx_split(vq, len, ctx); + return VIRTQUEUE_CALL(vq, get, len, ctx); } EXPORT_SYMBOL_GPL(virtqueue_get_buf_ctx); =20 @@ -2531,10 +2627,7 @@ void virtqueue_disable_cb(struct virtqueue *_vq) { struct vring_virtqueue *vq =3D to_vvq(_vq); =20 - if (vq->packed_ring) - virtqueue_disable_cb_packed(vq); - else - virtqueue_disable_cb_split(vq); + VOID_VIRTQUEUE_CALL(vq, disable_cb); } EXPORT_SYMBOL_GPL(virtqueue_disable_cb); =20 @@ -2557,8 +2650,7 @@ unsigned int virtqueue_enable_cb_prepare(struct virtq= ueue *_vq) if (vq->event_triggered) vq->event_triggered =3D false; =20 - return vq->packed_ring ? virtqueue_enable_cb_prepare_packed(vq) : - virtqueue_enable_cb_prepare_split(vq); + return VIRTQUEUE_CALL(vq, enable_cb_prepare); } EXPORT_SYMBOL_GPL(virtqueue_enable_cb_prepare); =20 @@ -2579,8 +2671,8 @@ bool virtqueue_poll(struct virtqueue *_vq, unsigned i= nt last_used_idx) return false; =20 virtio_mb(vq->weak_barriers); - return vq->packed_ring ? virtqueue_poll_packed(vq, last_used_idx) : - virtqueue_poll_split(vq, last_used_idx); + + return VIRTQUEUE_CALL(vq, poll, last_used_idx); } EXPORT_SYMBOL_GPL(virtqueue_poll); =20 @@ -2623,8 +2715,7 @@ bool virtqueue_enable_cb_delayed(struct virtqueue *_v= q) if (vq->event_triggered) data_race(vq->event_triggered =3D false); =20 - return vq->packed_ring ? virtqueue_enable_cb_delayed_packed(vq) : - virtqueue_enable_cb_delayed_split(vq); + return VIRTQUEUE_CALL(vq, enable_cb_delayed); } EXPORT_SYMBOL_GPL(virtqueue_enable_cb_delayed); =20 @@ -2640,14 +2731,13 @@ void *virtqueue_detach_unused_buf(struct virtqueue = *_vq) { struct vring_virtqueue *vq =3D to_vvq(_vq); =20 - return vq->packed_ring ? virtqueue_detach_unused_buf_packed(vq) : - virtqueue_detach_unused_buf_split(vq); + return VIRTQUEUE_CALL(vq, detach_unused_buf); } EXPORT_SYMBOL_GPL(virtqueue_detach_unused_buf); =20 static inline bool more_used(const struct vring_virtqueue *vq) { - return vq->packed_ring ? more_used_packed(vq) : more_used_split(vq); + return VIRTQUEUE_CALL(vq, more_used); } =20 /** @@ -2776,7 +2866,8 @@ int virtqueue_resize(struct virtqueue *_vq, u32 num, if (!num) return -EINVAL; =20 - if ((vq->packed_ring ? vq->packed.vring.num : vq->split.vring.num) =3D=3D= num) + if ((virtqueue_is_packed(vq) ? vq->packed.vring.num : + vq->split.vring.num) =3D=3D num) return 0; =20 err =3D virtqueue_disable_and_recycle(_vq, recycle); @@ -2785,10 +2876,7 @@ int virtqueue_resize(struct virtqueue *_vq, u32 num, if (recycle_done) recycle_done(_vq); =20 - if (vq->packed_ring) - err =3D virtqueue_resize_packed(vq, num); - else - err =3D virtqueue_resize_split(vq, num); + err =3D VIRTQUEUE_CALL(vq, resize, num); =20 return virtqueue_enable_after_reset(_vq); } @@ -2822,10 +2910,7 @@ int virtqueue_reset(struct virtqueue *_vq, if (recycle_done) recycle_done(_vq); =20 - if (vq->packed_ring) - virtqueue_reset_packed(vq); - else - virtqueue_reset_split(vq); + VOID_VIRTQUEUE_CALL(vq, reset); =20 return virtqueue_enable_after_reset(_vq); } @@ -2867,7 +2952,7 @@ static void vring_free(struct virtqueue *_vq) struct vring_virtqueue *vq =3D to_vvq(_vq); =20 if (vq->we_own_ring) { - if (vq->packed_ring) { + if (virtqueue_is_packed(vq)) { vring_free_queue(vq->vq.vdev, vq->packed.ring_size_in_bytes, vq->packed.vring.desc, @@ -2896,7 +2981,7 @@ static void vring_free(struct virtqueue *_vq) vring_dma_dev(vq)); } } - if (!vq->packed_ring) { + if (!virtqueue_is_packed(vq)) { kfree(vq->split.desc_state); kfree(vq->split.desc_extra); } @@ -2921,7 +3006,7 @@ u32 vring_notification_data(struct virtqueue *_vq) struct vring_virtqueue *vq =3D to_vvq(_vq); u16 next; =20 - if (vq->packed_ring) + if (virtqueue_is_packed(vq)) next =3D (vq->packed.next_avail_idx & ~(-(1 << VRING_PACKED_EVENT_F_WRAP_CTR))) | vq->packed.avail_wrap_counter << @@ -2974,7 +3059,8 @@ unsigned int virtqueue_get_vring_size(const struct vi= rtqueue *_vq) =20 const struct vring_virtqueue *vq =3D to_vvq(_vq); =20 - return vq->packed_ring ? vq->packed.vring.num : vq->split.vring.num; + return virtqueue_is_packed(vq) ? vq->packed.vring.num : + vq->split.vring.num; } EXPORT_SYMBOL_GPL(virtqueue_get_vring_size); =20 @@ -3057,7 +3143,7 @@ dma_addr_t virtqueue_get_desc_addr(const struct virtq= ueue *_vq) =20 BUG_ON(!vq->we_own_ring); =20 - if (vq->packed_ring) + if (virtqueue_is_packed(vq)) return vq->packed.ring_dma_addr; =20 return vq->split.queue_dma_addr; @@ -3070,7 +3156,7 @@ dma_addr_t virtqueue_get_avail_addr(const struct virt= queue *_vq) =20 BUG_ON(!vq->we_own_ring); =20 - if (vq->packed_ring) + if (virtqueue_is_packed(vq)) return vq->packed.driver_event_dma_addr; =20 return vq->split.queue_dma_addr + @@ -3084,7 +3170,7 @@ dma_addr_t virtqueue_get_used_addr(const struct virtq= ueue *_vq) =20 BUG_ON(!vq->we_own_ring); =20 - if (vq->packed_ring) + if (virtqueue_is_packed(vq)) return vq->packed.device_event_dma_addr; =20 return vq->split.queue_dma_addr + --=20 2.34.1 From nobody Wed Dec 17 03:50:56 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 74AF826E71E for ; Mon, 16 Jun 2025 08:26:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062390; cv=none; b=lWYWgBTCZu3X/EB6FV9TKUHawIJtedLS5mTL5hSB5MbDt/pHm/34P0WOqvS3qafYnbG6OyUckioyLz8vVwQLCwg4VRlVJBiVxYSRI7nGaK57I38hTcxsjM96joyiBQ/dgJRJ0bFAEFubrcTHkxaDjomLiGbOsHwy+9hMFg6wsDo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062390; c=relaxed/simple; bh=7ibluwZ05Ku8GxFsKg/HCDY7lLwD+ET4m094INhsKUo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nKA/tLrndYLhjZoEGQzXNpzsDgGytBGhhZH/1bgCHVWYjU4T4bDjDSv+yPfwga9irAS2wUdRbuXxwst4KKhUpL8t7VIH5mYlGraDy+nxtjBPojHvsh7WRXwMjsfsnYF6bUP3Xs9g3GnWC8oTFMoOO/ZGTXbillDqbbWyCbYfpVc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=NVdi/BM+; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="NVdi/BM+" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1750062387; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nm3UmQwERzvwBQjnk71E9B+OidwwIVFS/Ow+WmEslM4=; b=NVdi/BM+A8r1K3V5mABq84wGZWWAd9mN+oCsAsPBusfxp6g6lCC5pink/NtvAKRN+x8x2l 8O0GfZl/EGiLDKnhRUXtEpxilTAUOJH+WbScg1pkhUyI5cTwywC1C6jb71kQYuuqUnvjk7 rMk+EpzOEF6quI7fvap/Zt7JDVj1UeA= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-121-AsT5ybUWOim3_H1gF9UZtw-1; Mon, 16 Jun 2025 04:26:23 -0400 X-MC-Unique: AsT5ybUWOim3_H1gF9UZtw-1 X-Mimecast-MFC-AGG-ID: AsT5ybUWOim3_H1gF9UZtw_1750062382 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id B50BF19560BC; Mon, 16 Jun 2025 08:26:22 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.72]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 90D1B180045C; Mon, 16 Jun 2025 08:26:19 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com Cc: xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V3 14/19] virtio_ring: determine descriptor flags at one time Date: Mon, 16 Jun 2025 16:25:12 +0800 Message-ID: <20250616082518.10411-15-jasowang@redhat.com> In-Reply-To: <20250616082518.10411-1-jasowang@redhat.com> References: <20250616082518.10411-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" Let's determine the last descriptor by counting the number of sg. This would be consistent with packed virtqueue implementation and ease the future in-order implementation. Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang Acked-by: Eugenio P=C3=A9rez --- drivers/virtio/virtio_ring.c | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index af32d1a1a1db..d5e4d4cd2487 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -570,7 +570,7 @@ static inline int virtqueue_add_split(struct vring_virt= queue *vq, struct vring_desc_extra *extra; struct scatterlist *sg; struct vring_desc *desc; - unsigned int i, n, avail, descs_used, prev, err_idx; + unsigned int i, n, c, avail, descs_used, err_idx; int head; bool indirect; =20 @@ -626,46 +626,47 @@ static inline int virtqueue_add_split(struct vring_vi= rtqueue *vq, return -ENOSPC; } =20 + c =3D 0; for (n =3D 0; n < out_sgs; n++) { + sg =3D sgs[n]; for (sg =3D sgs[n]; sg; sg =3D sg_next(sg)) { dma_addr_t addr; u32 len; + u16 flags =3D 0; =20 if (vring_map_one_sg(vq, sg, DMA_TO_DEVICE, &addr, &len, premapped)) goto unmap_release; =20 - prev =3D i; + if (++c !=3D total_sg) + flags =3D VRING_DESC_F_NEXT; + /* Note that we trust indirect descriptor * table since it use stream DMA mapping. */ i =3D virtqueue_add_desc_split(vq, desc, extra, i, addr, len, - VRING_DESC_F_NEXT, + flags, premapped); } } for (; n < (out_sgs + in_sgs); n++) { for (sg =3D sgs[n]; sg; sg =3D sg_next(sg)) { + u16 flags =3D VRING_DESC_F_WRITE; dma_addr_t addr; u32 len; =20 if (vring_map_one_sg(vq, sg, DMA_FROM_DEVICE, &addr, &len, premapped)) goto unmap_release; =20 - prev =3D i; + if (++c !=3D total_sg) + flags |=3D VRING_DESC_F_NEXT; + /* Note that we trust indirect descriptor * table since it use stream DMA mapping. */ i =3D virtqueue_add_desc_split(vq, desc, extra, i, addr, len, - VRING_DESC_F_NEXT | - VRING_DESC_F_WRITE, - premapped); + flags, premapped); } } - /* Last one doesn't continue. */ - desc[prev].flags &=3D cpu_to_virtio16(vq->vq.vdev, ~VRING_DESC_F_NEXT); - if (!indirect && vring_need_unmap_buffer(vq, &extra[prev])) - vq->split.desc_extra[prev & (vq->split.vring.num - 1)].flags &=3D - ~VRING_DESC_F_NEXT; =20 if (indirect) { /* Now that the indirect table is filled in, map it. */ --=20 2.34.1 From nobody Wed Dec 17 03:50:56 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC24F27144D for ; Mon, 16 Jun 2025 08:26:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062393; cv=none; b=FKPQc0qhaRaphNvfTqvrwFjF69XrsYq7pD9q/KiIWFn/NMCts27kc1eH7niH9NPOQwjYATLePDylDNMYN401xzIYg2XqKJ4dxm0lmIk7LaUBBTjMKh/S0JrLRIoDDtX+Q+TLbI9wKuT9uQZH2SaIpwuVkMQi+RIuoVu5laxpwN0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062393; c=relaxed/simple; bh=V7iSblPSIgq8FJtLMhJzBmXPmaGVxdkqHLDwwhsL/5s=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=H+qCt8tZ5jWy/h62a8vIBXKu4LC7DMGNSAxL+FWZF0d4c7t8lNoIthtALpSbKXqPlTs/vaiJ+JsaqdYjQKs0/sJwEsnnCtIrzrYDFLXODOQunKxoLtbut2yNVxe1SB8qjLR5AkUNL1vUtxCGPCeOmbyURBrDukKr7y8Ac9P1bfk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=GueIMX4e; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="GueIMX4e" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1750062391; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OkHJ3Ok9zOyAKdcXuaYeCidcNG8+6ctRMQN9YsSqAXk=; b=GueIMX4eD5aU2ssTzLOR7AK52GnDI1ejnDJ9dbUnaXMnGLCZy7vyQgsbUUmGrhdLJHYeNW YhI9qauvR72iBKYgeed+xhF2Z9m8k2mo0WV5mrxagwnbZwb5GQsXxr3+3Pi9yXB3wRM5fc GRKBEGVir3ix/zeJLscOxbztjk7z17c= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-218-nuI9y9mjNNKIZnEMcX310A-1; Mon, 16 Jun 2025 04:26:27 -0400 X-MC-Unique: nuI9y9mjNNKIZnEMcX310A-1 X-Mimecast-MFC-AGG-ID: nuI9y9mjNNKIZnEMcX310A_1750062386 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id CADCC180028B; Mon, 16 Jun 2025 08:26:26 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.72]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 83D0F180045B; Mon, 16 Jun 2025 08:26:23 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com Cc: xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V3 15/19] virtio_ring: factor out core logic of buffer detaching Date: Mon, 16 Jun 2025 16:25:13 +0800 Message-ID: <20250616082518.10411-16-jasowang@redhat.com> In-Reply-To: <20250616082518.10411-1-jasowang@redhat.com> References: <20250616082518.10411-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" Factor out core logic of buffer detaching and leave the id population to the caller so in order can just call the core logic. Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang Acked-by: Eugenio P=C3=A9rez --- drivers/virtio/virtio_ring.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index d5e4d4cd2487..1bdd332d515e 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -1669,8 +1669,8 @@ static bool virtqueue_kick_prepare_packed(struct vrin= g_virtqueue *vq) return needs_kick; } =20 -static void detach_buf_packed(struct vring_virtqueue *vq, - unsigned int id, void **ctx) +static void detach_buf_packed_in_order(struct vring_virtqueue *vq, + unsigned int id, void **ctx) { struct vring_desc_state_packed *state =3D NULL; struct vring_packed_desc *desc; @@ -1681,8 +1681,6 @@ static void detach_buf_packed(struct vring_virtqueue = *vq, /* Clear data ptr. */ state->data =3D NULL; =20 - vq->packed.desc_extra[state->last].next =3D vq->free_head; - vq->free_head =3D id; vq->vq.num_free +=3D state->num; =20 if (unlikely(vq->use_dma_api)) { @@ -1719,6 +1717,17 @@ static void detach_buf_packed(struct vring_virtqueue= *vq, } } =20 +static void detach_buf_packed(struct vring_virtqueue *vq, + unsigned int id, void **ctx) +{ + struct vring_desc_state_packed *state =3D &vq->packed.desc_state[id]; + + vq->packed.desc_extra[state->last].next =3D vq->free_head; + vq->free_head =3D id; + + return detach_buf_packed_in_order(vq, id, ctx); +} + static inline bool is_used_desc_packed(const struct vring_virtqueue *vq, u16 idx, bool used_wrap_counter) { --=20 2.34.1 From nobody Wed Dec 17 03:50:56 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D234E26E143 for ; Mon, 16 Jun 2025 08:26:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062400; cv=none; b=qjuSpEhnuIjLuWrTI1DrxX/hzoitCni6J+daPTI1yeYdnBLOEthf2kS3a9BjqHnsOmxys9x9zdzqZAwai5/jhCjWWRdv1mpMpY0c2FzYdz0PqIjYXRDOWyf7v0hyx+/sqcatT95jRTpNhdkRfYF+a06LO4h1BZ3HL4bRd6aR/uE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062400; c=relaxed/simple; bh=6qM3MRfri/1e0rmpMhrib0LrcCkVIs1t7ATyLsuudVA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hjWOdShJSd4557pS6hDNnvEZHN5+LVr+7/oAJDpiFKfoBX88KZ3gv3Yin8eHPGRcTYlp6nVDznJlJxCfmk8sJc8o8k60DcRPTfk9v4K4tka0SVRZFruXKCy+xuTyfvuRAXi7dLPtaVsvK/o3WdCQWehiaLHCib9o/TJZF4ByouI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=BNdG+Sew; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="BNdG+Sew" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1750062396; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GvwJpvAV0V8j8IC4lojknQV0rvDLo94+r3JtkXAzhRg=; b=BNdG+SewRQ9yFMYuw8B2o0vtejpiYMq6x20ZXIEMzjmWuPhsBiYur36T3dNvG1f+r1l7YT du9zKE8lzQSYLndjlvCu74vZ3Bf87Vw+lPw9bE10klQXFE5SB5GHutuwC4Us7g/pyMcBA2 3TuwQDqEM6cIz77X+Fi9kdnD1qV1r8E= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-333-bSqgSvcfP5SvZuX7nUToCg-1; Mon, 16 Jun 2025 04:26:31 -0400 X-MC-Unique: bSqgSvcfP5SvZuX7nUToCg-1 X-Mimecast-MFC-AGG-ID: bSqgSvcfP5SvZuX7nUToCg_1750062390 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 9A9D5195608C; Mon, 16 Jun 2025 08:26:30 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.72]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 8C08418003FC; Mon, 16 Jun 2025 08:26:27 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com Cc: xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V3 16/19] virtio_ring: factor out core logic for updating last_used_idx Date: Mon, 16 Jun 2025 16:25:14 +0800 Message-ID: <20250616082518.10411-17-jasowang@redhat.com> In-Reply-To: <20250616082518.10411-1-jasowang@redhat.com> References: <20250616082518.10411-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" Factor out the core logic for updating last_used_idx to be reused by the packed in order implementation. Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang Acked-by: Eugenio P=C3=A9rez --- drivers/virtio/virtio_ring.c | 43 +++++++++++++++++++++--------------- 1 file changed, 25 insertions(+), 18 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 1bdd332d515e..e8e0d1204f52 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -1757,6 +1757,30 @@ static bool more_used_packed(const struct vring_virt= queue *vq) return virtqueue_poll_packed(vq, READ_ONCE(vq->last_used_idx)); } =20 +static void update_last_used_idx_packed(struct vring_virtqueue *vq, + u16 id, u16 last_used, + u16 used_wrap_counter) +{ + last_used +=3D vq->packed.desc_state[id].num; + if (unlikely(last_used >=3D vq->packed.vring.num)) { + last_used -=3D vq->packed.vring.num; + used_wrap_counter ^=3D 1; + } + + last_used =3D (last_used | (used_wrap_counter << VRING_PACKED_EVENT_F_WRA= P_CTR)); + WRITE_ONCE(vq->last_used_idx, last_used); + + /* + * If we expect an interrupt for the next entry, tell host + * by writing event index and flush out the write before + * the read in the next get_buf call. + */ + if (vq->packed.event_flags_shadow =3D=3D VRING_PACKED_EVENT_FLAG_DESC) + virtio_store_mb(vq->weak_barriers, + &vq->packed.vring.driver->off_wrap, + cpu_to_le16(vq->last_used_idx)); +} + static void *virtqueue_get_buf_ctx_packed(struct vring_virtqueue *vq, unsigned int *len, void **ctx) @@ -1800,24 +1824,7 @@ static void *virtqueue_get_buf_ctx_packed(struct vri= ng_virtqueue *vq, ret =3D vq->packed.desc_state[id].data; detach_buf_packed(vq, id, ctx); =20 - last_used +=3D vq->packed.desc_state[id].num; - if (unlikely(last_used >=3D vq->packed.vring.num)) { - last_used -=3D vq->packed.vring.num; - used_wrap_counter ^=3D 1; - } - - last_used =3D (last_used | (used_wrap_counter << VRING_PACKED_EVENT_F_WRA= P_CTR)); - WRITE_ONCE(vq->last_used_idx, last_used); - - /* - * If we expect an interrupt for the next entry, tell host - * by writing event index and flush out the write before - * the read in the next get_buf call. - */ - if (vq->packed.event_flags_shadow =3D=3D VRING_PACKED_EVENT_FLAG_DESC) - virtio_store_mb(vq->weak_barriers, - &vq->packed.vring.driver->off_wrap, - cpu_to_le16(vq->last_used_idx)); + update_last_used_idx_packed(vq, id, last_used, used_wrap_counter); =20 LAST_ADD_TIME_INVALID(vq); =20 --=20 2.34.1 From nobody Wed Dec 17 03:50:56 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B8F7C26E142 for ; Mon, 16 Jun 2025 08:26:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062402; cv=none; b=o8LMhVPLTBxk1PxYGcbaMqcoDZjllrRLAIbohFXwZQhtzSJ/libpVO0GxrtnQIHhznFkvGMq8yx4k2N8QTaxPQXGdDWO8V5sVch7x0efRLYQmGnBRLQhQYcTSUjJ01cRe2+X/LhvER2AkOR5TlA3+HYFSLQ+c7mYT3/Bb/ZdiEo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062402; c=relaxed/simple; bh=hDuc70GQ7P1V+yYqAUaSHOIFjrt3xaxT1gTDE+lfbdk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=jqksChMGkmG/xkYt3fDvoJ/55mv1GA7aknBljJtHsq5LklzbgiW57UbQhfb3ucLVKt4dPgrQDIXoUwFhAq7rcwlqqQ0KEUHW8CkE/Di45RRS7SfX8meXWbehOESNksqXFQsxtBUDK+oSSaSXa2DhDN7aJlc+ajvfs0AxbYI+E90= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=ha5mtTK9; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="ha5mtTK9" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1750062399; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gD+qTXbPzPYyO4+gwqdolI6YPrwMgHy4NQjRlgrLOBE=; b=ha5mtTK913XuZlV+4WohO8UfrPwh8abYWi9EPsQWaO5mvjTV9dGHl9gFUN43ECxcmTc1ap DeEPJW52zUW0xz+9iW0oM8+8z2HO3x5B6ojncUgBTNt/fvOg4IHS+eJUUOS+MeOOQnPgtu L+2Hx8RPxdkOPzo3CkS8/JSsbdSkdZY= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-550-mq1C_tR_NdqcElGRECt4Zw-1; Mon, 16 Jun 2025 04:26:36 -0400 X-MC-Unique: mq1C_tR_NdqcElGRECt4Zw-1 X-Mimecast-MFC-AGG-ID: mq1C_tR_NdqcElGRECt4Zw_1750062395 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 5A9331956080; Mon, 16 Jun 2025 08:26:35 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.72]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 0319318003FC; Mon, 16 Jun 2025 08:26:30 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com Cc: xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V3 17/19] virtio_ring: factor out split indirect detaching logic Date: Mon, 16 Jun 2025 16:25:15 +0800 Message-ID: <20250616082518.10411-18-jasowang@redhat.com> In-Reply-To: <20250616082518.10411-1-jasowang@redhat.com> References: <20250616082518.10411-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" Factor out the split indirect descriptor detaching logic in order to make it be reused by the in order support. Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang Acked-by: Eugenio P=C3=A9rez --- drivers/virtio/virtio_ring.c | 63 ++++++++++++++++++++---------------- 1 file changed, 35 insertions(+), 28 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index e8e0d1204f52..259380797ec4 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -773,11 +773,42 @@ static bool virtqueue_kick_prepare_split(struct vring= _virtqueue *vq) return needs_kick; } =20 +static void detach_indirect_split(struct vring_virtqueue *vq, + unsigned int head) +{ + struct vring_desc_extra *extra =3D vq->split.desc_extra; + struct vring_desc *indir_desc =3D + vq->split.desc_state[head].indir_desc; + unsigned int j; + u32 len, num; + + /* Free the indirect table, if any, now that it's unmapped. */ + if (!indir_desc) + return; + len =3D vq->split.desc_extra[head].len; + + BUG_ON(!(vq->split.desc_extra[head].flags & + VRING_DESC_F_INDIRECT)); + BUG_ON(len =3D=3D 0 || len % sizeof(struct vring_desc)); + + num =3D len / sizeof(struct vring_desc); + + extra =3D (struct vring_desc_extra *)&indir_desc[num]; + + if (vq->use_dma_api) { + for (j =3D 0; j < num; j++) + vring_unmap_one_split(vq, &extra[j]); + } + + kfree(indir_desc); + vq->split.desc_state[head].indir_desc =3D NULL; +} + static void detach_buf_split(struct vring_virtqueue *vq, unsigned int head, void **ctx) { struct vring_desc_extra *extra; - unsigned int i, j; + unsigned int i; __virtio16 nextflag =3D cpu_to_virtio16(vq->vq.vdev, VRING_DESC_F_NEXT); =20 /* Clear data ptr. */ @@ -801,34 +832,10 @@ static void detach_buf_split(struct vring_virtqueue *= vq, unsigned int head, /* Plus final descriptor */ vq->vq.num_free++; =20 - if (vq->indirect) { - struct vring_desc *indir_desc =3D - vq->split.desc_state[head].indir_desc; - u32 len, num; - - /* Free the indirect table, if any, now that it's unmapped. */ - if (!indir_desc) - return; - len =3D vq->split.desc_extra[head].len; - - BUG_ON(!(vq->split.desc_extra[head].flags & - VRING_DESC_F_INDIRECT)); - BUG_ON(len =3D=3D 0 || len % sizeof(struct vring_desc)); - - num =3D len / sizeof(struct vring_desc); - - extra =3D (struct vring_desc_extra *)&indir_desc[num]; - - if (vq->use_dma_api) { - for (j =3D 0; j < num; j++) - vring_unmap_one_split(vq, &extra[j]); - } - - kfree(indir_desc); - vq->split.desc_state[head].indir_desc =3D NULL; - } else if (ctx) { + if (vq->indirect) + detach_indirect_split(vq, head); + else if (ctx) *ctx =3D vq->split.desc_state[head].indir_desc; - } } =20 static bool virtqueue_poll_split(const struct vring_virtqueue *vq, --=20 2.34.1 From nobody Wed Dec 17 03:50:56 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BF93E2727F8 for ; Mon, 16 Jun 2025 08:26:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062409; cv=none; b=bgdX5YnlxsPSseR0ZSMjV0HG+AfjnM3RTrWVm2KDsa2fzhqgwXUwTKf+mTtl2o6dkI/Gid4KkutGIPq212BuxnLWfFAEGPYDOD2hrI54kOuYUcs2HScLuXIWFgRi3U5LEEcbZg/UJuVjOAWs49frWqbs1ryfaLwNqo4c0NCjAeg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062409; c=relaxed/simple; bh=VkMffjC3br4KqWb9F1BjbMrO6wT3d18hmzyLnKHBA0c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YDxIziVeQk7ezwrOaGLbWOJ1A6f4anZxcm+T5hCPtxaM+r3XoqB2iJBFqPhaP0mq3kOeo+dU6a7R1QUZ3w/PPO8kdI4DwKJET9hJCNzoks4A2nj2xixhAHmaV9YjuhIeREE8akd4VOy0BvtLISlZAGmWU+uyt4yZzN64V+tJNyY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=QaNjJlNV; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="QaNjJlNV" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1750062403; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zjU0N2LMlgapfpld7SchegzzL+cNmGtiDFPxiXYw+T0=; b=QaNjJlNVafJW9h2T70vo78u0uN8uZC4Kl+2cDUJHTHVgHuB42+KkctJzgLSlGwd/QruqRO SuBjFtAEYG64KaLc/W3kc6I4GglICeJlTjLjZT2W9lyXLIY0T1LRxKa3zWRz0f5BBiTKH9 VwOcRhqFIRBFSmhHasrQMrUv0eq73sE= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-249-N3rQzruDMFy6UQJY9VZqWA-1; Mon, 16 Jun 2025 04:26:40 -0400 X-MC-Unique: N3rQzruDMFy6UQJY9VZqWA-1 X-Mimecast-MFC-AGG-ID: N3rQzruDMFy6UQJY9VZqWA_1750062399 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 39591195608C; Mon, 16 Jun 2025 08:26:39 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.72]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 1A366180045B; Mon, 16 Jun 2025 08:26:35 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com Cc: xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V3 18/19] virtio_ring: factor out split detaching logic Date: Mon, 16 Jun 2025 16:25:16 +0800 Message-ID: <20250616082518.10411-19-jasowang@redhat.com> In-Reply-To: <20250616082518.10411-1-jasowang@redhat.com> References: <20250616082518.10411-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" This patch factors out the split core detaching logic that could be reused by in order feature into a dedicated function. Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang Acked-by: Eugenio P=C3=A9rez --- drivers/virtio/virtio_ring.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 259380797ec4..27a9459a0555 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -804,8 +804,9 @@ static void detach_indirect_split(struct vring_virtqueu= e *vq, vq->split.desc_state[head].indir_desc =3D NULL; } =20 -static void detach_buf_split(struct vring_virtqueue *vq, unsigned int head, - void **ctx) +static unsigned detach_buf_split_in_order(struct vring_virtqueue *vq, + unsigned int head, + void **ctx) { struct vring_desc_extra *extra; unsigned int i; @@ -826,8 +827,6 @@ static void detach_buf_split(struct vring_virtqueue *vq= , unsigned int head, } =20 vring_unmap_one_split(vq, &extra[i]); - vq->split.desc_extra[i].next =3D vq->free_head; - vq->free_head =3D head; =20 /* Plus final descriptor */ vq->vq.num_free++; @@ -836,6 +835,17 @@ static void detach_buf_split(struct vring_virtqueue *v= q, unsigned int head, detach_indirect_split(vq, head); else if (ctx) *ctx =3D vq->split.desc_state[head].indir_desc; + + return i; +} + +static void detach_buf_split(struct vring_virtqueue *vq, unsigned int head, + void **ctx) +{ + unsigned int i =3D detach_buf_split_in_order(vq, head, ctx); + + vq->split.desc_extra[i].next =3D vq->free_head; + vq->free_head =3D head; } =20 static bool virtqueue_poll_split(const struct vring_virtqueue *vq, --=20 2.34.1 From nobody Wed Dec 17 03:50:56 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0D93326FA5C for ; Mon, 16 Jun 2025 08:26:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062411; cv=none; b=BCJPPL/t4o/ruKgqQGkxiekG/QjaG6baU5fuu/YguHV566nlke7jek2vNnnGhLnnyh3wQbDoL/XVrdS/a02PtCPETIvAuMcqxVot0psPW6/NEx7ioe20a6skyPxb6FHwsHARIPM+yUOAhHlERT/Q8gvBKnUiHCoySn6aWNBU2Zs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750062411; c=relaxed/simple; bh=N7ZrFk0MojvVI3HwegZPtcPdY3fD65D8XCyLncIapso=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YEhfI/bvfnSiwzkuoF5BVCLqjhsZ0/EEfiNTgEBR28gSs/arJqvYw8qD/qFrJsiLmrIq5a3c4H96wKkXLKM71VCgCEnxoFQgoRJTDAupGPR2w/z9ZoBODjik3DGMJcGNc4md8kHOHDG2NUE7B/AYdJZsdtGURjPDOvCv+o9D1cE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=KP2lfEQM; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="KP2lfEQM" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1750062408; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Kas7id6StVSvHa+0oZoUtaSrjNfN4yruC+OlGkYlj2E=; b=KP2lfEQM2w7Ib2RzD3XYf9kcYjj75T4l0vfoo1MJggT2FdsLaRcCvgkLROvrBzd28wB216 tsggQ9WsjgQZILpL8gE0uyBzVMQsr//5QM0SPtaqYaTTcbx6kOo7Ov1gEV6oAG7UG5+jNO Dcwh1hxehOs4XLr0vgfxTxOsGABp5uo= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-22-5c-q-AnYM6WY2tObDdHTsQ-1; Mon, 16 Jun 2025 04:26:44 -0400 X-MC-Unique: 5c-q-AnYM6WY2tObDdHTsQ-1 X-Mimecast-MFC-AGG-ID: 5c-q-AnYM6WY2tObDdHTsQ_1750062403 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 0DF90180034E; Mon, 16 Jun 2025 08:26:43 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.72]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id EDFF118003FC; Mon, 16 Jun 2025 08:26:39 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com Cc: xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V3 19/19] virtio_ring: add in order support Date: Mon, 16 Jun 2025 16:25:17 +0800 Message-ID: <20250616082518.10411-20-jasowang@redhat.com> In-Reply-To: <20250616082518.10411-1-jasowang@redhat.com> References: <20250616082518.10411-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" This patch implements in order support for both split virtqueue and packed virtqueue. Benchmark with KVM guest + testpmd on the host shows: For split virtqueue: no obvious differences were noticed For packed virtqueue: 1) RX gets 3.1% PPS improvements from 6.3 Mpps to 6.5 Mpps 2) TX gets 4.6% PPS improvements from 8.6 Mpps to 9.0 Mpps Signed-off-by: Jason Wang Acked-by: Eugenio P=C3=A9rez --- drivers/virtio/virtio_ring.c | 423 +++++++++++++++++++++++++++++++++-- 1 file changed, 402 insertions(+), 21 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 27a9459a0555..21d456392ba0 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -70,11 +70,14 @@ enum vq_layout { SPLIT =3D 0, PACKED, + SPLIT_IN_ORDER, + PACKED_IN_ORDER, VQ_TYPE_MAX, }; =20 struct vring_desc_state_split { void *data; /* Data for callback. */ + u32 total_len; /* Buffer Length */ =20 /* Indirect desc table and extra table, if any. These two will be * allocated together. So we won't stress more to the memory allocator. @@ -84,6 +87,7 @@ struct vring_desc_state_split { =20 struct vring_desc_state_packed { void *data; /* Data for callback. */ + u32 total_len; /* Buffer Length */ =20 /* Indirect desc table and extra table, if any. These two will be * allocated together. So we won't stress more to the memory allocator. @@ -206,6 +210,12 @@ struct vring_virtqueue { =20 /* Head of free buffer list. */ unsigned int free_head; + + /* Head of the batched used buffers, vq->num means no batching */ + unsigned int batch_head; + + unsigned int batch_len; + /* Number we've added since last sync. */ unsigned int num_added; =20 @@ -256,10 +266,14 @@ static void vring_free(struct virtqueue *_vq); =20 #define to_vvq(_vq) container_of_const(_vq, struct vring_virtqueue, vq) =20 - static inline bool virtqueue_is_packed(const struct vring_virtqueue *vq) { - return vq->layout =3D=3D PACKED; + return vq->layout =3D=3D PACKED || vq->layout =3D=3D PACKED_IN_ORDER; +} + +static inline bool virtqueue_is_in_order(const struct vring_virtqueue *vq) +{ + return vq->layout =3D=3D SPLIT_IN_ORDER || vq->layout =3D=3D PACKED_IN_OR= DER; } =20 static bool virtqueue_use_indirect(const struct vring_virtqueue *vq, @@ -570,7 +584,7 @@ static inline int virtqueue_add_split(struct vring_virt= queue *vq, struct vring_desc_extra *extra; struct scatterlist *sg; struct vring_desc *desc; - unsigned int i, n, c, avail, descs_used, err_idx; + unsigned int i, n, c, avail, descs_used, err_idx, total_len =3D 0; int head; bool indirect; =20 @@ -646,6 +660,7 @@ static inline int virtqueue_add_split(struct vring_virt= queue *vq, i =3D virtqueue_add_desc_split(vq, desc, extra, i, addr, len, flags, premapped); + total_len +=3D len; } } for (; n < (out_sgs + in_sgs); n++) { @@ -665,6 +680,7 @@ static inline int virtqueue_add_split(struct vring_virt= queue *vq, */ i =3D virtqueue_add_desc_split(vq, desc, extra, i, addr, len, flags, premapped); + total_len +=3D len; } } =20 @@ -687,7 +703,12 @@ static inline int virtqueue_add_split(struct vring_vir= tqueue *vq, vq->vq.num_free -=3D descs_used; =20 /* Update free pointer */ - if (indirect) + if (virtqueue_is_in_order(vq)) { + vq->free_head +=3D descs_used; + if (vq->free_head >=3D vq->split.vring.num) + vq->free_head -=3D vq->split.vring.num; + vq->split.desc_state[head].total_len =3D total_len;; + } else if (indirect) vq->free_head =3D vq->split.desc_extra[head].next; else vq->free_head =3D i; @@ -860,6 +881,14 @@ static bool more_used_split(const struct vring_virtque= ue *vq) return virtqueue_poll_split(vq, vq->last_used_idx); } =20 +static bool more_used_split_in_order(const struct vring_virtqueue *vq) +{ + if (vq->batch_head !=3D vq->packed.vring.num) + return true; + + return virtqueue_poll_split(vq, vq->last_used_idx); +} + static void *virtqueue_get_buf_ctx_split(struct vring_virtqueue *vq, unsigned int *len, void **ctx) @@ -917,6 +946,73 @@ static void *virtqueue_get_buf_ctx_split(struct vring_= virtqueue *vq, return ret; } =20 +static void *virtqueue_get_buf_ctx_split_in_order(struct vring_virtqueue *= vq, + unsigned int *len, + void **ctx) +{ + void *ret; + unsigned int num =3D vq->split.vring.num; + u16 last_used; + + START_USE(vq); + + if (unlikely(vq->broken)) { + END_USE(vq); + return NULL; + } + + last_used =3D (vq->last_used_idx & (vq->split.vring.num - 1)); + + if (vq->batch_head =3D=3D num) { + if (!more_used_split(vq)) { + pr_debug("No more buffers in queue\n"); + END_USE(vq); + return NULL; + } + + /* Only get used array entries after they have been + * exposed by host. */ + virtio_rmb(vq->weak_barriers); + vq->batch_head =3D virtio32_to_cpu(vq->vq.vdev, + vq->split.vring.used->ring[last_used].id); + vq->batch_len =3D virtio32_to_cpu(vq->vq.vdev, + vq->split.vring.used->ring[last_used].len); + } + + if (vq->batch_head =3D=3D last_used) { + vq->batch_head =3D num; + *len =3D vq->batch_len; + } else + *len =3D vq->split.desc_state[last_used].total_len; + + if (unlikely(last_used >=3D num)) { + BAD_RING(vq, "id %u out of range\n", last_used); + return NULL; + } + if (unlikely(!vq->split.desc_state[last_used].data)) { + BAD_RING(vq, "id %u is not a head!\n", last_used); + return NULL; + } + + /* detach_buf_split clears data, so grab it now. */ + ret =3D vq->split.desc_state[last_used].data; + detach_buf_split_in_order(vq, last_used, ctx); + + vq->last_used_idx++; + /* If we expect an interrupt for the next entry, tell host + * by writing event index and flush out the write before + * the read in the next get_buf call. */ + if (!(vq->split.avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT)) + virtio_store_mb(vq->weak_barriers, + &vring_used_event(&vq->split.vring), + cpu_to_virtio16(vq->vq.vdev, vq->last_used_idx)); + + LAST_ADD_TIME_INVALID(vq); + + END_USE(vq); + return ret; +} + static void virtqueue_disable_cb_split(struct vring_virtqueue *vq) { if (!(vq->split.avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT)) { @@ -1010,7 +1106,10 @@ static void *virtqueue_detach_unused_buf_split(struc= t vring_virtqueue *vq) continue; /* detach_buf_split clears data, so grab it now. */ buf =3D vq->split.desc_state[i].data; - detach_buf_split(vq, i, NULL); + if (virtqueue_is_in_order(vq)) + detach_buf_split_in_order(vq, i, NULL); + else + detach_buf_split(vq, i, NULL); vq->split.avail_idx_shadow--; vq->split.vring.avail->idx =3D cpu_to_virtio16(vq->vq.vdev, vq->split.avail_idx_shadow); @@ -1073,6 +1172,7 @@ static void virtqueue_vring_attach_split(struct vring= _virtqueue *vq, =20 /* Put everything in free lists. */ vq->free_head =3D 0; + vq->batch_head =3D vq->split.vring.num; } =20 static int vring_alloc_state_extra_split(struct vring_virtqueue_split *vri= ng_split) @@ -1183,7 +1283,6 @@ static struct virtqueue *__vring_new_virtqueue_split(= unsigned int index, if (!vq) return NULL; =20 - vq->layout =3D SPLIT; vq->vq.callback =3D callback; vq->vq.vdev =3D vdev; vq->vq.name =3D name; @@ -1203,6 +1302,8 @@ static struct virtqueue *__vring_new_virtqueue_split(= unsigned int index, vq->indirect =3D virtio_has_feature(vdev, VIRTIO_RING_F_INDIRECT_DESC) && !context; vq->event =3D virtio_has_feature(vdev, VIRTIO_RING_F_EVENT_IDX); + vq->layout =3D virtio_has_feature(vdev, VIRTIO_F_IN_ORDER) ? + SPLIT_IN_ORDER : SPLIT; =20 if (virtio_has_feature(vdev, VIRTIO_F_ORDER_PLATFORM)) vq->weak_barriers =3D false; @@ -1366,13 +1467,14 @@ static int virtqueue_add_indirect_packed(struct vri= ng_virtqueue *vq, unsigned int in_sgs, void *data, bool premapped, - gfp_t gfp) + gfp_t gfp, + u16 id) { struct vring_desc_extra *extra; struct vring_packed_desc *desc; struct scatterlist *sg; - unsigned int i, n, err_idx, len; - u16 head, id; + unsigned int i, n, err_idx, len, total_len =3D 0; + u16 head; dma_addr_t addr; =20 head =3D vq->packed.next_avail_idx; @@ -1390,8 +1492,6 @@ static int virtqueue_add_indirect_packed(struct vring= _virtqueue *vq, } =20 i =3D 0; - id =3D vq->free_head; - BUG_ON(id =3D=3D vq->packed.vring.num); =20 for (n =3D 0; n < out_sgs + in_sgs; n++) { for (sg =3D sgs[n]; sg; sg =3D sg_next(sg)) { @@ -1411,6 +1511,7 @@ static int virtqueue_add_indirect_packed(struct vring= _virtqueue *vq, extra[i].flags =3D n < out_sgs ? 0 : VRING_DESC_F_WRITE; } =20 + total_len +=3D len; i++; } } @@ -1464,6 +1565,7 @@ static int virtqueue_add_indirect_packed(struct vring= _virtqueue *vq, vq->packed.desc_state[id].data =3D data; vq->packed.desc_state[id].indir_desc =3D desc; vq->packed.desc_state[id].last =3D id; + vq->packed.desc_state[id].total_len =3D total_len; =20 vq->num_added +=3D 1; =20 @@ -1516,8 +1618,11 @@ static inline int virtqueue_add_packed(struct vring_= virtqueue *vq, BUG_ON(total_sg =3D=3D 0); =20 if (virtqueue_use_indirect(vq, total_sg)) { + id =3D vq->free_head; + BUG_ON(id =3D=3D vq->packed.vring.num); err =3D virtqueue_add_indirect_packed(vq, sgs, total_sg, out_sgs, - in_sgs, data, premapped, gfp); + in_sgs, data, premapped, + gfp, id); if (err !=3D -ENOMEM) { END_USE(vq); return err; @@ -1638,6 +1743,152 @@ static inline int virtqueue_add_packed(struct vring= _virtqueue *vq, return -EIO; } =20 +static inline int virtqueue_add_packed_in_order(struct vring_virtqueue *vq, + struct scatterlist *sgs[], + unsigned int total_sg, + unsigned int out_sgs, + unsigned int in_sgs, + void *data, + void *ctx, + bool premapped, + gfp_t gfp) +{ + struct vring_packed_desc *desc; + struct scatterlist *sg; + unsigned int i, n, c, err_idx, total_len =3D 0; + __le16 head_flags, flags; + u16 head, avail_used_flags; + int err; + + START_USE(vq); + + BUG_ON(data =3D=3D NULL); + BUG_ON(ctx && vq->indirect); + + if (unlikely(vq->broken)) { + END_USE(vq); + return -EIO; + } + + LAST_ADD_TIME_UPDATE(vq); + + BUG_ON(total_sg =3D=3D 0); + + if (virtqueue_use_indirect(vq, total_sg)) { + err =3D virtqueue_add_indirect_packed(vq, sgs, total_sg, out_sgs, + in_sgs, data, premapped, gfp, + vq->packed.next_avail_idx); + if (err !=3D -ENOMEM) { + END_USE(vq); + return err; + } + + /* fall back on direct */ + } + + head =3D vq->packed.next_avail_idx; + avail_used_flags =3D vq->packed.avail_used_flags; + + WARN_ON_ONCE(total_sg > vq->packed.vring.num && !vq->indirect); + + desc =3D vq->packed.vring.desc; + i =3D head; + + if (unlikely(vq->vq.num_free < total_sg)) { + pr_debug("Can't add buf len %i - avail =3D %i\n", + total_sg, vq->vq.num_free); + END_USE(vq); + return -ENOSPC; + } + + c =3D 0; + for (n =3D 0; n < out_sgs + in_sgs; n++) { + for (sg =3D sgs[n]; sg; sg =3D sg_next(sg)) { + dma_addr_t addr; + u32 len; + + if (vring_map_one_sg(vq, sg, n < out_sgs ? + DMA_TO_DEVICE : DMA_FROM_DEVICE, + &addr, &len, premapped)) + goto unmap_release; + + flags =3D cpu_to_le16(vq->packed.avail_used_flags | + (++c =3D=3D total_sg ? 0 : VRING_DESC_F_NEXT) | + (n < out_sgs ? 0 : VRING_DESC_F_WRITE)); + if (i =3D=3D head) + head_flags =3D flags; + else + desc[i].flags =3D flags; + + + desc[i].addr =3D cpu_to_le64(addr); + desc[i].len =3D cpu_to_le32(len); + desc[i].id =3D cpu_to_le16(head); + + if (unlikely(vq->use_dma_api)) { + vq->packed.desc_extra[i].addr =3D premapped ? + DMA_MAPPING_ERROR: addr; + vq->packed.desc_extra[i].len =3D len; + vq->packed.desc_extra[i].flags =3D + le16_to_cpu(flags); + } + + if ((unlikely(++i >=3D vq->packed.vring.num))) { + i =3D 0; + vq->packed.avail_used_flags ^=3D + 1 << VRING_PACKED_DESC_F_AVAIL | + 1 << VRING_PACKED_DESC_F_USED; + vq->packed.avail_wrap_counter ^=3D 1; + } + + total_len +=3D len; + } + } + + /* We're using some buffers from the free list. */ + vq->vq.num_free -=3D total_sg; + + /* Update free pointer */ + vq->packed.next_avail_idx =3D i; + + /* Store token. */ + vq->packed.desc_state[head].num =3D total_sg; + vq->packed.desc_state[head].data =3D data; + vq->packed.desc_state[head].indir_desc =3D ctx; + vq->packed.desc_state[head].total_len =3D total_len; + + /* + * A driver MUST NOT make the first descriptor in the list + * available before all subsequent descriptors comprising + * the list are made available. + */ + virtio_wmb(vq->weak_barriers); + vq->packed.vring.desc[head].flags =3D head_flags; + vq->num_added +=3D total_sg; + + pr_debug("Added buffer head %i to %p\n", head, vq); + END_USE(vq); + + return 0; + +unmap_release: + err_idx =3D i; + i =3D head; + vq->packed.avail_used_flags =3D avail_used_flags; + + for (n =3D 0; n < total_sg; n++) { + if (i =3D=3D err_idx) + break; + vring_unmap_extra_packed(vq, &vq->packed.desc_extra[i]); + i++; + if (i >=3D vq->packed.vring.num) + i =3D 0; + } + + END_USE(vq); + return -EIO; +} + static bool virtqueue_kick_prepare_packed(struct vring_virtqueue *vq) { u16 new, old, off_wrap, flags, wrap_counter, event_idx; @@ -1758,7 +2009,7 @@ static inline bool is_used_desc_packed(const struct v= ring_virtqueue *vq, return avail =3D=3D used && used =3D=3D used_wrap_counter; } =20 -static bool virtqueue_poll_packed(const struct vring_virtqueue *vq, u16 of= f_wrap) +static bool __virtqueue_poll_packed(const struct vring_virtqueue *vq, u16 = off_wrap) { bool wrap_counter; u16 used_idx; @@ -1769,6 +2020,11 @@ static bool virtqueue_poll_packed(const struct vring= _virtqueue *vq, u16 off_wrap return is_used_desc_packed(vq, used_idx, wrap_counter); } =20 +static bool virtqueue_poll_packed(const struct vring_virtqueue *vq, u16 of= f_wrap) +{ + return __virtqueue_poll_packed(vq, off_wrap); +} + static bool more_used_packed(const struct vring_virtqueue *vq) { return virtqueue_poll_packed(vq, READ_ONCE(vq->last_used_idx)); @@ -1798,10 +2054,84 @@ static void update_last_used_idx_packed(struct vrin= g_virtqueue *vq, cpu_to_le16(vq->last_used_idx)); } =20 +static bool more_used_packed_in_order(const struct vring_virtqueue *vq) +{ + if (vq->batch_head !=3D vq->packed.vring.num) + return true; + + return virtqueue_poll_packed(vq, READ_ONCE(vq->last_used_idx)); +} + +static bool __more_used_packed(const struct vring_virtqueue *vq) +{ + return __virtqueue_poll_packed(vq, READ_ONCE(vq->last_used_idx)); +} + +static void *virtqueue_get_buf_ctx_packed_in_order(struct vring_virtqueue = *vq, + unsigned int *len, + void **ctx) +{ + unsigned int num =3D vq->packed.vring.num; + u16 last_used, id, last_used_idx; + bool used_wrap_counter; + void *ret; + + START_USE(vq); + + if (unlikely(vq->broken)) { + END_USE(vq); + return NULL; + } + + last_used_idx =3D vq->last_used_idx; + used_wrap_counter =3D packed_used_wrap_counter(last_used_idx); + last_used =3D packed_last_used(last_used_idx); + + if (vq->batch_head =3D=3D num) { + if (!__more_used_packed(vq)) { + pr_debug("No more buffers in queue\n"); + END_USE(vq); + return NULL; + } + /* Only get used elements after they have been exposed by host. */ + virtio_rmb(vq->weak_barriers); + vq->batch_head =3D le16_to_cpu(vq->packed.vring.desc[last_used].id); + vq->batch_len =3D le32_to_cpu(vq->packed.vring.desc[last_used].len); + } + + if (vq->batch_head =3D=3D last_used) { + vq->batch_head =3D num; + *len =3D vq->batch_len; + } else + *len =3D vq->packed.desc_state[last_used].total_len; + + if (unlikely(last_used >=3D num)) { + BAD_RING(vq, "id %u out of range\n", id); + return NULL; + } + if (unlikely(!vq->packed.desc_state[last_used].data)) { + BAD_RING(vq, "id %u is not a head!\n", id); + return NULL; + } + + /* detach_buf_packed clears data, so grab it now. */ + ret =3D vq->packed.desc_state[last_used].data; + detach_buf_packed_in_order(vq, last_used, ctx); + + update_last_used_idx_packed(vq, last_used, last_used, + used_wrap_counter); + + LAST_ADD_TIME_INVALID(vq); + + END_USE(vq); + return ret; +} + static void *virtqueue_get_buf_ctx_packed(struct vring_virtqueue *vq, unsigned int *len, void **ctx) { + unsigned int num =3D vq->packed.vring.num; u16 last_used, id, last_used_idx; bool used_wrap_counter; void *ret; @@ -1813,7 +2143,7 @@ static void *virtqueue_get_buf_ctx_packed(struct vrin= g_virtqueue *vq, return NULL; } =20 - if (!more_used_packed(vq)) { + if (!__more_used_packed(vq)) { pr_debug("No more buffers in queue\n"); END_USE(vq); return NULL; @@ -1828,7 +2158,7 @@ static void *virtqueue_get_buf_ctx_packed(struct vrin= g_virtqueue *vq, id =3D le16_to_cpu(vq->packed.vring.desc[last_used].id); *len =3D le32_to_cpu(vq->packed.vring.desc[last_used].len); =20 - if (unlikely(id >=3D vq->packed.vring.num)) { + if (unlikely(id >=3D num)) { BAD_RING(vq, "id %u out of range\n", id); return NULL; } @@ -1948,6 +2278,7 @@ static bool virtqueue_enable_cb_delayed_packed(struct= vring_virtqueue *vq) last_used_idx =3D READ_ONCE(vq->last_used_idx); wrap_counter =3D packed_used_wrap_counter(last_used_idx); used_idx =3D packed_last_used(last_used_idx); + if (is_used_desc_packed(vq, used_idx, wrap_counter)) { END_USE(vq); return false; @@ -1969,7 +2300,7 @@ static void *virtqueue_detach_unused_buf_packed(struc= t vring_virtqueue *vq) continue; /* detach_buf clears data, so grab it now. */ buf =3D vq->packed.desc_state[i].data; - detach_buf_packed(vq, i, NULL); + detach_buf_packed_in_order(vq, i, NULL); END_USE(vq); return buf; } @@ -1995,6 +2326,8 @@ static struct vring_desc_extra *vring_alloc_desc_extr= a(unsigned int num) for (i =3D 0; i < num - 1; i++) desc_extra[i].next =3D i + 1; =20 + desc_extra[num - 1].next =3D 0; + return desc_extra; } =20 @@ -2126,8 +2459,12 @@ static void virtqueue_vring_attach_packed(struct vri= ng_virtqueue *vq, { vq->packed =3D *vring_packed; =20 - /* Put everything in free lists. */ - vq->free_head =3D 0; + if (virtqueue_is_in_order(vq)) + vq->batch_head =3D vq->split.vring.num; + else { + /* Put everything in free lists. */ + vq->free_head =3D 0; + } } =20 static void virtqueue_reset_packed(struct vring_virtqueue *vq) @@ -2174,13 +2511,14 @@ static struct virtqueue *__vring_new_virtqueue_pack= ed(unsigned int index, #else vq->broken =3D false; #endif - vq->layout =3D PACKED; vq->dma_dev =3D dma_dev; vq->use_dma_api =3D vring_use_dma_api(vdev); =20 vq->indirect =3D virtio_has_feature(vdev, VIRTIO_RING_F_INDIRECT_DESC) && !context; vq->event =3D virtio_has_feature(vdev, VIRTIO_RING_F_EVENT_IDX); + vq->layout =3D virtio_has_feature(vdev, VIRTIO_F_IN_ORDER) ? + PACKED_IN_ORDER : PACKED; =20 if (virtio_has_feature(vdev, VIRTIO_F_ORDER_PLATFORM)) vq->weak_barriers =3D false; @@ -2290,9 +2628,39 @@ static const struct virtqueue_ops packed_ops =3D { .reset =3D virtqueue_reset_packed, }; =20 +static const struct virtqueue_ops split_in_order_ops =3D { + .add =3D virtqueue_add_split, + .get =3D virtqueue_get_buf_ctx_split_in_order, + .kick_prepare =3D virtqueue_kick_prepare_split, + .disable_cb =3D virtqueue_disable_cb_split, + .enable_cb_delayed =3D virtqueue_enable_cb_delayed_split, + .enable_cb_prepare =3D virtqueue_enable_cb_prepare_split, + .poll =3D virtqueue_poll_split, + .detach_unused_buf =3D virtqueue_detach_unused_buf_split, + .more_used =3D more_used_split_in_order, + .resize =3D virtqueue_resize_split, + .reset =3D virtqueue_reset_split, +}; + +static const struct virtqueue_ops packed_in_order_ops =3D { + .add =3D virtqueue_add_packed_in_order, + .get =3D virtqueue_get_buf_ctx_packed_in_order, + .kick_prepare =3D virtqueue_kick_prepare_packed, + .disable_cb =3D virtqueue_disable_cb_packed, + .enable_cb_delayed =3D virtqueue_enable_cb_delayed_packed, + .enable_cb_prepare =3D virtqueue_enable_cb_prepare_packed, + .poll =3D virtqueue_poll_packed, + .detach_unused_buf =3D virtqueue_detach_unused_buf_packed, + .more_used =3D more_used_packed_in_order, + .resize =3D virtqueue_resize_packed, + .reset =3D virtqueue_reset_packed, +}; + static const struct virtqueue_ops *const all_ops[VQ_TYPE_MAX] =3D { [SPLIT] =3D &split_ops, - [PACKED] =3D &packed_ops + [PACKED] =3D &packed_ops, + [SPLIT_IN_ORDER] =3D &split_in_order_ops, + [PACKED_IN_ORDER] =3D &packed_in_order_ops, }; =20 static int virtqueue_disable_and_recycle(struct virtqueue *_vq, @@ -2336,7 +2704,6 @@ static int virtqueue_enable_after_reset(struct virtqu= eue *_vq) /* * Generic functions and exported symbols. */ - #define VIRTQUEUE_CALL(vq, op, ...) \ ({ \ typeof(all_ops[SPLIT]->op(vq, ##__VA_ARGS__)) ret; \ @@ -2347,6 +2714,12 @@ static int virtqueue_enable_after_reset(struct virtq= ueue *_vq) case PACKED: \ ret =3D all_ops[PACKED]->op(vq, ##__VA_ARGS__); \ break; \ + case SPLIT_IN_ORDER: \ + ret =3D all_ops[SPLIT_IN_ORDER]->op(vq, ##__VA_ARGS__); \ + break; \ + case PACKED_IN_ORDER: \ + ret =3D all_ops[PACKED_IN_ORDER]->op(vq, ##__VA_ARGS__); \ + break; \ default: \ BUG(); \ break; \ @@ -2363,6 +2736,12 @@ static int virtqueue_enable_after_reset(struct virtq= ueue *_vq) case PACKED: \ all_ops[PACKED]->op(vq, ##__VA_ARGS__); \ break; \ + case SPLIT_IN_ORDER: \ + all_ops[SPLIT_IN_ORDER]->op(vq, ##__VA_ARGS__); \ + break; \ + case PACKED_IN_ORDER: \ + all_ops[PACKED_IN_ORDER]->op(vq, ##__VA_ARGS__); \ + break; \ default: \ BUG(); \ break; \ @@ -3073,6 +3452,8 @@ void vring_transport_features(struct virtio_device *v= dev) break; case VIRTIO_F_NOTIFICATION_DATA: break; + case VIRTIO_F_IN_ORDER: + break; default: /* We don't understand this bit. */ __virtio_clear_bit(vdev, i); --=20 2.34.1