From nobody Sat Feb 7 15:21:59 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 374192DBF6E for ; Mon, 20 Oct 2025 07:10:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944225; cv=none; b=DcbNkxZUx0pZwVsewNs1Fl2ka87vohol+Z18eLnO/xiHwjsscp1ZwEZ0CulHQ25Mf0CE57x4C6qRkXn9s3xoG2jb/0JxPj3N1Ld6SXLpSmf3Af+BakuRbbgLq0Db7hdF1I35ssiFwvWFt/OKeDU49fUu6tLWpqMh6mqPjRAGUvU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944225; c=relaxed/simple; bh=Prhd+LkqoYp0VJ2YSuQDc0m2lgDkGgYa2aWwnVG/VQw=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=OCKejBTlVMgMJQUKeTM8IUfWtFud8cPHB37Gc3bT4iABoctDYiA8Pncst2xhM5wB/DqbzeQu3pdmi6k0JTTJfkRARfUTaHPH1y853V57ZNjwDcf9iJagmNoT2+gw85va2Zrq+yM8H3+gFAcEVYNc7E4gKq0afIRlLMydbryjB1c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Xm2b/evy; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Xm2b/evy" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1760944222; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9HvYkyijRiL6uD+3Th5PyNitUXfDZOQ8r0roTL8rtAQ=; b=Xm2b/evyK1uAoQDv8EmJ52HIl17IJcSr+6Z0PsI+Ku9op6bJ2prgOFIt8Dp6RwAtkqR+13 6HexSOCMQE1hVeBRUp6Ma4+Cp/+JwhJR7Ux5p7qPzlbeSya+oBBMSTtpqvesAOP4Y3yO8W uKhDYJ21YY+bSWmgoCXbK5icC5c6wBo= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-573-RkFbBXOmOBSXz-aTfQVs0g-1; Mon, 20 Oct 2025 03:10:17 -0400 X-MC-Unique: RkFbBXOmOBSXz-aTfQVs0g-1 X-Mimecast-MFC-AGG-ID: RkFbBXOmOBSXz-aTfQVs0g_1760944217 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id EDEE81800345; Mon, 20 Oct 2025 07:10:16 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.113.29]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id AC6B7300019F; Mon, 20 Oct 2025 07:10:13 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V8 01/19] virtio_ring: rename virtqueue_reinit_xxx to virtqueue_reset_xxx() Date: Mon, 20 Oct 2025 15:09:45 +0800 Message-ID: <20251020071003.28834-2-jasowang@redhat.com> In-Reply-To: <20251020071003.28834-1-jasowang@redhat.com> References: <20251020071003.28834-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 To be consistent with virtqueue_reset(). Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index f91a432b3e53..73790593523a 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -1009,7 +1009,7 @@ static void virtqueue_vring_init_split(struct vring_v= irtqueue_split *vring_split } } =20 -static void virtqueue_reinit_split(struct vring_virtqueue *vq) +static void virtqueue_reset_split(struct vring_virtqueue *vq) { int num; =20 @@ -1253,7 +1253,7 @@ static int virtqueue_resize_split(struct virtqueue *_= vq, u32 num) err_state_extra: vring_free_split(&vring_split, vdev, vq->map); err: - virtqueue_reinit_split(vq); + virtqueue_reset_split(vq); return -ENOMEM; } =20 @@ -2091,7 +2091,7 @@ static void virtqueue_vring_attach_packed(struct vrin= g_virtqueue *vq, vq->free_head =3D 0; } =20 -static void virtqueue_reinit_packed(struct vring_virtqueue *vq) +static void virtqueue_reset_packed(struct vring_virtqueue *vq) { memset(vq->packed.vring.device, 0, vq->packed.event_size_in_bytes); memset(vq->packed.vring.driver, 0, vq->packed.event_size_in_bytes); @@ -2218,7 +2218,7 @@ static int virtqueue_resize_packed(struct virtqueue *= _vq, u32 num) err_state_extra: vring_free_packed(&vring_packed, vdev, vq->map); err_ring: - virtqueue_reinit_packed(vq); + virtqueue_reset_packed(vq); return -ENOMEM; } =20 @@ -2860,9 +2860,9 @@ int virtqueue_reset(struct virtqueue *_vq, recycle_done(_vq); =20 if (vq->packed_ring) - virtqueue_reinit_packed(vq); + virtqueue_reset_packed(vq); else - virtqueue_reinit_split(vq); + virtqueue_reset_split(vq); =20 return virtqueue_enable_after_reset(_vq); } --=20 2.31.1 From nobody Sat Feb 7 15:21:59 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 965552DAFDE for ; Mon, 20 Oct 2025 07:10:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944234; cv=none; b=tmPAoH68VsRMVyZQkNXNW2U89wY1jetmRgmlxhYYHtg619dgsFCukMvhsHZolIzE6bjkuIPSriVcOzatx2xRDt0OGOj6RQx1de84tbiy8zT/diuC0SdHJN+qtDhx5/6CAqCgOxgeX/uOQHSHZneEKNyInYp1Y86YgozaDGTpT2g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944234; c=relaxed/simple; bh=HYsHOGN3EYNOH6tOJeaoOb+gHk0SE0FJEJ4VPLlA9Z4=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=UrVpJitRhy9FYnjrNSN1D6LH5JPhSW3NchaIBGzXZmi9oK2EiQ8dqiVb+Hvgpt6cI+7xUCiGfQ+KJWdTJMO4IvSQ7gM6MeOQYVHNa5qRiDpcMK/dw9CGBSnxD3My0aoV6zFO43QgUlkO9/eg+TsDm3dAUGNPi/jGgVLuzF9KxfA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=O1WFYftx; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="O1WFYftx" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1760944231; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JDhkOP2irC9AYhWmwD5Iofw1hRzZ9N7g2ReMYfK0ONA=; b=O1WFYftxraCMx9jFDGnvNZdMODAYI0BINjYh3zWQBOJG4AEfxoLxh9UVo4+mkoKhAv3i9T fepkEZ8c2qEa3STLDy7hV55lxNJxycL4VBeQ1uMq6IC2iGfpOW99KzJ6gFbXaQSnYDEihj BHxp17sXFlZVxUubomQ+WwpkXSD19/Q= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-601-5k4J9FGrNWq2gOCjYgmTrQ-1; Mon, 20 Oct 2025 03:10:22 -0400 X-MC-Unique: 5k4J9FGrNWq2gOCjYgmTrQ-1 X-Mimecast-MFC-AGG-ID: 5k4J9FGrNWq2gOCjYgmTrQ_1760944221 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id D2B49180035D; Mon, 20 Oct 2025 07:10:20 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.113.29]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id A173A300019F; Mon, 20 Oct 2025 07:10:17 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V8 02/19] virtio_ring: switch to use vring_virtqueue in virtqueue_poll variants Date: Mon, 20 Oct 2025 15:09:46 +0800 Message-ID: <20251020071003.28834-3-jasowang@redhat.com> In-Reply-To: <20251020071003.28834-1-jasowang@redhat.com> References: <20251020071003.28834-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Those variants are used internally so let's switch to use vring_virtqueue as parameter to be consistent with other internal virtqueue helpers. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 73790593523a..fed3962411a1 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -919,11 +919,10 @@ static unsigned int virtqueue_enable_cb_prepare_split= (struct virtqueue *_vq) return last_used_idx; } =20 -static bool virtqueue_poll_split(struct virtqueue *_vq, unsigned int last_= used_idx) +static bool virtqueue_poll_split(struct vring_virtqueue *vq, + unsigned int last_used_idx) { - struct vring_virtqueue *vq =3D to_vvq(_vq); - - return (u16)last_used_idx !=3D virtio16_to_cpu(_vq->vdev, + return (u16)last_used_idx !=3D virtio16_to_cpu(vq->vq.vdev, vq->split.vring.used->idx); } =20 @@ -1844,9 +1843,8 @@ static unsigned int virtqueue_enable_cb_prepare_packe= d(struct virtqueue *_vq) return vq->last_used_idx; } =20 -static bool virtqueue_poll_packed(struct virtqueue *_vq, u16 off_wrap) +static bool virtqueue_poll_packed(struct vring_virtqueue *vq, u16 off_wrap) { - struct vring_virtqueue *vq =3D to_vvq(_vq); bool wrap_counter; u16 used_idx; =20 @@ -2611,8 +2609,8 @@ bool virtqueue_poll(struct virtqueue *_vq, unsigned i= nt last_used_idx) return false; =20 virtio_mb(vq->weak_barriers); - return vq->packed_ring ? virtqueue_poll_packed(_vq, last_used_idx) : - virtqueue_poll_split(_vq, last_used_idx); + return vq->packed_ring ? virtqueue_poll_packed(vq, last_used_idx) : + virtqueue_poll_split(vq, last_used_idx); } EXPORT_SYMBOL_GPL(virtqueue_poll); =20 --=20 2.31.1 From nobody Sat Feb 7 15:21:59 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 883F12DC775 for ; Mon, 20 Oct 2025 07:10:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944230; cv=none; b=m0GwAUHSZEGcAfStOrbB6MVqJUgniDyOlfTwyoiCMUivYp0nzGLcM0H+ZP0esKEBVbLZX7+zcqdMorcdvqVrjeICHWLQi5d7zk+RhIWFR3LffhcSryZhlBOfSw5ag2HhWcaBYR4NKAWsVxP0K4H71htwXhiyAOGkwNe5bW818Vk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944230; c=relaxed/simple; bh=4t6ixKfADhlr9xzvfuDQl77U06Jc1JbbTyleI9sfI7U=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=JhfdPQ1JSu63IVOpZNRQYZyF+1Y3yVlD/8n7/mtjdigQbncrA3C/Nm+mlnBiQv1Uy2JIjI1wMg8MpS+P3wrm6NtvdPj6LZpvRapssoc3sy68F8SQNnbqbHsfCXDzYUM8pIEmIHpIxy/8dC+eTAeQ9fyQiu2AaqvSJ4yui3KT/zQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Cfoq4U9n; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Cfoq4U9n" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1760944227; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vFIZuPKUpfLWKhMoVS+8GFV45HPQ85k5vLQT16WEDbY=; b=Cfoq4U9nJRs1N1OvB9ClmT3qConDaR1rgGHJ29iONHqSwXRTyOYUjGBY2LOkNLPvip5Cgs ECIwfS7xMHosiCEk+6m1d1riNj84vgEwuYoA9NnEB91qkpgZ82AcU0P10gcHFNYAB784/x b59pvTKSNAQzLaJquxvpTbSBALkIWFY= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-494-ifj8Ev2aP3estD_PQrBscg-1; Mon, 20 Oct 2025 03:10:25 -0400 X-MC-Unique: ifj8Ev2aP3estD_PQrBscg-1 X-Mimecast-MFC-AGG-ID: ifj8Ev2aP3estD_PQrBscg_1760944224 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 7D0DF1954200; Mon, 20 Oct 2025 07:10:24 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.113.29]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 7F3F530001A2; Mon, 20 Oct 2025 07:10:21 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V8 03/19] virtio_ring: unify logic of virtqueue_poll() and more_used() Date: Mon, 20 Oct 2025 15:09:47 +0800 Message-ID: <20251020071003.28834-4-jasowang@redhat.com> In-Reply-To: <20251020071003.28834-1-jasowang@redhat.com> References: <20251020071003.28834-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 This patch unifies the logic of virtqueue_poll() and more_used() for better code reusing and ease the future in order implementation. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 48 +++++++++++++++--------------------- 1 file changed, 20 insertions(+), 28 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index fed3962411a1..d8a07e0d9fa8 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -806,12 +806,18 @@ static void detach_buf_split(struct vring_virtqueue *= vq, unsigned int head, } } =20 -static bool more_used_split(const struct vring_virtqueue *vq) +static bool virtqueue_poll_split(const struct vring_virtqueue *vq, + unsigned int last_used_idx) { - return vq->last_used_idx !=3D virtio16_to_cpu(vq->vq.vdev, + return (u16)last_used_idx !=3D virtio16_to_cpu(vq->vq.vdev, vq->split.vring.used->idx); } =20 +static bool more_used_split(const struct vring_virtqueue *vq) +{ + return virtqueue_poll_split(vq, vq->last_used_idx); +} + static void *virtqueue_get_buf_ctx_split(struct virtqueue *_vq, unsigned int *len, void **ctx) @@ -919,13 +925,6 @@ static unsigned int virtqueue_enable_cb_prepare_split(= struct virtqueue *_vq) return last_used_idx; } =20 -static bool virtqueue_poll_split(struct vring_virtqueue *vq, - unsigned int last_used_idx) -{ - return (u16)last_used_idx !=3D virtio16_to_cpu(vq->vq.vdev, - vq->split.vring.used->idx); -} - static bool virtqueue_enable_cb_delayed_split(struct virtqueue *_vq) { struct vring_virtqueue *vq =3D to_vvq(_vq); @@ -1710,16 +1709,20 @@ static inline bool is_used_desc_packed(const struct= vring_virtqueue *vq, return avail =3D=3D used && used =3D=3D used_wrap_counter; } =20 -static bool more_used_packed(const struct vring_virtqueue *vq) +static bool virtqueue_poll_packed(const struct vring_virtqueue *vq, u16 of= f_wrap) { - u16 last_used; - u16 last_used_idx; - bool used_wrap_counter; + bool wrap_counter; + u16 used_idx; =20 - last_used_idx =3D READ_ONCE(vq->last_used_idx); - last_used =3D packed_last_used(last_used_idx); - used_wrap_counter =3D packed_used_wrap_counter(last_used_idx); - return is_used_desc_packed(vq, last_used, used_wrap_counter); + wrap_counter =3D off_wrap >> VRING_PACKED_EVENT_F_WRAP_CTR; + used_idx =3D off_wrap & ~(1 << VRING_PACKED_EVENT_F_WRAP_CTR); + + return is_used_desc_packed(vq, used_idx, wrap_counter); +} + +static bool more_used_packed(const struct vring_virtqueue *vq) +{ + return virtqueue_poll_packed(vq, READ_ONCE(vq->last_used_idx)); } =20 static void *virtqueue_get_buf_ctx_packed(struct virtqueue *_vq, @@ -1843,17 +1846,6 @@ static unsigned int virtqueue_enable_cb_prepare_pack= ed(struct virtqueue *_vq) return vq->last_used_idx; } =20 -static bool virtqueue_poll_packed(struct vring_virtqueue *vq, u16 off_wrap) -{ - bool wrap_counter; - u16 used_idx; - - wrap_counter =3D off_wrap >> VRING_PACKED_EVENT_F_WRAP_CTR; - used_idx =3D off_wrap & ~(1 << VRING_PACKED_EVENT_F_WRAP_CTR); - - return is_used_desc_packed(vq, used_idx, wrap_counter); -} - static bool virtqueue_enable_cb_delayed_packed(struct virtqueue *_vq) { struct vring_virtqueue *vq =3D to_vvq(_vq); --=20 2.31.1 From nobody Sat Feb 7 15:21:59 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 77CDD2DAFDE for ; Mon, 20 Oct 2025 07:10:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944239; cv=none; b=h+0EIxO2QzNBjZ2nAXjAYDWRbXixmkY4WPm01GDACsoavcD2AU/D7r5ij1s7joiWhT1gYPKDXuM9MK5l8A9FU3nfEDK2dRLMAmjNCzTnG7zDz2kOv9AT9uJwMu8LRm0KSBlv922ImA5FaGvKntHcDQmH20JQHB4vbexz+gq4r4k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944239; c=relaxed/simple; bh=92dzLgz7GPIFGxrV7u59xZvyZkD4ZOwGTYSuMOriM2E=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=R4+2V4q/NqseNxwUzWeAxjLSgnkprjKIGggbad0jdcRe/VbXE8nv9+vE9vEvvyBEZUxwdKr3zKYV1hRIhTb1oi87a7mCHBlZfSHyE1ZA+oNgRHULHlAArA/iY/fNQ39xB1Vhaz4IozZmW6yJodmPZFHbJjd/JSJ4ZcxZHjXyQnI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=TleQ7rSk; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="TleQ7rSk" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1760944236; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/9X+8nJzs108oA3zz8mRtrO71MkKZ7x5Ta9F7UOdDxY=; b=TleQ7rSk6jTKSaOyZoY2Nxu81iIPbTi5o4aBkONfAq6FEMu0nphjQ9WuulDK6N4KLVL7QV lR+PxmO1KyPpuzk2nNcSH2OWdUZ1uUFwMev+bkkBIgCLhL6UL5Lz5bbsW5V+r4bSoCFaCB a03dm7dhyG9/pg0iUycTZqaxFKEI8sc= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-683-b2ik-4mjMiemGa3J-GQTzQ-1; Mon, 20 Oct 2025 03:10:29 -0400 X-MC-Unique: b2ik-4mjMiemGa3J-GQTzQ-1 X-Mimecast-MFC-AGG-ID: b2ik-4mjMiemGa3J-GQTzQ_1760944228 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 458B9180057A; Mon, 20 Oct 2025 07:10:28 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.113.29]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 2FBFE300019F; Mon, 20 Oct 2025 07:10:24 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V8 04/19] virtio_ring: switch to use vring_virtqueue for virtqueue resize variants Date: Mon, 20 Oct 2025 15:09:48 +0800 Message-ID: <20251020071003.28834-5-jasowang@redhat.com> In-Reply-To: <20251020071003.28834-1-jasowang@redhat.com> References: <20251020071003.28834-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Those variants are used internally so let's switch to use vring_virtqueue as parameter to be consistent with other internal virtqueue helpers. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index d8a07e0d9fa8..693671bac841 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -1221,11 +1221,10 @@ static struct virtqueue *vring_create_virtqueue_spl= it( return vq; } =20 -static int virtqueue_resize_split(struct virtqueue *_vq, u32 num) +static int virtqueue_resize_split(struct vring_virtqueue *vq, u32 num) { struct vring_virtqueue_split vring_split =3D {}; - struct vring_virtqueue *vq =3D to_vvq(_vq); - struct virtio_device *vdev =3D _vq->vdev; + struct virtio_device *vdev =3D vq->vq.vdev; int err; =20 err =3D vring_alloc_queue_split(&vring_split, vdev, num, @@ -2182,11 +2181,10 @@ static struct virtqueue *vring_create_virtqueue_pac= ked( return vq; } =20 -static int virtqueue_resize_packed(struct virtqueue *_vq, u32 num) +static int virtqueue_resize_packed(struct vring_virtqueue *vq, u32 num) { struct vring_virtqueue_packed vring_packed =3D {}; - struct vring_virtqueue *vq =3D to_vvq(_vq); - struct virtio_device *vdev =3D _vq->vdev; + struct virtio_device *vdev =3D vq->vq.vdev; int err; =20 if (vring_alloc_queue_packed(&vring_packed, vdev, num, vq->map)) @@ -2809,9 +2807,9 @@ int virtqueue_resize(struct virtqueue *_vq, u32 num, recycle_done(_vq); =20 if (vq->packed_ring) - err =3D virtqueue_resize_packed(_vq, num); + err =3D virtqueue_resize_packed(vq, num); else - err =3D virtqueue_resize_split(_vq, num); + err =3D virtqueue_resize_split(vq, num); =20 err_reset =3D virtqueue_enable_after_reset(_vq); if (err_reset) --=20 2.31.1 From nobody Sat Feb 7 15:21:59 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BB4F717BB35 for ; Mon, 20 Oct 2025 07:10:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944237; cv=none; b=HUuV0hc0tKgziA8XM7H1/gwSVEIdOSuuL7p0fVe20rUXZu/Mlu7rQsOqBgEkRWaJDm6vQ9hZQu9O5dwBld164NpwfgRKvmWO5CnGzhFpivFOgnGRMRs+wKDuRrxdM7mqDZEA8QazyT78khlZ7U5hIFPjGS+PqURE4DXvA+/OH1A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944237; c=relaxed/simple; bh=/P9F+XZD4Jw4baApW319M4i8p1Tx8WVEcfea/cWb69Q=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=NauQGLYz4Bg+4jECAClmtXPI5cd3ocxhcZQwEm3Vpx21eFsgR2WXZHc/P+MsVbg0erGKKEp8K8vGNpqiT9jymfXonuE/YGirJvsg06MO5Y7d6/qQ16PnYWV+Cjvc6p9oqlK9OGAtXXdcWE7iaeDE4puZ1+cavbNZ40rZc4ycSZo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=h/cHfe/W; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="h/cHfe/W" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1760944234; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RirLQpnZ2Apia0UR72xDNP7krHXcdWwsna0GSCf+FV4=; b=h/cHfe/WM2h9XSZray1C+bxZrnbbD/07r7kHKmCQ1Nt9ENGn4wTUXTKw5ctjf+JxFYiO1y +g3AsxvQlsivbGG28wElWPHfxxa1q6LAdCtVqtkEdkKh0O0xz6lkv9BdyBxKJe5vRE6w8T 3oZyih1GoWpFUfmdIMeCS5MgKZsRNko= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-327-ofHfmnAYNc-yE-oarHq6OA-1; Mon, 20 Oct 2025 03:10:33 -0400 X-MC-Unique: ofHfmnAYNc-yE-oarHq6OA-1 X-Mimecast-MFC-AGG-ID: ofHfmnAYNc-yE-oarHq6OA_1760944232 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 463F419560A7; Mon, 20 Oct 2025 07:10:32 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.113.29]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 26E9A300019F; Mon, 20 Oct 2025 07:10:28 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V8 05/19] virtio_ring: switch to use vring_virtqueue for virtqueue_kick_prepare variants Date: Mon, 20 Oct 2025 15:09:49 +0800 Message-ID: <20251020071003.28834-6-jasowang@redhat.com> In-Reply-To: <20251020071003.28834-1-jasowang@redhat.com> References: <20251020071003.28834-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Those variants are used internally so let's switch to use vring_virtqueue as parameter to be consistent with other internal virtqueue helpers. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 693671bac841..aadeab66e57c 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -717,9 +717,8 @@ static inline int virtqueue_add_split(struct virtqueue = *_vq, return -ENOMEM; } =20 -static bool virtqueue_kick_prepare_split(struct virtqueue *_vq) +static bool virtqueue_kick_prepare_split(struct vring_virtqueue *vq) { - struct vring_virtqueue *vq =3D to_vvq(_vq); u16 new, old; bool needs_kick; =20 @@ -736,12 +735,12 @@ static bool virtqueue_kick_prepare_split(struct virtq= ueue *_vq) LAST_ADD_TIME_INVALID(vq); =20 if (vq->event) { - needs_kick =3D vring_need_event(virtio16_to_cpu(_vq->vdev, + needs_kick =3D vring_need_event(virtio16_to_cpu(vq->vq.vdev, vring_avail_event(&vq->split.vring)), new, old); } else { needs_kick =3D !(vq->split.vring.used->flags & - cpu_to_virtio16(_vq->vdev, + cpu_to_virtio16(vq->vq.vdev, VRING_USED_F_NO_NOTIFY)); } END_USE(vq); @@ -1596,9 +1595,8 @@ static inline int virtqueue_add_packed(struct virtque= ue *_vq, return -EIO; } =20 -static bool virtqueue_kick_prepare_packed(struct virtqueue *_vq) +static bool virtqueue_kick_prepare_packed(struct vring_virtqueue *vq) { - struct vring_virtqueue *vq =3D to_vvq(_vq); u16 new, old, off_wrap, flags, wrap_counter, event_idx; bool needs_kick; union { @@ -2457,8 +2455,8 @@ bool virtqueue_kick_prepare(struct virtqueue *_vq) { struct vring_virtqueue *vq =3D to_vvq(_vq); =20 - return vq->packed_ring ? virtqueue_kick_prepare_packed(_vq) : - virtqueue_kick_prepare_split(_vq); + return vq->packed_ring ? virtqueue_kick_prepare_packed(vq) : + virtqueue_kick_prepare_split(vq); } EXPORT_SYMBOL_GPL(virtqueue_kick_prepare); =20 --=20 2.31.1 From nobody Sat Feb 7 15:21:59 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6C4DD2DFF13 for ; Mon, 20 Oct 2025 07:10:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944243; cv=none; b=uCluZwzdSKDLhl+0GGz2nUEtyHUXLEecfFCccLKig8XZzWagPUcZOv1gWQph4+4yil5VzcQVFT4WzWaNB1UEp4siiHBQwAzt6Y9eJDn0JVB/SrR/YryQjD81hx6e4EG6mGyOLgymrmAk94gKF6MPBL0ybd1FIbp4hZ8Qa7+E6Xc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944243; c=relaxed/simple; bh=2AF9JULOtNLoExoVfNQOkRMExvLAyJLGg/Da80HqSKA=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=SKJ2lhu7mgS+tOfMsztn2MYDn62qBlWuHo6jb+GDQ/kdbaLWJVvKFRFjavWiuNlpWLkyum1Hftb5A7YVBYXAVXZkarIxFJR9c2yu6U8haLzTO/LNcoAUoqZ4MIINJ6BGmffcGboLcR0Uke6Oyyney6ewC2Ct22vdM6C4mx+FjIQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=dI50a7xV; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="dI50a7xV" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1760944240; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TaiPNv+1QjEcx0X1VKNsOwZsuKWVqyQbYrUVmXFh3Qo=; b=dI50a7xV0p6aFAwbexaXpLmTdmW22Q+3EaKRTVbf3jKLjPoROTXu87d9QcgCYoM88oLElp g87Pyr4ro/jm498dJ65fH3+iefiyp6tRco93hMkMy+OgjeEwrNBbD8meQ74EaLftnW+o3X 7RRN1P2PRyOuwrPNLnogoV1yH1WIwUg= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-636-K0LxA86PPm-q9Ef9NQs94A-1; Mon, 20 Oct 2025 03:10:36 -0400 X-MC-Unique: K0LxA86PPm-q9Ef9NQs94A-1 X-Mimecast-MFC-AGG-ID: K0LxA86PPm-q9Ef9NQs94A_1760944236 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id EA9901800669; Mon, 20 Oct 2025 07:10:35 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.113.29]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id F363C30001A2; Mon, 20 Oct 2025 07:10:32 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V8 06/19] virtio_ring: switch to use vring_virtqueue for virtqueue_add variants Date: Mon, 20 Oct 2025 15:09:50 +0800 Message-ID: <20251020071003.28834-7-jasowang@redhat.com> In-Reply-To: <20251020071003.28834-1-jasowang@redhat.com> References: <20251020071003.28834-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Those variants are used internally so let's switch to use vring_virtqueue as parameter to be consistent with other internal virtqueue helpers. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 39 ++++++++++++++++++------------------ 1 file changed, 19 insertions(+), 20 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index aadeab66e57c..2c0c677cb6fc 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -476,7 +476,7 @@ static unsigned int vring_unmap_one_split(const struct = vring_virtqueue *vq, return extra->next; } =20 -static struct vring_desc *alloc_indirect_split(struct virtqueue *_vq, +static struct vring_desc *alloc_indirect_split(struct vring_virtqueue *vq, unsigned int total_sg, gfp_t gfp) { @@ -505,7 +505,7 @@ static struct vring_desc *alloc_indirect_split(struct v= irtqueue *_vq, return desc; } =20 -static inline unsigned int virtqueue_add_desc_split(struct virtqueue *vq, +static inline unsigned int virtqueue_add_desc_split(struct vring_virtqueue= *vq, struct vring_desc *desc, struct vring_desc_extra *extra, unsigned int i, @@ -513,11 +513,12 @@ static inline unsigned int virtqueue_add_desc_split(s= truct virtqueue *vq, unsigned int len, u16 flags, bool premapped) { + struct virtio_device *vdev =3D vq->vq.vdev; u16 next; =20 - desc[i].flags =3D cpu_to_virtio16(vq->vdev, flags); - desc[i].addr =3D cpu_to_virtio64(vq->vdev, addr); - desc[i].len =3D cpu_to_virtio32(vq->vdev, len); + desc[i].flags =3D cpu_to_virtio16(vdev, flags); + desc[i].addr =3D cpu_to_virtio64(vdev, addr); + desc[i].len =3D cpu_to_virtio32(vdev, len); =20 extra[i].addr =3D premapped ? DMA_MAPPING_ERROR : addr; extra[i].len =3D len; @@ -525,12 +526,12 @@ static inline unsigned int virtqueue_add_desc_split(s= truct virtqueue *vq, =20 next =3D extra[i].next; =20 - desc[i].next =3D cpu_to_virtio16(vq->vdev, next); + desc[i].next =3D cpu_to_virtio16(vdev, next); =20 return next; } =20 -static inline int virtqueue_add_split(struct virtqueue *_vq, +static inline int virtqueue_add_split(struct vring_virtqueue *vq, struct scatterlist *sgs[], unsigned int total_sg, unsigned int out_sgs, @@ -540,7 +541,6 @@ static inline int virtqueue_add_split(struct virtqueue = *_vq, bool premapped, gfp_t gfp) { - struct vring_virtqueue *vq =3D to_vvq(_vq); struct vring_desc_extra *extra; struct scatterlist *sg; struct vring_desc *desc; @@ -565,7 +565,7 @@ static inline int virtqueue_add_split(struct virtqueue = *_vq, head =3D vq->free_head; =20 if (virtqueue_use_indirect(vq, total_sg)) - desc =3D alloc_indirect_split(_vq, total_sg, gfp); + desc =3D alloc_indirect_split(vq, total_sg, gfp); else { desc =3D NULL; WARN_ON_ONCE(total_sg > vq->split.vring.num && !vq->indirect); @@ -612,7 +612,7 @@ static inline int virtqueue_add_split(struct virtqueue = *_vq, /* Note that we trust indirect descriptor * table since it use stream DMA mapping. */ - i =3D virtqueue_add_desc_split(_vq, desc, extra, i, addr, len, + i =3D virtqueue_add_desc_split(vq, desc, extra, i, addr, len, VRING_DESC_F_NEXT, premapped); } @@ -629,14 +629,14 @@ static inline int virtqueue_add_split(struct virtqueu= e *_vq, /* Note that we trust indirect descriptor * table since it use stream DMA mapping. */ - i =3D virtqueue_add_desc_split(_vq, desc, extra, i, addr, len, + i =3D virtqueue_add_desc_split(vq, desc, extra, i, addr, len, VRING_DESC_F_NEXT | VRING_DESC_F_WRITE, premapped); } } /* Last one doesn't continue. */ - desc[prev].flags &=3D cpu_to_virtio16(_vq->vdev, ~VRING_DESC_F_NEXT); + desc[prev].flags &=3D cpu_to_virtio16(vq->vq.vdev, ~VRING_DESC_F_NEXT); if (!indirect && vring_need_unmap_buffer(vq, &extra[prev])) vq->split.desc_extra[prev & (vq->split.vring.num - 1)].flags &=3D ~VRING_DESC_F_NEXT; @@ -649,7 +649,7 @@ static inline int virtqueue_add_split(struct virtqueue = *_vq, if (vring_mapping_error(vq, addr)) goto unmap_release; =20 - virtqueue_add_desc_split(_vq, vq->split.vring.desc, + virtqueue_add_desc_split(vq, vq->split.vring.desc, vq->split.desc_extra, head, addr, total_sg * sizeof(struct vring_desc), @@ -675,13 +675,13 @@ static inline int virtqueue_add_split(struct virtqueu= e *_vq, /* Put entry in available array (but don't update avail->idx until they * do sync). */ avail =3D vq->split.avail_idx_shadow & (vq->split.vring.num - 1); - vq->split.vring.avail->ring[avail] =3D cpu_to_virtio16(_vq->vdev, head); + vq->split.vring.avail->ring[avail] =3D cpu_to_virtio16(vq->vq.vdev, head); =20 /* Descriptors and available array need to be set before we expose the * new available array entries. */ virtio_wmb(vq->weak_barriers); vq->split.avail_idx_shadow++; - vq->split.vring.avail->idx =3D cpu_to_virtio16(_vq->vdev, + vq->split.vring.avail->idx =3D cpu_to_virtio16(vq->vq.vdev, vq->split.avail_idx_shadow); vq->num_added++; =20 @@ -691,7 +691,7 @@ static inline int virtqueue_add_split(struct virtqueue = *_vq, /* This is very unlikely, but theoretically possible. Kick * just in case. */ if (unlikely(vq->num_added =3D=3D (1 << 16) - 1)) - virtqueue_kick(_vq); + virtqueue_kick(&vq->vq); =20 return 0; =20 @@ -1440,7 +1440,7 @@ static int virtqueue_add_indirect_packed(struct vring= _virtqueue *vq, return -ENOMEM; } =20 -static inline int virtqueue_add_packed(struct virtqueue *_vq, +static inline int virtqueue_add_packed(struct vring_virtqueue *vq, struct scatterlist *sgs[], unsigned int total_sg, unsigned int out_sgs, @@ -1450,7 +1450,6 @@ static inline int virtqueue_add_packed(struct virtque= ue *_vq, bool premapped, gfp_t gfp) { - struct vring_virtqueue *vq =3D to_vvq(_vq); struct vring_packed_desc *desc; struct scatterlist *sg; unsigned int i, n, c, descs_used, err_idx, len; @@ -2262,9 +2261,9 @@ static inline int virtqueue_add(struct virtqueue *_vq, { struct vring_virtqueue *vq =3D to_vvq(_vq); =20 - return vq->packed_ring ? virtqueue_add_packed(_vq, sgs, total_sg, + return vq->packed_ring ? virtqueue_add_packed(vq, sgs, total_sg, out_sgs, in_sgs, data, ctx, premapped, gfp) : - virtqueue_add_split(_vq, sgs, total_sg, + virtqueue_add_split(vq, sgs, total_sg, out_sgs, in_sgs, data, ctx, premapped, gfp); } =20 --=20 2.31.1 From nobody Sat Feb 7 15:21:59 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8EB622E0408 for ; Mon, 20 Oct 2025 07:10:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944245; cv=none; b=QM0nYgz2WIKqKPvHmrxB1kcXDQa1oogVcoTo4cTYCmxDtt+N+pCm+wLrGv0Yhd0ZfFFmTBzFbzsT2e2QObQgQ6TXCk13izbgqe3sHBchEmff/GvPc8fiVTxAw9ZOCORc2ATn3nn/a06x98CzqVP+JwiWxSnswWlVX4pAmZEfyEI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944245; c=relaxed/simple; bh=7qDg5goKB2eLhHZ+Ed9qX3fTW71RZzlughkjBIvkQ7E=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=cIE8crOJbF9gYdroGFtReXXOEdxxH5sDWzQRgvaMPTv5Hp+hHp0hsLebcVIIzgj4T6QIAzIML37d9R4HhgXhqK5xECP9P57swMxlozKl5N2VDSKtgxiA0dHF21XhmnmWie4cSAjJ3kyhIOvCk+2nBuE8DLf87r2WjcwcEY4tTm0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=TGqk/tZU; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="TGqk/tZU" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1760944242; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CTDGrU9uhGdFfnzChF1KM1YGBCuNVqCio2exfzbo6VA=; b=TGqk/tZUPXZGtsSf438SzNaW/vCUIPfmIDfSAULwFoLTmb2j5FYTxm6BgB8iRhEGV2hJ1M lFsgWECrYtXrTZcOkRsFT62Rvza55xLIKE/VJQFc54PfuCS6lMp/P8U+jijGjVgPe0Lb6I HU5dQupB9lW4OE4UcnY8O+3HEjA6CZ4= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-644-xPsBO8hFPSitKtx5bHhSRQ-1; Mon, 20 Oct 2025 03:10:40 -0400 X-MC-Unique: xPsBO8hFPSitKtx5bHhSRQ-1 X-Mimecast-MFC-AGG-ID: xPsBO8hFPSitKtx5bHhSRQ_1760944239 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 888E618002C1; Mon, 20 Oct 2025 07:10:39 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.113.29]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id A5CA3300019F; Mon, 20 Oct 2025 07:10:36 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V8 07/19] virtio: switch to use vring_virtqueue for virtqueue_get variants Date: Mon, 20 Oct 2025 15:09:51 +0800 Message-ID: <20251020071003.28834-8-jasowang@redhat.com> In-Reply-To: <20251020071003.28834-1-jasowang@redhat.com> References: <20251020071003.28834-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Those variants are used internally so let's switch to use vring_virtqueue as parameter to be consistent with other internal virtqueue helpers. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 2c0c677cb6fc..9d084ee9f4d6 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -817,11 +817,10 @@ static bool more_used_split(const struct vring_virtqu= eue *vq) return virtqueue_poll_split(vq, vq->last_used_idx); } =20 -static void *virtqueue_get_buf_ctx_split(struct virtqueue *_vq, +static void *virtqueue_get_buf_ctx_split(struct vring_virtqueue *vq, unsigned int *len, void **ctx) { - struct vring_virtqueue *vq =3D to_vvq(_vq); void *ret; unsigned int i; u16 last_used; @@ -843,9 +842,9 @@ static void *virtqueue_get_buf_ctx_split(struct virtque= ue *_vq, virtio_rmb(vq->weak_barriers); =20 last_used =3D (vq->last_used_idx & (vq->split.vring.num - 1)); - i =3D virtio32_to_cpu(_vq->vdev, + i =3D virtio32_to_cpu(vq->vq.vdev, vq->split.vring.used->ring[last_used].id); - *len =3D virtio32_to_cpu(_vq->vdev, + *len =3D virtio32_to_cpu(vq->vq.vdev, vq->split.vring.used->ring[last_used].len); =20 if (unlikely(i >=3D vq->split.vring.num)) { @@ -867,7 +866,7 @@ static void *virtqueue_get_buf_ctx_split(struct virtque= ue *_vq, if (!(vq->split.avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT)) virtio_store_mb(vq->weak_barriers, &vring_used_event(&vq->split.vring), - cpu_to_virtio16(_vq->vdev, vq->last_used_idx)); + cpu_to_virtio16(vq->vq.vdev, vq->last_used_idx)); =20 LAST_ADD_TIME_INVALID(vq); =20 @@ -1721,11 +1720,10 @@ static bool more_used_packed(const struct vring_vir= tqueue *vq) return virtqueue_poll_packed(vq, READ_ONCE(vq->last_used_idx)); } =20 -static void *virtqueue_get_buf_ctx_packed(struct virtqueue *_vq, +static void *virtqueue_get_buf_ctx_packed(struct vring_virtqueue *vq, unsigned int *len, void **ctx) { - struct vring_virtqueue *vq =3D to_vvq(_vq); u16 last_used, id, last_used_idx; bool used_wrap_counter; void *ret; @@ -2525,8 +2523,8 @@ void *virtqueue_get_buf_ctx(struct virtqueue *_vq, un= signed int *len, { struct vring_virtqueue *vq =3D to_vvq(_vq); =20 - return vq->packed_ring ? virtqueue_get_buf_ctx_packed(_vq, len, ctx) : - virtqueue_get_buf_ctx_split(_vq, len, ctx); + return vq->packed_ring ? virtqueue_get_buf_ctx_packed(vq, len, ctx) : + virtqueue_get_buf_ctx_split(vq, len, ctx); } EXPORT_SYMBOL_GPL(virtqueue_get_buf_ctx); =20 --=20 2.31.1 From nobody Sat Feb 7 15:21:59 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4AEC72E1F02 for ; Mon, 20 Oct 2025 07:10:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944248; cv=none; b=LTUVosCmRJMiYlyjQsFIIk13ELcE0HLvk6mWj6C23/LQzL4ppKMBwjIQlcX3OGb5He17dkKHZZ0cJf9QEgcg/srgJ2hcYx3R2yBL48HdX4oKdJzhfaUAAXqlX+F8CPK3I78GiHVdr4QdqIqqzISv5sOyHw/GGfb+xoKHDUw5B1M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944248; c=relaxed/simple; bh=XEtnrHECrq/n5FpADL3LfMWZyQVu01mbzxn//nz4phE=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=chNzHdFy8mq4m3I3Ap6woSEwHfoWcnUpHyRntTOgqH+yLyMIOlURF+yIz1DVYRbE6iMAlCp+R96a5wIp3nx7aLoKpXXaCtGtbk43huL6oefEMjYQz6xyPMYVrskK2gZuEDgO/99bD0MKJJ7IRCcBnNKEKl16BIzH3k3zoATXRcE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=L3BcmfOh; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="L3BcmfOh" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1760944246; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Q4FPBkheUtQFJ5C3vs4XZzaAF/OMZmMv9SqfzyXlOTM=; b=L3BcmfOhoV6RIWaVULrbjILHyFJ3imv8vvU23WffNU3gzRc+6Wu2x8xM1J996eKsv2npDi fhLNCPKHg/QDRiwbdmMC1ziEGT1SkNFa3A0Whs/Cga24K+MQpGovYGaz2RLiKcDKNngD0Y r0/SHaW0kdxM+4BK/irSGb541nFj9lc= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-612-x9lxyAXLMHuoTsKisuWhag-1; Mon, 20 Oct 2025 03:10:44 -0400 X-MC-Unique: x9lxyAXLMHuoTsKisuWhag-1 X-Mimecast-MFC-AGG-ID: x9lxyAXLMHuoTsKisuWhag_1760944243 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 6F3CF18002F5; Mon, 20 Oct 2025 07:10:43 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.113.29]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 3C53630001A2; Mon, 20 Oct 2025 07:10:39 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V8 08/19] virtio_ring: switch to use vring_virtqueue for enable_cb_prepare variants Date: Mon, 20 Oct 2025 15:09:52 +0800 Message-ID: <20251020071003.28834-9-jasowang@redhat.com> In-Reply-To: <20251020071003.28834-1-jasowang@redhat.com> References: <20251020071003.28834-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Those variants are used internally so let's switch to use vring_virtqueue as parameter to be consistent with other internal virtqueue helpers. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 9d084ee9f4d6..f46ebc60f911 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -898,9 +898,8 @@ static void virtqueue_disable_cb_split(struct virtqueue= *_vq) } } =20 -static unsigned int virtqueue_enable_cb_prepare_split(struct virtqueue *_v= q) +static unsigned int virtqueue_enable_cb_prepare_split(struct vring_virtque= ue *vq) { - struct vring_virtqueue *vq =3D to_vvq(_vq); u16 last_used_idx; =20 START_USE(vq); @@ -914,10 +913,10 @@ static unsigned int virtqueue_enable_cb_prepare_split= (struct virtqueue *_vq) vq->split.avail_flags_shadow &=3D ~VRING_AVAIL_F_NO_INTERRUPT; if (!vq->event) vq->split.vring.avail->flags =3D - cpu_to_virtio16(_vq->vdev, + cpu_to_virtio16(vq->vq.vdev, vq->split.avail_flags_shadow); } - vring_used_event(&vq->split.vring) =3D cpu_to_virtio16(_vq->vdev, + vring_used_event(&vq->split.vring) =3D cpu_to_virtio16(vq->vq.vdev, last_used_idx =3D vq->last_used_idx); END_USE(vq); return last_used_idx; @@ -1807,10 +1806,8 @@ static void virtqueue_disable_cb_packed(struct virtq= ueue *_vq) } } =20 -static unsigned int virtqueue_enable_cb_prepare_packed(struct virtqueue *_= vq) +static unsigned int virtqueue_enable_cb_prepare_packed(struct vring_virtqu= eue *vq) { - struct vring_virtqueue *vq =3D to_vvq(_vq); - START_USE(vq); =20 /* @@ -2572,8 +2569,8 @@ unsigned int virtqueue_enable_cb_prepare(struct virtq= ueue *_vq) if (vq->event_triggered) vq->event_triggered =3D false; =20 - return vq->packed_ring ? virtqueue_enable_cb_prepare_packed(_vq) : - virtqueue_enable_cb_prepare_split(_vq); + return vq->packed_ring ? virtqueue_enable_cb_prepare_packed(vq) : + virtqueue_enable_cb_prepare_split(vq); } EXPORT_SYMBOL_GPL(virtqueue_enable_cb_prepare); =20 --=20 2.31.1 From nobody Sat Feb 7 15:21:59 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 24B422DE70B for ; Mon, 20 Oct 2025 07:11:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944263; cv=none; b=SM0JkihgyUkUlKYanZBiPp2WqP3dqgDfVdlBTcqttn4IaDO4xz5r2C1F0wd6VPiuO40Pk0gUWnDCamuktlLPVV/HgWQ1sRNATZ3m/SV5OrWx0Yqjfim/ZmSTnxpnqDqk+1MZlINDP+XhLV2ajuVLzTPXcg5RZ1Bl07fWQtLtL10= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944263; c=relaxed/simple; bh=B5xA+ac81zYIhYc/+K2WNv21b+y7giMM4O33HDQSuh8=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Laj5DO75vYPSulp5z9wqP2d1fpcmreLvkjLMHWmCgvdimKYSpcUP0wJlTe5c1DdBGO2dlu6dkQRU1WKLgobzlJr5jOjaSVn4afSdP93JS6riaqwRL+e6zsijctoWEZhP2K8ovhNa2e8xyUUhVniUuZoOf/oxzdXPno43djdqlxg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=NKqGjUgV; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="NKqGjUgV" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1760944260; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eNJ+gQqwInDlOP5Vb/dsZyS5rNI1v31bs1pRaNUkug8=; b=NKqGjUgVXkznvZwdJTA11jpxxUW1qze9Myb2xpH4HEC694EvkP/FnLAcXdaoZGrDZSAOGx CLg27JSwAVy7KHqnNOx+C/NwYp5mMcKb7RUxM+I/ppw3If4n4LfFN3dov4ONJcJr2tj0gB yhqgZe5aKJLOjoSzyKvT1X27jYCIngs= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-173-FRCGiauOM_2uDw9isJIBrg-1; Mon, 20 Oct 2025 03:10:51 -0400 X-MC-Unique: FRCGiauOM_2uDw9isJIBrg-1 X-Mimecast-MFC-AGG-ID: FRCGiauOM_2uDw9isJIBrg_1760944247 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 110A018002F9; Mon, 20 Oct 2025 07:10:47 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.113.29]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 22384300019F; Mon, 20 Oct 2025 07:10:43 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V8 09/19] virtio_ring: use vring_virtqueue for enable_cb_delayed variants Date: Mon, 20 Oct 2025 15:09:53 +0800 Message-ID: <20251020071003.28834-10-jasowang@redhat.com> In-Reply-To: <20251020071003.28834-1-jasowang@redhat.com> References: <20251020071003.28834-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Those variants are used internally so let's switch to use vring_virtqueue as parameter to be consistent with other internal virtqueue helpers. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index f46ebc60f911..61ce9645c3a6 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -922,9 +922,8 @@ static unsigned int virtqueue_enable_cb_prepare_split(s= truct vring_virtqueue *vq return last_used_idx; } =20 -static bool virtqueue_enable_cb_delayed_split(struct virtqueue *_vq) +static bool virtqueue_enable_cb_delayed_split(struct vring_virtqueue *vq) { - struct vring_virtqueue *vq =3D to_vvq(_vq); u16 bufs; =20 START_USE(vq); @@ -938,7 +937,7 @@ static bool virtqueue_enable_cb_delayed_split(struct vi= rtqueue *_vq) vq->split.avail_flags_shadow &=3D ~VRING_AVAIL_F_NO_INTERRUPT; if (!vq->event) vq->split.vring.avail->flags =3D - cpu_to_virtio16(_vq->vdev, + cpu_to_virtio16(vq->vq.vdev, vq->split.avail_flags_shadow); } /* TODO: tune this threshold */ @@ -946,9 +945,9 @@ static bool virtqueue_enable_cb_delayed_split(struct vi= rtqueue *_vq) =20 virtio_store_mb(vq->weak_barriers, &vring_used_event(&vq->split.vring), - cpu_to_virtio16(_vq->vdev, vq->last_used_idx + bufs)); + cpu_to_virtio16(vq->vq.vdev, vq->last_used_idx + bufs)); =20 - if (unlikely((u16)(virtio16_to_cpu(_vq->vdev, vq->split.vring.used->idx) + if (unlikely((u16)(virtio16_to_cpu(vq->vq.vdev, vq->split.vring.used->idx) - vq->last_used_idx) > bufs)) { END_USE(vq); return false; @@ -1837,9 +1836,8 @@ static unsigned int virtqueue_enable_cb_prepare_packe= d(struct vring_virtqueue *v return vq->last_used_idx; } =20 -static bool virtqueue_enable_cb_delayed_packed(struct virtqueue *_vq) +static bool virtqueue_enable_cb_delayed_packed(struct vring_virtqueue *vq) { - struct vring_virtqueue *vq =3D to_vvq(_vq); u16 used_idx, wrap_counter, last_used_idx; u16 bufs; =20 @@ -2635,8 +2633,8 @@ bool virtqueue_enable_cb_delayed(struct virtqueue *_v= q) if (vq->event_triggered) data_race(vq->event_triggered =3D false); =20 - return vq->packed_ring ? virtqueue_enable_cb_delayed_packed(_vq) : - virtqueue_enable_cb_delayed_split(_vq); + return vq->packed_ring ? virtqueue_enable_cb_delayed_packed(vq) : + virtqueue_enable_cb_delayed_split(vq); } EXPORT_SYMBOL_GPL(virtqueue_enable_cb_delayed); =20 --=20 2.31.1 From nobody Sat Feb 7 15:21:59 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5FF2B2DBF45 for ; Mon, 20 Oct 2025 07:10:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944257; cv=none; b=VxrJK7FCeYtl96EBQxeoovWQyI6h6XdP9JFwsJMYw4U0sBNuMXM6g92+G19vLLVu8BYYob8UWTbfLgDNeZi0ybJTyJ5uRzKGWHzuqvrvxnWkYny6tAObUHdheCyoHA2ilh6Po0WHQ2e4c1OBKB3seM02hQUAsDKI1i5I8niThvE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944257; c=relaxed/simple; bh=RR0+NM9tP19wOBCCAz+KfyimeAaDE1SQocegbFBadpg=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=bdKRjNhtuTXg8tJWzzl7MnwQi9op2LWMyZl4TXQbXz8LJX28//vYwT5khBdVB53YlugkcLZWKMVit9S4zQdoHX/xglBzXGhoGuk4oawZUIs+er5jCXFU9aH6j+jFvaT7zcci9qC2ld2s6D8GqMUob8ETj7olVJZez6clMkbkKrw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=ISFKS8Gv; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="ISFKS8Gv" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1760944255; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Rt9CavJgBPAgRO3Nlj+9uYFblWnPYDRcaD5WdFy9CE0=; b=ISFKS8GvyVUZ1V6lG50Dwth8QyU8utMyLazQWCWw1ogwHCr8XSeiYuZt80hbwjxCXKPGAs CRwE3Le2T5VBLfHnz1EYN/h9hdHxupq3jmn/eRNa9bLAjnVBNKtb4yhWoATjz5w9N3FXre coSvUQYASaqAP7heNai/iP4GXN9Q4UU= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-516-hsUvmDxbPPCqRX_dX8ItAA-1; Mon, 20 Oct 2025 03:10:51 -0400 X-MC-Unique: hsUvmDxbPPCqRX_dX8ItAA-1 X-Mimecast-MFC-AGG-ID: hsUvmDxbPPCqRX_dX8ItAA_1760944251 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id D654119560B0; Mon, 20 Oct 2025 07:10:50 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.113.29]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id EDFAA300019F; Mon, 20 Oct 2025 07:10:47 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V8 10/19] virtio_ring: switch to use vring_virtqueue for disable_cb variants Date: Mon, 20 Oct 2025 15:09:54 +0800 Message-ID: <20251020071003.28834-11-jasowang@redhat.com> In-Reply-To: <20251020071003.28834-1-jasowang@redhat.com> References: <20251020071003.28834-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Those variants are used internally so let's switch to use vring_virtqueue as parameter to be consistent with other internal virtqueue helpers. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 61ce9645c3a6..768933daba9a 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -874,10 +874,8 @@ static void *virtqueue_get_buf_ctx_split(struct vring_= virtqueue *vq, return ret; } =20 -static void virtqueue_disable_cb_split(struct virtqueue *_vq) +static void virtqueue_disable_cb_split(struct vring_virtqueue *vq) { - struct vring_virtqueue *vq =3D to_vvq(_vq); - if (!(vq->split.avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT)) { vq->split.avail_flags_shadow |=3D VRING_AVAIL_F_NO_INTERRUPT; =20 @@ -893,7 +891,7 @@ static void virtqueue_disable_cb_split(struct virtqueue= *_vq) vring_used_event(&vq->split.vring) =3D 0x0; else vq->split.vring.avail->flags =3D - cpu_to_virtio16(_vq->vdev, + cpu_to_virtio16(vq->vq.vdev, vq->split.avail_flags_shadow); } } @@ -1786,10 +1784,8 @@ static void *virtqueue_get_buf_ctx_packed(struct vri= ng_virtqueue *vq, return ret; } =20 -static void virtqueue_disable_cb_packed(struct virtqueue *_vq) +static void virtqueue_disable_cb_packed(struct vring_virtqueue *vq) { - struct vring_virtqueue *vq =3D to_vvq(_vq); - if (vq->packed.event_flags_shadow !=3D VRING_PACKED_EVENT_FLAG_DISABLE) { vq->packed.event_flags_shadow =3D VRING_PACKED_EVENT_FLAG_DISABLE; =20 @@ -2542,9 +2538,9 @@ void virtqueue_disable_cb(struct virtqueue *_vq) struct vring_virtqueue *vq =3D to_vvq(_vq); =20 if (vq->packed_ring) - virtqueue_disable_cb_packed(_vq); + virtqueue_disable_cb_packed(vq); else - virtqueue_disable_cb_split(_vq); + virtqueue_disable_cb_split(vq); } EXPORT_SYMBOL_GPL(virtqueue_disable_cb); =20 --=20 2.31.1 From nobody Sat Feb 7 15:21:59 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 21DC22DBF47 for ; Mon, 20 Oct 2025 07:10:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944259; cv=none; b=N4hcFl3IEEjLQfy5DKGTbpc/OM5JwD58PbGjE4e55BH+9G8cML4XhcNbzQpQxR3fVXa+oBzQIdFnleAxA6UEjlQmzvtuTwYJBsMrzTEBVi8QoXN9UOwJrPQCzhS0qcYsAHQ5a0LcPUl7GgwyhioaiErJxJ2OpWWVmTRX8JYSkfE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944259; c=relaxed/simple; bh=MALz/FofoEbSHEf6mQJP1GyBxEkwL9YayW4mkSbywGE=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=VGigvgN1R5sIFNPfjs3LF2lYygjUBMrY+PsWaQsJh7Gnsqh7gKDELuTpFwQuw1Nbkf+1xIbj4NHeLO36+RBrSknpmWoNZt8nU6dxAMWfzQVxBaShFGN7u+JGv9uRYPyVptpNW0vPXUO9Pbyk6JFCY8pEEi/Zf70Z/9+aTsSUpBw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=LVajvaj5; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="LVajvaj5" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1760944257; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DYxyVzlU8ucx3XFZYxzyS++vlvHqCYS1VTslr24T/xE=; b=LVajvaj5NF7mBL73QOyEDkGNRljOOrogte6rXkJgAlCIDRguPHuQtF3pyOawv2QV5r/DJs bKbzqSPCMDgXWnFzAFrQYR+aC97G2It6vQ24mQrzevenkwdDJFejbAfmlyJ1bPqUDGQVGS DdYWawGFlr6meyDys6tQ9Jb3CVUKW10= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-283-Tlwm4jBrNZyQj-wTpHmEoQ-1; Mon, 20 Oct 2025 03:10:55 -0400 X-MC-Unique: Tlwm4jBrNZyQj-wTpHmEoQ-1 X-Mimecast-MFC-AGG-ID: Tlwm4jBrNZyQj-wTpHmEoQ_1760944254 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 646E818002EC; Mon, 20 Oct 2025 07:10:54 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.113.29]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 894F4300019F; Mon, 20 Oct 2025 07:10:51 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V8 11/19] virtio_ring: switch to use vring_virtqueue for detach_unused_buf variants Date: Mon, 20 Oct 2025 15:09:55 +0800 Message-ID: <20251020071003.28834-12-jasowang@redhat.com> In-Reply-To: <20251020071003.28834-1-jasowang@redhat.com> References: <20251020071003.28834-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Those variants are used internally so let's switch to use vring_virtqueue as parameter to be consistent with other internal virtqueue helpers. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 768933daba9a..58c03a8aab85 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -955,9 +955,8 @@ static bool virtqueue_enable_cb_delayed_split(struct vr= ing_virtqueue *vq) return true; } =20 -static void *virtqueue_detach_unused_buf_split(struct virtqueue *_vq) +static void *virtqueue_detach_unused_buf_split(struct vring_virtqueue *vq) { - struct vring_virtqueue *vq =3D to_vvq(_vq); unsigned int i; void *buf; =20 @@ -970,7 +969,7 @@ static void *virtqueue_detach_unused_buf_split(struct v= irtqueue *_vq) buf =3D vq->split.desc_state[i].data; detach_buf_split(vq, i, NULL); vq->split.avail_idx_shadow--; - vq->split.vring.avail->idx =3D cpu_to_virtio16(_vq->vdev, + vq->split.vring.avail->idx =3D cpu_to_virtio16(vq->vq.vdev, vq->split.avail_idx_shadow); END_USE(vq); return buf; @@ -1892,9 +1891,8 @@ static bool virtqueue_enable_cb_delayed_packed(struct= vring_virtqueue *vq) return true; } =20 -static void *virtqueue_detach_unused_buf_packed(struct virtqueue *_vq) +static void *virtqueue_detach_unused_buf_packed(struct vring_virtqueue *vq) { - struct vring_virtqueue *vq =3D to_vvq(_vq); unsigned int i; void *buf; =20 @@ -2646,8 +2644,8 @@ void *virtqueue_detach_unused_buf(struct virtqueue *_= vq) { struct vring_virtqueue *vq =3D to_vvq(_vq); =20 - return vq->packed_ring ? virtqueue_detach_unused_buf_packed(_vq) : - virtqueue_detach_unused_buf_split(_vq); + return vq->packed_ring ? virtqueue_detach_unused_buf_packed(vq) : + virtqueue_detach_unused_buf_split(vq); } EXPORT_SYMBOL_GPL(virtqueue_detach_unused_buf); =20 --=20 2.31.1 From nobody Sat Feb 7 15:21:59 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CD08C2E5400 for ; Mon, 20 Oct 2025 07:11:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944263; cv=none; b=En9RX/JsOzdhp76iSDurAj/YvLGZMDNh2B24RmgEpAlnMVYbn+4bS30vlwYqvDH/qA9no2TU2I51rUTTcT/jBg+/W/S68GqWARvSwnrhAwmPnCeYq7JPeYUarnVrSf9bVwgNF0TCQ5vERPJ00ixQOrbmLMfjWYhTmJ7ksvJoyus= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944263; c=relaxed/simple; bh=6fux0mFBeJyCMolbqj8dnkmRAAZQiLMWN8ZAIi12dRY=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TaWUMLcpQ8QERLZxA9Ib1MPtJYTW+MFEHkKiSvpAWs1L+KODvuXy8mMQA5H8p5Epz+h7QPRh/n+AgKEyqAVbhnDJZ7/NJc7+aTgj5jYgmIYjGt1my9YFHuEl5FtisYpeTHwCqIGpNUViK0J4PAduA2bDsghrxf19RGZqiYD8Mg8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=gNlR1uNW; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="gNlR1uNW" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1760944260; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=a+U3weGKyjXNfQsssiioiLWp1O/UIc0AXkQ+N/oPFdY=; b=gNlR1uNWs2YmsTNXCBSRrPyT6Y0aiAvODTrVUS5ERgdKWZy5QsBOa8iVxLw823Xy9tiErC lpEDqx7V8L2CcF03E87gYdiqf+rJG6ufSC1rdv+ukMADeqkvxI9+Pd7y/2PDbT5ASskpBA +FMMWPBusddHEHCnb//42l4H/YwFW3I= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-158-364wItU9Ow-HGoZBJGK1oQ-1; Mon, 20 Oct 2025 03:10:59 -0400 X-MC-Unique: 364wItU9Ow-HGoZBJGK1oQ-1 X-Mimecast-MFC-AGG-ID: 364wItU9Ow-HGoZBJGK1oQ_1760944258 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 135F418001C6; Mon, 20 Oct 2025 07:10:58 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.113.29]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 174F8300019F; Mon, 20 Oct 2025 07:10:54 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V8 12/19] virtio_ring: switch to use unsigned int for virtqueue_poll_packed() Date: Mon, 20 Oct 2025 15:09:56 +0800 Message-ID: <20251020071003.28834-13-jasowang@redhat.com> In-Reply-To: <20251020071003.28834-1-jasowang@redhat.com> References: <20251020071003.28834-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Content-Type: text/plain; charset="utf-8" Switch to use unsigned int for virtqueue_poll_packed() to match virtqueue_poll() and virtqueue_poll_split() and ease the abstraction the virtqueue ops. Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 58c03a8aab85..73dcc6984e33 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -1699,7 +1699,8 @@ static inline bool is_used_desc_packed(const struct v= ring_virtqueue *vq, return avail =3D=3D used && used =3D=3D used_wrap_counter; } =20 -static bool virtqueue_poll_packed(const struct vring_virtqueue *vq, u16 of= f_wrap) +static bool virtqueue_poll_packed(const struct vring_virtqueue *vq, + unsigned int off_wrap) { bool wrap_counter; u16 used_idx; --=20 2.31.1 From nobody Sat Feb 7 15:21:59 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E4AFE2DECB0 for ; Mon, 20 Oct 2025 07:11:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944270; cv=none; b=qrwaI7flxo9hS6z7FSNifmolyPtgyfvt7LwRsBUaHinqhuI7Rd5x2s3afgPgRnaVq9U/JWHumGKoIOeSIzqKu9A3HAsfEAvxBQgHC3kUgyV4iOtR0gqbV7i4bD+gjILcDGXDGpTwoPty+aO8q8yJQkyQ59zbkVu6Do8TiFpR3tk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944270; c=relaxed/simple; bh=n4qnZmnfS2mwv2BpThe8M/xasa9JEdJVRODsbBiF1GI=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=rIzscuLXDYlxmJoHzORGvscFh0YpMq8MIDeE+BiRzae+cgSaOIbwUvoi5xXf4jIlTPG1RsB4Ph4T0fHsVZOmoaAeZ8qmqaDjn1mJHfwbz25IqI96hOcjowhV6pTFBDuBEGmf9f2q6sWFjA86InCWjj6qn5H8aJrXx3CDBvx4rLs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=RuWFuHGN; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="RuWFuHGN" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1760944266; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kS17Zk7KvDYvRR+H/pC5KiOZrwyrXDmQQ+P4FZ/f9Z0=; b=RuWFuHGNOdrMgeQI33sTGALe10aTmTrzR+R4mzXeXgYIERI6ZooZFwpwuefSQx/fFc7uK4 DR/zsckERzGuJpajcpXjNorsEMNSOaFPLJqAj7wwrwqb9E8LNI9H3RGvSViAG7QWRlSGQG 9RAqO/VshKFN64hdljFiDqnl8AVi744= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-453-DnHhwiRhM2GUyEaDMZvGTw-1; Mon, 20 Oct 2025 03:11:02 -0400 X-MC-Unique: DnHhwiRhM2GUyEaDMZvGTw-1 X-Mimecast-MFC-AGG-ID: DnHhwiRhM2GUyEaDMZvGTw_1760944261 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id A58B5195609F; Mon, 20 Oct 2025 07:11:01 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.113.29]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id BAD12300019F; Mon, 20 Oct 2025 07:10:58 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V8 13/19] virtio_ring: introduce virtqueue ops Date: Mon, 20 Oct 2025 15:09:57 +0800 Message-ID: <20251020071003.28834-14-jasowang@redhat.com> In-Reply-To: <20251020071003.28834-1-jasowang@redhat.com> References: <20251020071003.28834-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 This patch introduces virtqueue ops which is a set of the callbacks that will be called for different queue layout or features. This would help to avoid branches for split/packed and will ease the future implementation like in order. Note that in order to eliminate the indirect calls this patch uses global array of const ops to allow compiler to avoid indirect branches. Tested with CONFIG_MITIGATION_RETPOLINE, no performance differences were noticed. Acked-by: Eugenio P=C3=A9rez Suggested-by: Michael S. Tsirkin Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 174 ++++++++++++++++++++++++++--------- 1 file changed, 131 insertions(+), 43 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 73dcc6984e33..37b16ef906a4 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -67,6 +67,12 @@ #define LAST_ADD_TIME_INVALID(vq) #endif =20 +enum vq_layout { + SPLIT =3D 0, + PACKED, + VQ_TYPE_MAX, +}; + struct vring_desc_state_split { void *data; /* Data for callback. */ =20 @@ -159,12 +165,29 @@ struct vring_virtqueue_packed { size_t event_size_in_bytes; }; =20 +struct vring_virtqueue; + +struct virtqueue_ops { + int (*add)(struct vring_virtqueue *_vq, struct scatterlist *sgs[], + unsigned int total_sg, unsigned int out_sgs, + unsigned int in_sgs, void *data, + void *ctx, bool premapped, gfp_t gfp); + void *(*get)(struct vring_virtqueue *vq, unsigned int *len, void **ctx); + bool (*kick_prepare)(struct vring_virtqueue *vq); + void (*disable_cb)(struct vring_virtqueue *vq); + bool (*enable_cb_delayed)(struct vring_virtqueue *vq); + unsigned int (*enable_cb_prepare)(struct vring_virtqueue *vq); + bool (*poll)(const struct vring_virtqueue *vq, + unsigned int last_used_idx); + void *(*detach_unused_buf)(struct vring_virtqueue *vq); + bool (*more_used)(const struct vring_virtqueue *vq); + int (*resize)(struct vring_virtqueue *vq, u32 num); + void (*reset)(struct vring_virtqueue *vq); +}; + struct vring_virtqueue { struct virtqueue vq; =20 - /* Is this a packed ring? */ - bool packed_ring; - /* Is DMA API used? */ bool use_map_api; =20 @@ -180,6 +203,8 @@ struct vring_virtqueue { /* Host publishes avail event idx */ bool event; =20 + enum vq_layout layout; + /* Head of free buffer list. */ unsigned int free_head; /* Number we've added since last sync. */ @@ -231,6 +256,12 @@ static void vring_free(struct virtqueue *_vq); =20 #define to_vvq(_vq) container_of_const(_vq, struct vring_virtqueue, vq) =20 + +static inline bool virtqueue_is_packed(const struct vring_virtqueue *vq) +{ + return vq->layout =3D=3D PACKED; +} + static bool virtqueue_use_indirect(const struct vring_virtqueue *vq, unsigned int total_sg) { @@ -433,7 +464,7 @@ static void virtqueue_init(struct vring_virtqueue *vq, = u32 num) { vq->vq.num_free =3D num; =20 - if (vq->packed_ring) + if (virtqueue_is_packed(vq)) vq->last_used_idx =3D 0 | (1 << VRING_PACKED_EVENT_F_WRAP_CTR); else vq->last_used_idx =3D 0; @@ -1122,6 +1153,8 @@ static int vring_alloc_queue_split(struct vring_virtq= ueue_split *vring_split, return 0; } =20 +static const struct virtqueue_ops split_ops; + static struct virtqueue *__vring_new_virtqueue_split(unsigned int index, struct vring_virtqueue_split *vring_split, struct virtio_device *vdev, @@ -1139,7 +1172,7 @@ static struct virtqueue *__vring_new_virtqueue_split(= unsigned int index, if (!vq) return NULL; =20 - vq->packed_ring =3D false; + vq->layout =3D SPLIT; vq->vq.callback =3D callback; vq->vq.vdev =3D vdev; vq->vq.name =3D name; @@ -2077,6 +2110,8 @@ static void virtqueue_reset_packed(struct vring_virtq= ueue *vq) virtqueue_vring_init_packed(&vq->packed, !!vq->vq.callback); } =20 +static const struct virtqueue_ops packed_ops; + static struct virtqueue *__vring_new_virtqueue_packed(unsigned int index, struct vring_virtqueue_packed *vring_packed, struct virtio_device *vdev, @@ -2107,7 +2142,7 @@ static struct virtqueue *__vring_new_virtqueue_packed= (unsigned int index, #else vq->broken =3D false; #endif - vq->packed_ring =3D true; + vq->layout =3D PACKED; vq->map =3D map; vq->use_map_api =3D vring_use_map_api(vdev); =20 @@ -2195,6 +2230,39 @@ static int virtqueue_resize_packed(struct vring_virt= queue *vq, u32 num) return -ENOMEM; } =20 +static const struct virtqueue_ops split_ops =3D { + .add =3D virtqueue_add_split, + .get =3D virtqueue_get_buf_ctx_split, + .kick_prepare =3D virtqueue_kick_prepare_split, + .disable_cb =3D virtqueue_disable_cb_split, + .enable_cb_delayed =3D virtqueue_enable_cb_delayed_split, + .enable_cb_prepare =3D virtqueue_enable_cb_prepare_split, + .poll =3D virtqueue_poll_split, + .detach_unused_buf =3D virtqueue_detach_unused_buf_split, + .more_used =3D more_used_split, + .resize =3D virtqueue_resize_split, + .reset =3D virtqueue_reset_split, +}; + +static const struct virtqueue_ops packed_ops =3D { + .add =3D virtqueue_add_packed, + .get =3D virtqueue_get_buf_ctx_packed, + .kick_prepare =3D virtqueue_kick_prepare_packed, + .disable_cb =3D virtqueue_disable_cb_packed, + .enable_cb_delayed =3D virtqueue_enable_cb_delayed_packed, + .enable_cb_prepare =3D virtqueue_enable_cb_prepare_packed, + .poll =3D virtqueue_poll_packed, + .detach_unused_buf =3D virtqueue_detach_unused_buf_packed, + .more_used =3D more_used_packed, + .resize =3D virtqueue_resize_packed, + .reset =3D virtqueue_reset_packed, +}; + +static const struct virtqueue_ops *const all_ops[VQ_TYPE_MAX] =3D { + [SPLIT] =3D &split_ops, + [PACKED] =3D &packed_ops +}; + static int virtqueue_disable_and_recycle(struct virtqueue *_vq, void (*recycle)(struct virtqueue *vq, void *buf)) { @@ -2237,6 +2305,39 @@ static int virtqueue_enable_after_reset(struct virtq= ueue *_vq) * Generic functions and exported symbols. */ =20 +#define VIRTQUEUE_CALL(vq, op, ...) \ + ({ \ + typeof(all_ops[SPLIT]->op(vq, ##__VA_ARGS__)) ret; \ + \ + switch (vq->layout) { \ + case SPLIT: \ + ret =3D all_ops[SPLIT]->op(vq, ##__VA_ARGS__); \ + break; \ + case PACKED: \ + ret =3D all_ops[PACKED]->op(vq, ##__VA_ARGS__); \ + break; \ + default: \ + BUG(); \ + break; \ + } \ + ret; \ +}) + +#define VOID_VIRTQUEUE_CALL(vq, op, ...) \ + ({ \ + switch ((vq)->layout) { \ + case SPLIT: \ + all_ops[SPLIT]->op(vq, ##__VA_ARGS__); \ + break; \ + case PACKED: \ + all_ops[PACKED]->op(vq, ##__VA_ARGS__); \ + break; \ + default: \ + BUG(); \ + break; \ + } \ +}) + static inline int virtqueue_add(struct virtqueue *_vq, struct scatterlist *sgs[], unsigned int total_sg, @@ -2249,10 +2350,9 @@ static inline int virtqueue_add(struct virtqueue *_v= q, { struct vring_virtqueue *vq =3D to_vvq(_vq); =20 - return vq->packed_ring ? virtqueue_add_packed(vq, sgs, total_sg, - out_sgs, in_sgs, data, ctx, premapped, gfp) : - virtqueue_add_split(vq, sgs, total_sg, - out_sgs, in_sgs, data, ctx, premapped, gfp); + return VIRTQUEUE_CALL(vq, add, sgs, total_sg, + out_sgs, in_sgs, data, + ctx, premapped, gfp); } =20 /** @@ -2442,8 +2542,7 @@ bool virtqueue_kick_prepare(struct virtqueue *_vq) { struct vring_virtqueue *vq =3D to_vvq(_vq); =20 - return vq->packed_ring ? virtqueue_kick_prepare_packed(vq) : - virtqueue_kick_prepare_split(vq); + return VIRTQUEUE_CALL(vq, kick_prepare); } EXPORT_SYMBOL_GPL(virtqueue_kick_prepare); =20 @@ -2513,8 +2612,7 @@ void *virtqueue_get_buf_ctx(struct virtqueue *_vq, un= signed int *len, { struct vring_virtqueue *vq =3D to_vvq(_vq); =20 - return vq->packed_ring ? virtqueue_get_buf_ctx_packed(vq, len, ctx) : - virtqueue_get_buf_ctx_split(vq, len, ctx); + return VIRTQUEUE_CALL(vq, get, len, ctx); } EXPORT_SYMBOL_GPL(virtqueue_get_buf_ctx); =20 @@ -2536,10 +2634,7 @@ void virtqueue_disable_cb(struct virtqueue *_vq) { struct vring_virtqueue *vq =3D to_vvq(_vq); =20 - if (vq->packed_ring) - virtqueue_disable_cb_packed(vq); - else - virtqueue_disable_cb_split(vq); + VOID_VIRTQUEUE_CALL(vq, disable_cb); } EXPORT_SYMBOL_GPL(virtqueue_disable_cb); =20 @@ -2562,8 +2657,7 @@ unsigned int virtqueue_enable_cb_prepare(struct virtq= ueue *_vq) if (vq->event_triggered) vq->event_triggered =3D false; =20 - return vq->packed_ring ? virtqueue_enable_cb_prepare_packed(vq) : - virtqueue_enable_cb_prepare_split(vq); + return VIRTQUEUE_CALL(vq, enable_cb_prepare); } EXPORT_SYMBOL_GPL(virtqueue_enable_cb_prepare); =20 @@ -2584,8 +2678,8 @@ bool virtqueue_poll(struct virtqueue *_vq, unsigned i= nt last_used_idx) return false; =20 virtio_mb(vq->weak_barriers); - return vq->packed_ring ? virtqueue_poll_packed(vq, last_used_idx) : - virtqueue_poll_split(vq, last_used_idx); + + return VIRTQUEUE_CALL(vq, poll, last_used_idx); } EXPORT_SYMBOL_GPL(virtqueue_poll); =20 @@ -2628,8 +2722,7 @@ bool virtqueue_enable_cb_delayed(struct virtqueue *_v= q) if (vq->event_triggered) data_race(vq->event_triggered =3D false); =20 - return vq->packed_ring ? virtqueue_enable_cb_delayed_packed(vq) : - virtqueue_enable_cb_delayed_split(vq); + return VIRTQUEUE_CALL(vq, enable_cb_delayed); } EXPORT_SYMBOL_GPL(virtqueue_enable_cb_delayed); =20 @@ -2645,14 +2738,13 @@ void *virtqueue_detach_unused_buf(struct virtqueue = *_vq) { struct vring_virtqueue *vq =3D to_vvq(_vq); =20 - return vq->packed_ring ? virtqueue_detach_unused_buf_packed(vq) : - virtqueue_detach_unused_buf_split(vq); + return VIRTQUEUE_CALL(vq, detach_unused_buf); } EXPORT_SYMBOL_GPL(virtqueue_detach_unused_buf); =20 static inline bool more_used(const struct vring_virtqueue *vq) { - return vq->packed_ring ? more_used_packed(vq) : more_used_split(vq); + return VIRTQUEUE_CALL(vq, more_used); } =20 /** @@ -2782,7 +2874,8 @@ int virtqueue_resize(struct virtqueue *_vq, u32 num, if (!num) return -EINVAL; =20 - if ((vq->packed_ring ? vq->packed.vring.num : vq->split.vring.num) =3D=3D= num) + if ((virtqueue_is_packed(vq) ? vq->packed.vring.num : + vq->split.vring.num) =3D=3D num) return 0; =20 err =3D virtqueue_disable_and_recycle(_vq, recycle); @@ -2791,10 +2884,7 @@ int virtqueue_resize(struct virtqueue *_vq, u32 num, if (recycle_done) recycle_done(_vq); =20 - if (vq->packed_ring) - err =3D virtqueue_resize_packed(vq, num); - else - err =3D virtqueue_resize_split(vq, num); + err =3D VIRTQUEUE_CALL(vq, resize, num); =20 err_reset =3D virtqueue_enable_after_reset(_vq); if (err_reset) @@ -2832,10 +2922,7 @@ int virtqueue_reset(struct virtqueue *_vq, if (recycle_done) recycle_done(_vq); =20 - if (vq->packed_ring) - virtqueue_reset_packed(vq); - else - virtqueue_reset_split(vq); + VOID_VIRTQUEUE_CALL(vq, reset); =20 return virtqueue_enable_after_reset(_vq); } @@ -2878,7 +2965,7 @@ static void vring_free(struct virtqueue *_vq) struct vring_virtqueue *vq =3D to_vvq(_vq); =20 if (vq->we_own_ring) { - if (vq->packed_ring) { + if (virtqueue_is_packed(vq)) { vring_free_queue(vq->vq.vdev, vq->packed.ring_size_in_bytes, vq->packed.vring.desc, @@ -2907,7 +2994,7 @@ static void vring_free(struct virtqueue *_vq) vq->map); } } - if (!vq->packed_ring) { + if (!virtqueue_is_packed(vq)) { kfree(vq->split.desc_state); kfree(vq->split.desc_extra); } @@ -2932,7 +3019,7 @@ u32 vring_notification_data(struct virtqueue *_vq) struct vring_virtqueue *vq =3D to_vvq(_vq); u16 next; =20 - if (vq->packed_ring) + if (virtqueue_is_packed(vq)) next =3D (vq->packed.next_avail_idx & ~(-(1 << VRING_PACKED_EVENT_F_WRAP_CTR))) | vq->packed.avail_wrap_counter << @@ -2985,7 +3072,8 @@ unsigned int virtqueue_get_vring_size(const struct vi= rtqueue *_vq) =20 const struct vring_virtqueue *vq =3D to_vvq(_vq); =20 - return vq->packed_ring ? vq->packed.vring.num : vq->split.vring.num; + return virtqueue_is_packed(vq) ? vq->packed.vring.num : + vq->split.vring.num; } EXPORT_SYMBOL_GPL(virtqueue_get_vring_size); =20 @@ -3068,7 +3156,7 @@ dma_addr_t virtqueue_get_desc_addr(const struct virtq= ueue *_vq) =20 BUG_ON(!vq->we_own_ring); =20 - if (vq->packed_ring) + if (virtqueue_is_packed(vq)) return vq->packed.ring_dma_addr; =20 return vq->split.queue_dma_addr; @@ -3081,7 +3169,7 @@ dma_addr_t virtqueue_get_avail_addr(const struct virt= queue *_vq) =20 BUG_ON(!vq->we_own_ring); =20 - if (vq->packed_ring) + if (virtqueue_is_packed(vq)) return vq->packed.driver_event_dma_addr; =20 return vq->split.queue_dma_addr + @@ -3095,7 +3183,7 @@ dma_addr_t virtqueue_get_used_addr(const struct virtq= ueue *_vq) =20 BUG_ON(!vq->we_own_ring); =20 - if (vq->packed_ring) + if (virtqueue_is_packed(vq)) return vq->packed.device_event_dma_addr; =20 return vq->split.queue_dma_addr + --=20 2.31.1 From nobody Sat Feb 7 15:21:59 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C4E862E7BBC for ; Mon, 20 Oct 2025 07:11:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944279; cv=none; b=lsHxNnKx39bFJW3tq3cWX80u0l+gtB6uVCr/9cWQmkktueO03TNERVi+093BIj/jngkQwvb/WJ0iqvn8D7+Nae048JUwAUG6TMtDtvnM+ebGxj3JIBp2XANQjkGcoLqRW9XtXZab2AUMyu987GDtFkxZzzWmUgeBxBLtASRElJs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944279; c=relaxed/simple; bh=dNNCy30nvPJf7Pj6/+q+qtxhShGub8YeDfIF1mCuNKo=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=M3j6B3S1DM6sePh3GR1fy5dfNgYe+BBs0JhTQBEWqV1gSEzOZnZqECrjtrKJEYyMvaaQ0tZes+4Dau/ogq+ICLJHGo8MzXrx9NpaV4QXj0yQbEKtz79QgYFAKsgsTA1C4C0XqzQzsSfP9BpDmhUnkemLk55I7wS3scFMx17lW28= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=AG6R83Kl; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="AG6R83Kl" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1760944275; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CzFpi+MbdYGHcrbODt1UdYARRJA55E5Zp2RLvWcxMyk=; b=AG6R83KlRtWVTGWtC938dF8qQ7ycYJ09QCXoCGbXXPZUi4zOEaeKnt+JWbbgTbbRYbrPMc atgmjbQTOlxDV7ClofUJXEs77Vnoujb3ocpO8kfD/nA6wOLo4Yhe+4TnWjxEJtsYvLqc8g /01ceiuHlJAXZNQS5pokWqipt904ICQ= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-19-6dUQ2vrJP82aoe3wfaOEPw-1; Mon, 20 Oct 2025 03:11:06 -0400 X-MC-Unique: 6dUQ2vrJP82aoe3wfaOEPw-1 X-Mimecast-MFC-AGG-ID: 6dUQ2vrJP82aoe3wfaOEPw_1760944265 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 4F7271956048; Mon, 20 Oct 2025 07:11:05 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.113.29]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 58A7D300019F; Mon, 20 Oct 2025 07:11:02 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V8 14/19] virtio_ring: determine descriptor flags at one time Date: Mon, 20 Oct 2025 15:09:58 +0800 Message-ID: <20251020071003.28834-15-jasowang@redhat.com> In-Reply-To: <20251020071003.28834-1-jasowang@redhat.com> References: <20251020071003.28834-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Let's determine the last descriptor by counting the number of sg. This would be consistent with packed virtqueue implementation and ease the future in-order implementation. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 19 ++++++------------- 1 file changed, 6 insertions(+), 13 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 37b16ef906a4..20bc48b1241e 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -575,7 +575,7 @@ static inline int virtqueue_add_split(struct vring_virt= queue *vq, struct vring_desc_extra *extra; struct scatterlist *sg; struct vring_desc *desc; - unsigned int i, n, avail, descs_used, prev, err_idx; + unsigned int i, n, avail, descs_used, err_idx, c =3D 0; int head; bool indirect; =20 @@ -639,12 +639,11 @@ static inline int virtqueue_add_split(struct vring_vi= rtqueue *vq, if (vring_map_one_sg(vq, sg, DMA_TO_DEVICE, &addr, &len, premapped)) goto unmap_release; =20 - prev =3D i; /* Note that we trust indirect descriptor * table since it use stream DMA mapping. */ i =3D virtqueue_add_desc_split(vq, desc, extra, i, addr, len, - VRING_DESC_F_NEXT, + ++c =3D=3D total_sg ? 0 : VRING_DESC_F_NEXT, premapped); } } @@ -656,21 +655,15 @@ static inline int virtqueue_add_split(struct vring_vi= rtqueue *vq, if (vring_map_one_sg(vq, sg, DMA_FROM_DEVICE, &addr, &len, premapped)) goto unmap_release; =20 - prev =3D i; /* Note that we trust indirect descriptor * table since it use stream DMA mapping. */ - i =3D virtqueue_add_desc_split(vq, desc, extra, i, addr, len, - VRING_DESC_F_NEXT | - VRING_DESC_F_WRITE, - premapped); + i =3D virtqueue_add_desc_split(vq, desc, extra, + i, addr, len, + (++c =3D=3D total_sg ? 0 : VRING_DESC_F_NEXT) | + VRING_DESC_F_WRITE, premapped); } } - /* Last one doesn't continue. */ - desc[prev].flags &=3D cpu_to_virtio16(vq->vq.vdev, ~VRING_DESC_F_NEXT); - if (!indirect && vring_need_unmap_buffer(vq, &extra[prev])) - vq->split.desc_extra[prev & (vq->split.vring.num - 1)].flags &=3D - ~VRING_DESC_F_NEXT; =20 if (indirect) { /* Now that the indirect table is filled in, map it. */ --=20 2.31.1 From nobody Sat Feb 7 15:21:59 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A9FB72DF152 for ; Mon, 20 Oct 2025 07:11:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944275; cv=none; b=QmadhGaHkE87Qlp28zEeKGqG0YiwAruT05imnWYowMTh4D+icR+rwH3xvx03CNac/TMujkWcqkXO2ubNDOxwa9bpRw5BKUZd8YZi7kbsfBrwPCdeUB0fHuBIDC3BBb5tC32uLjpiRs2S8zBicrKA7cLvw6FwDnaK/NtpjtUvWSw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944275; c=relaxed/simple; bh=WxXX45So0xuh/RbhpnlScTCEz3yBI7cpSsDxFPZfQb0=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=fPfqLVPG5xu+BsQpT+bHmWV2KAe/8DlINdB7yrbfy74IwOXfzd1xONlIkVAWhbDTt9VhRlaTw6K7fcQTtvTYD964EFNezn9JL2uJD7YM5cyq/fDf50znF/bu9T5iheQx8EpgXSb8OBPd5zZDiIH6MkmMtMvMri9xld3jCSspwuo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=XvebB9uE; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="XvebB9uE" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1760944272; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AUZbPz5bWcqYgiCLQ5ZIhZpSAA1/ryTVCTrj9GfjMUs=; b=XvebB9uEDjh5Q+9CyUkgPV2NchknwDkdbkhW79p2pD+ApdGK7s7VY2oQBh2cxpubM6L0E9 YN6pPL36hEtGrS/ougcV2e6QDlZQ1xv5WeeGd9RMk4PWsk1DIQj3bST/K9V45qGLC0rBEL MjumBH7XTvyrct+3+R393QkBfHAWBMg= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-94-_5W78_GnNVSEFQyBmkqqzg-1; Mon, 20 Oct 2025 03:11:10 -0400 X-MC-Unique: _5W78_GnNVSEFQyBmkqqzg-1 X-Mimecast-MFC-AGG-ID: _5W78_GnNVSEFQyBmkqqzg_1760944269 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 313141956054; Mon, 20 Oct 2025 07:11:09 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.113.29]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 3B9C130001BC; Mon, 20 Oct 2025 07:11:05 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V8 15/19] virtio_ring: factor out core logic of buffer detaching Date: Mon, 20 Oct 2025 15:09:59 +0800 Message-ID: <20251020071003.28834-16-jasowang@redhat.com> In-Reply-To: <20251020071003.28834-1-jasowang@redhat.com> References: <20251020071003.28834-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Factor out core logic of buffer detaching and leave the id population to the caller so in order can just call the core logic. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 20bc48b1241e..16e432fda93d 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -1662,8 +1662,8 @@ static bool virtqueue_kick_prepare_packed(struct vrin= g_virtqueue *vq) return needs_kick; } =20 -static void detach_buf_packed(struct vring_virtqueue *vq, - unsigned int id, void **ctx) +static void detach_buf_packed_in_order(struct vring_virtqueue *vq, + unsigned int id, void **ctx) { struct vring_desc_state_packed *state =3D NULL; struct vring_packed_desc *desc; @@ -1674,8 +1674,6 @@ static void detach_buf_packed(struct vring_virtqueue = *vq, /* Clear data ptr. */ state->data =3D NULL; =20 - vq->packed.desc_extra[state->last].next =3D vq->free_head; - vq->free_head =3D id; vq->vq.num_free +=3D state->num; =20 if (unlikely(vq->use_map_api)) { @@ -1712,6 +1710,17 @@ static void detach_buf_packed(struct vring_virtqueue= *vq, } } =20 +static void detach_buf_packed(struct vring_virtqueue *vq, + unsigned int id, void **ctx) +{ + struct vring_desc_state_packed *state =3D &vq->packed.desc_state[id]; + + vq->packed.desc_extra[state->last].next =3D vq->free_head; + vq->free_head =3D id; + + detach_buf_packed_in_order(vq, id, ctx); +} + static inline bool is_used_desc_packed(const struct vring_virtqueue *vq, u16 idx, bool used_wrap_counter) { --=20 2.31.1 From nobody Sat Feb 7 15:21:59 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C4F392E7BD9 for ; Mon, 20 Oct 2025 07:11:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944279; cv=none; b=buIkeJzBzvuAK+EtzXEMskb8bDGyProghFvQFxXiC0PbvsoBX8PXavh9csuoR4qnt9aKQV6UypoZASGrb6f/Tg0y5Ibkt/nsfb56MvKSFNerxD+HLXR/qb+oniOb5Jk8Q8odwIzDRz0uxawsd0NbxhJCGDbyE0WeA5few6MHfUI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944279; c=relaxed/simple; bh=zMSguGuKGxoHhvr1RW9fxPgjAuaWTQZtX5JJHzv8RbQ=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=mONpHrnghWPH6RAVbAk8d/gAg1FnBmaDan9VNONCGD+Lw1dOUFNHX/NCSltO54SVm+7H/8BbwOg6mPBiM4q7lshUqGGuzk/Llz4x+q/Bv8hErgeqPYjTLIsBc2QvQiYVUZY+mtY4bRhj/UXI0ZpzEZP2IoLjaC/VMnjqMYwolno= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=WNlipPfx; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="WNlipPfx" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1760944276; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1fwTZo2nfDnQlgWZWhEhqf2fn4n6CrJWorLQA7f4krM=; b=WNlipPfx/QZH9MdfWZKMVA8NMrlFiDQvl9PthlFYjUqm90cZpFFhDYgprJRpmwgfmf7QOI c5LD406qjlJ42H0LalqckfMh301pf2L+qhkMNHtTQkrbvwjDiictoc1mfuKCtDxbvYXsG/ smkihzmnEC8i8ZWiQnQ3w4DRpQ28yg4= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-307-QpUd7TexPVebd4uq0DwCZA-1; Mon, 20 Oct 2025 03:11:13 -0400 X-MC-Unique: QpUd7TexPVebd4uq0DwCZA-1 X-Mimecast-MFC-AGG-ID: QpUd7TexPVebd4uq0DwCZA_1760944272 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id D08D019560AF; Mon, 20 Oct 2025 07:11:12 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.113.29]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id D914A30001A2; Mon, 20 Oct 2025 07:11:09 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V8 16/19] virtio_ring: factor out core logic for updating last_used_idx Date: Mon, 20 Oct 2025 15:10:00 +0800 Message-ID: <20251020071003.28834-17-jasowang@redhat.com> In-Reply-To: <20251020071003.28834-1-jasowang@redhat.com> References: <20251020071003.28834-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Factor out the core logic for updating last_used_idx to be reused by the packed in order implementation. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 43 +++++++++++++++++++++--------------- 1 file changed, 25 insertions(+), 18 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 16e432fda93d..c59e27e2ad68 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -1751,6 +1751,30 @@ static bool more_used_packed(const struct vring_virt= queue *vq) return virtqueue_poll_packed(vq, READ_ONCE(vq->last_used_idx)); } =20 +static void update_last_used_idx_packed(struct vring_virtqueue *vq, + u16 id, u16 last_used, + u16 used_wrap_counter) +{ + last_used +=3D vq->packed.desc_state[id].num; + if (unlikely(last_used >=3D vq->packed.vring.num)) { + last_used -=3D vq->packed.vring.num; + used_wrap_counter ^=3D 1; + } + + last_used =3D (last_used | (used_wrap_counter << VRING_PACKED_EVENT_F_WRA= P_CTR)); + WRITE_ONCE(vq->last_used_idx, last_used); + + /* + * If we expect an interrupt for the next entry, tell host + * by writing event index and flush out the write before + * the read in the next get_buf call. + */ + if (vq->packed.event_flags_shadow =3D=3D VRING_PACKED_EVENT_FLAG_DESC) + virtio_store_mb(vq->weak_barriers, + &vq->packed.vring.driver->off_wrap, + cpu_to_le16(vq->last_used_idx)); +} + static void *virtqueue_get_buf_ctx_packed(struct vring_virtqueue *vq, unsigned int *len, void **ctx) @@ -1794,24 +1818,7 @@ static void *virtqueue_get_buf_ctx_packed(struct vri= ng_virtqueue *vq, ret =3D vq->packed.desc_state[id].data; detach_buf_packed(vq, id, ctx); =20 - last_used +=3D vq->packed.desc_state[id].num; - if (unlikely(last_used >=3D vq->packed.vring.num)) { - last_used -=3D vq->packed.vring.num; - used_wrap_counter ^=3D 1; - } - - last_used =3D (last_used | (used_wrap_counter << VRING_PACKED_EVENT_F_WRA= P_CTR)); - WRITE_ONCE(vq->last_used_idx, last_used); - - /* - * If we expect an interrupt for the next entry, tell host - * by writing event index and flush out the write before - * the read in the next get_buf call. - */ - if (vq->packed.event_flags_shadow =3D=3D VRING_PACKED_EVENT_FLAG_DESC) - virtio_store_mb(vq->weak_barriers, - &vq->packed.vring.driver->off_wrap, - cpu_to_le16(vq->last_used_idx)); + update_last_used_idx_packed(vq, id, last_used, used_wrap_counter); =20 LAST_ADD_TIME_INVALID(vq); =20 --=20 2.31.1 From nobody Sat Feb 7 15:21:59 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7317E2E92C7 for ; Mon, 20 Oct 2025 07:11:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944284; cv=none; b=UcPINOxbxA4PobKaEZC+wRBADTHpRkuOQPG2P5CLdtcekC7jlc3lE9/7LHVUyyLyCzusP8R6gAWVEigzzsJK3TFnksdJbyy1XPseteVzsFPoAUCt0Ak00Gi+zzYtSB4F1GIzvYFUOw9N5cecniFLyDxuUMewpR5ggS+iEg8OYyM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944284; c=relaxed/simple; bh=If4snOkMfLQbL1VYmq7vhi5uBH6a4l2Xi0s8Njt9BtI=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=eVwXMPtorA+e48d5mTSv25b6UWGumhaQFBmiQfKs6cFpVPm4De1jzZlGMQZ6vtAhOSzxJMQDA1o3RL0QJFsP98wPlKcHwtpsYG/dLK7dzk+bM4M84hrlVlD5oWXHZiu6vG0prT9o/GSNE4II6ZIS4mrTCBhxRNbgUpraLDE5iBo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=du+m84YV; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="du+m84YV" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1760944281; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dkwIkfBggUdJGM3dCT6Zmq2fReJpmpq7neySAXJ1oi4=; b=du+m84YVYMnLFENtbrzAP5FSz83HNKLKG1IxHcHeu6UfbfHsISJLnw4bSPoItgMd74QIM5 2lnaQwio5lvNY62p9w+ig+TBqdwJhn9gzA5E7QaZXPShTaElrcj0XqUV760T2UrVg6LObL 38qbe6TUxdEtaTzmjImrJv3R3ewFdRc= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-423-OxWlUojMO6m9CBb6Bb7gZA-1; Mon, 20 Oct 2025 03:11:17 -0400 X-MC-Unique: OxWlUojMO6m9CBb6Bb7gZA-1 X-Mimecast-MFC-AGG-ID: OxWlUojMO6m9CBb6Bb7gZA_1760944276 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 73D5418002F9; Mon, 20 Oct 2025 07:11:16 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.113.29]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 87936300019F; Mon, 20 Oct 2025 07:11:13 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V8 17/19] virtio_ring: factor out split indirect detaching logic Date: Mon, 20 Oct 2025 15:10:01 +0800 Message-ID: <20251020071003.28834-18-jasowang@redhat.com> In-Reply-To: <20251020071003.28834-1-jasowang@redhat.com> References: <20251020071003.28834-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Factor out the split indirect descriptor detaching logic in order to allow it to be reused by the in order support. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 63 ++++++++++++++++++++---------------- 1 file changed, 35 insertions(+), 28 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index c59e27e2ad68..0f07a6637acb 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -771,11 +771,42 @@ static bool virtqueue_kick_prepare_split(struct vring= _virtqueue *vq) return needs_kick; } =20 +static void detach_indirect_split(struct vring_virtqueue *vq, + unsigned int head) +{ + struct vring_desc_extra *extra =3D vq->split.desc_extra; + struct vring_desc *indir_desc =3D + vq->split.desc_state[head].indir_desc; + unsigned int j; + u32 len, num; + + /* Free the indirect table, if any, now that it's unmapped. */ + if (!indir_desc) + return; + len =3D vq->split.desc_extra[head].len; + + BUG_ON(!(vq->split.desc_extra[head].flags & + VRING_DESC_F_INDIRECT)); + BUG_ON(len =3D=3D 0 || len % sizeof(struct vring_desc)); + + num =3D len / sizeof(struct vring_desc); + + extra =3D (struct vring_desc_extra *)&indir_desc[num]; + + if (vq->use_map_api) { + for (j =3D 0; j < num; j++) + vring_unmap_one_split(vq, &extra[j]); + } + + kfree(indir_desc); + vq->split.desc_state[head].indir_desc =3D NULL; +} + static void detach_buf_split(struct vring_virtqueue *vq, unsigned int head, void **ctx) { struct vring_desc_extra *extra; - unsigned int i, j; + unsigned int i; __virtio16 nextflag =3D cpu_to_virtio16(vq->vq.vdev, VRING_DESC_F_NEXT); =20 /* Clear data ptr. */ @@ -799,34 +830,10 @@ static void detach_buf_split(struct vring_virtqueue *= vq, unsigned int head, /* Plus final descriptor */ vq->vq.num_free++; =20 - if (vq->indirect) { - struct vring_desc *indir_desc =3D - vq->split.desc_state[head].indir_desc; - u32 len, num; - - /* Free the indirect table, if any, now that it's unmapped. */ - if (!indir_desc) - return; - len =3D vq->split.desc_extra[head].len; - - BUG_ON(!(vq->split.desc_extra[head].flags & - VRING_DESC_F_INDIRECT)); - BUG_ON(len =3D=3D 0 || len % sizeof(struct vring_desc)); - - num =3D len / sizeof(struct vring_desc); - - extra =3D (struct vring_desc_extra *)&indir_desc[num]; - - if (vq->use_map_api) { - for (j =3D 0; j < num; j++) - vring_unmap_one_split(vq, &extra[j]); - } - - kfree(indir_desc); - vq->split.desc_state[head].indir_desc =3D NULL; - } else if (ctx) { + if (vq->indirect) + detach_indirect_split(vq, head); + else if (ctx) *ctx =3D vq->split.desc_state[head].indir_desc; - } } =20 static bool virtqueue_poll_split(const struct vring_virtqueue *vq, --=20 2.31.1 From nobody Sat Feb 7 15:21:59 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A0E0A2DFA46 for ; Mon, 20 Oct 2025 07:11:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944287; cv=none; b=HCj4I6x/Aiqag8PqdNe4r01jJpjdQZJ6ic1QeEKPAQ6/1KCxUcghJW9uom/IvZlwIks8OBzKi0Suyw3zliI2KZJ58kI57I9yXLhIccoCTUfEeUxejZnNZ9eM+1NYDTucyqPhk/zj9qI7efEt8WIvuRIWxGCqPOVr00Z/AO20Ulc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944287; c=relaxed/simple; bh=rHrGNT24xmrdhC5F+YR1TnjDpsX+U6XDSXhyVRYmpsU=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=glBFxqWlOYy9iA7Im2lXWQI8ZY3JscBwygJTKVTVMfL+emqG96Ke5aHZ3nV5ERL+geGc2q5Yk/osAaaRsY+ok1aiubVZrvtC1aZq/+cOY7OZHMIuNSdRrXzga3vxzDyBbtYvUz93hbmNkuxw0/U6zJvbRtEqgaykNcHTVHsiwWM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Dqv1mVB+; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Dqv1mVB+" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1760944284; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gDhYYb5/vqjcZ2X9K8PW960PeOO9CL79kbWB0k4US8U=; b=Dqv1mVB+nAKd7eUKUJoiCIvlGK0/LkQ6cHTniUDDlYZ+67wC/dW7SUgBDI4n02p5kO/e+y K8Bv1lgI/d6je+YLQwUHIq8SDnSpaL7wYKLakbQ4zttzkdwR7Siz/iE0g2YatriPVVP3/c oZymGt7GVh4VsP+WzvBcNBq+b3+FXKo= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-422-OS_9lLAhPsCBZ0L2HPOwrA-1; Mon, 20 Oct 2025 03:11:21 -0400 X-MC-Unique: OS_9lLAhPsCBZ0L2HPOwrA-1 X-Mimecast-MFC-AGG-ID: OS_9lLAhPsCBZ0L2HPOwrA_1760944280 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 5AD9E1954213; Mon, 20 Oct 2025 07:11:20 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.113.29]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 4277B300019F; Mon, 20 Oct 2025 07:11:16 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V8 18/19] virtio_ring: factor out split detaching logic Date: Mon, 20 Oct 2025 15:10:02 +0800 Message-ID: <20251020071003.28834-19-jasowang@redhat.com> In-Reply-To: <20251020071003.28834-1-jasowang@redhat.com> References: <20251020071003.28834-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 This patch factors out the split core detaching logic that could be reused by in order feature into a dedicated function. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 0f07a6637acb..96d7f165ec88 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -802,8 +802,9 @@ static void detach_indirect_split(struct vring_virtqueu= e *vq, vq->split.desc_state[head].indir_desc =3D NULL; } =20 -static void detach_buf_split(struct vring_virtqueue *vq, unsigned int head, - void **ctx) +static unsigned detach_buf_split_in_order(struct vring_virtqueue *vq, + unsigned int head, + void **ctx) { struct vring_desc_extra *extra; unsigned int i; @@ -824,8 +825,6 @@ static void detach_buf_split(struct vring_virtqueue *vq= , unsigned int head, } =20 vring_unmap_one_split(vq, &extra[i]); - vq->split.desc_extra[i].next =3D vq->free_head; - vq->free_head =3D head; =20 /* Plus final descriptor */ vq->vq.num_free++; @@ -834,6 +833,17 @@ static void detach_buf_split(struct vring_virtqueue *v= q, unsigned int head, detach_indirect_split(vq, head); else if (ctx) *ctx =3D vq->split.desc_state[head].indir_desc; + + return i; +} + +static void detach_buf_split(struct vring_virtqueue *vq, unsigned int head, + void **ctx) +{ + unsigned int i =3D detach_buf_split_in_order(vq, head, ctx); + + vq->split.desc_extra[i].next =3D vq->free_head; + vq->free_head =3D head; } =20 static bool virtqueue_poll_split(const struct vring_virtqueue *vq, --=20 2.31.1 From nobody Sat Feb 7 15:21:59 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AF1582E9756 for ; Mon, 20 Oct 2025 07:11:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944292; cv=none; b=Itg5kwoJZvFjBrSkkOBuXlILLFufoBH7e1vMiycMYcIF5yXcHMFQPlmei1chTRSsfagiuv05B0W/aXPmidIUHRnqqu3DoJgHn7ZqLn60B3Q6sytE5brEsKkisB8XFalP293cpd/2r2l4K0MAePIZDGa+nxA4DQTtOxyXOQEYvog= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1760944292; c=relaxed/simple; bh=ZHpVfA7VH4k0ukmUjuC57cblvYQ0Q2C4Ww5momBNFFI=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kJ3mqOhhEt5Hs9+F4Cy0q+rvMNzSL0Iu+tzOv381PwOq1zMno2vl4xHcQ2hfIzZ0K76emlBSxr1zm3x8BJKUKjmK2x+4aTtJs8cEPXxc8sQVtLv1zqG7xfXU/kNHXetiQqAGokO48+Zy15W1bELDyZ4wf1Lx1ijUEFcV+TNyAS0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=O1gdT396; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="O1gdT396" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1760944288; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=y0P58SE7iwk0BlqDpIZDuPjyCcIOrAKQwNpO9ZjhvfA=; b=O1gdT396DE45pViODiGBZr43m2MqJJuCFsHlaKPeibv6KoZY4eQ3d6sniSHStJIkS4AN8n k1Zr1mBGSOco0mDJQS1xuQ1SJD9VXVK5bTibLSeHwLIR/Ie0c5mDNKHj6jvD22k4ixHZQ2 akd88Wql2/6lMWHNFHqwRT3Bp3AJJPE= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-220-L7eQXWU6M-6LyITHYdIZFg-1; Mon, 20 Oct 2025 03:11:24 -0400 X-MC-Unique: L7eQXWU6M-6LyITHYdIZFg-1 X-Mimecast-MFC-AGG-ID: L7eQXWU6M-6LyITHYdIZFg_1760944284 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 0904D1800657; Mon, 20 Oct 2025 07:11:24 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.113.29]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 0F5B1300019F; Mon, 20 Oct 2025 07:11:20 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V8 19/19] virtio_ring: add in order support Date: Mon, 20 Oct 2025 15:10:03 +0800 Message-ID: <20251020071003.28834-20-jasowang@redhat.com> In-Reply-To: <20251020071003.28834-1-jasowang@redhat.com> References: <20251020071003.28834-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Content-Type: text/plain; charset="utf-8" This patch implements in order support for both split virtqueue and packed virtqueue. Performance could be gained for the device where the memory access could be expensive (e.g vhost-net or a real PCI device): Benchmark with KVM guest: Vhost-net on the host: (pktgen + XDP_DROP): in_order=3Doff | in_order=3Don | +% TX: 5.20Mpps | 6.20Mpps | +19% RX: 3.47Mpps | 3.61Mpps | + 4% Vhost-user(testpmd) on the host: (pktgen/XDP_DROP): For split virtqueue: in_order=3Doff | in_order=3Don | +% TX: 5.60Mpps | 5.60Mpps | +0.0% RX: 9.16Mpps | 9.61Mpps | +4.9% For packed virtqueue: in_order=3Doff | in_order=3Don | +% TX: 5.60Mpps | 5.70Mpps | +1.7% RX: 10.6Mpps | 10.8Mpps | +1.8% Benchmark also shows no performance impact for in_order=3Doff for queue size with 256 and 1024. Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 440 +++++++++++++++++++++++++++++++++-- 1 file changed, 416 insertions(+), 24 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 96d7f165ec88..411bfa31707d 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -70,6 +70,8 @@ enum vq_layout { SPLIT =3D 0, PACKED, + SPLIT_IN_ORDER, + PACKED_IN_ORDER, VQ_TYPE_MAX, }; =20 @@ -80,6 +82,7 @@ struct vring_desc_state_split { * allocated together. So we won't stress more to the memory allocator. */ struct vring_desc *indir_desc; + u32 total_len; /* Buffer Length */ }; =20 struct vring_desc_state_packed { @@ -91,6 +94,7 @@ struct vring_desc_state_packed { struct vring_packed_desc *indir_desc; u16 num; /* Descriptor list length. */ u16 last; /* The last desc state in a list. */ + u32 total_len; /* Buffer Length */ }; =20 struct vring_desc_extra { @@ -168,7 +172,7 @@ struct vring_virtqueue_packed { struct vring_virtqueue; =20 struct virtqueue_ops { - int (*add)(struct vring_virtqueue *_vq, struct scatterlist *sgs[], + int (*add)(struct vring_virtqueue *vq, struct scatterlist *sgs[], unsigned int total_sg, unsigned int out_sgs, unsigned int in_sgs, void *data, void *ctx, bool premapped, gfp_t gfp); @@ -205,8 +209,23 @@ struct vring_virtqueue { =20 enum vq_layout layout; =20 - /* Head of free buffer list. */ + /* + * Without IN_ORDER it's the head of free buffer list. With + * IN_ORDER and SPLIT, it's the next available buffer + * index. With IN_ORDER and PACKED, it's unused. + */ unsigned int free_head; + + /* + * With IN_ORDER, devices write a single used ring entry with + * the id corresponding to the head entry of the descriptor chain + * describing the last buffer in the batch + */ + struct used_entry { + u32 id; + u32 len; + } batch_last; + /* Number we've added since last sync. */ unsigned int num_added; =20 @@ -259,7 +278,12 @@ static void vring_free(struct virtqueue *_vq); =20 static inline bool virtqueue_is_packed(const struct vring_virtqueue *vq) { - return vq->layout =3D=3D PACKED; + return vq->layout =3D=3D PACKED || vq->layout =3D=3D PACKED_IN_ORDER; +} + +static inline bool virtqueue_is_in_order(const struct vring_virtqueue *vq) +{ + return vq->layout =3D=3D SPLIT_IN_ORDER || vq->layout =3D=3D PACKED_IN_OR= DER; } =20 static bool virtqueue_use_indirect(const struct vring_virtqueue *vq, @@ -576,6 +600,8 @@ static inline int virtqueue_add_split(struct vring_virt= queue *vq, struct scatterlist *sg; struct vring_desc *desc; unsigned int i, n, avail, descs_used, err_idx, c =3D 0; + /* Total length for in-order */ + unsigned int total_len =3D 0; int head; bool indirect; =20 @@ -645,6 +671,7 @@ static inline int virtqueue_add_split(struct vring_virt= queue *vq, i =3D virtqueue_add_desc_split(vq, desc, extra, i, addr, len, ++c =3D=3D total_sg ? 0 : VRING_DESC_F_NEXT, premapped); + total_len +=3D len; } } for (; n < (out_sgs + in_sgs); n++) { @@ -662,6 +689,7 @@ static inline int virtqueue_add_split(struct vring_virt= queue *vq, i, addr, len, (++c =3D=3D total_sg ? 0 : VRING_DESC_F_NEXT) | VRING_DESC_F_WRITE, premapped); + total_len +=3D len; } } =20 @@ -684,7 +712,12 @@ static inline int virtqueue_add_split(struct vring_vir= tqueue *vq, vq->vq.num_free -=3D descs_used; =20 /* Update free pointer */ - if (indirect) + if (virtqueue_is_in_order(vq)) { + vq->free_head +=3D descs_used; + if (vq->free_head >=3D vq->split.vring.num) + vq->free_head -=3D vq->split.vring.num; + vq->split.desc_state[head].total_len =3D total_len;; + } else if (indirect) vq->free_head =3D vq->split.desc_extra[head].next; else vq->free_head =3D i; @@ -858,6 +891,14 @@ static bool more_used_split(const struct vring_virtque= ue *vq) return virtqueue_poll_split(vq, vq->last_used_idx); } =20 +static bool more_used_split_in_order(const struct vring_virtqueue *vq) +{ + if (vq->batch_last.id !=3D vq->split.vring.num) + return true; + + return virtqueue_poll_split(vq, vq->last_used_idx); +} + static void *virtqueue_get_buf_ctx_split(struct vring_virtqueue *vq, unsigned int *len, void **ctx) @@ -915,6 +956,73 @@ static void *virtqueue_get_buf_ctx_split(struct vring_= virtqueue *vq, return ret; } =20 +static void *virtqueue_get_buf_ctx_split_in_order(struct vring_virtqueue *= vq, + unsigned int *len, + void **ctx) +{ + void *ret; + unsigned int num =3D vq->split.vring.num; + u16 last_used; + + START_USE(vq); + + if (unlikely(vq->broken)) { + END_USE(vq); + return NULL; + } + + last_used =3D (vq->last_used_idx & (vq->split.vring.num - 1)); + + if (vq->batch_last.id =3D=3D num) { + if (!more_used_split(vq)) { + pr_debug("No more buffers in queue\n"); + END_USE(vq); + return NULL; + } + + /* Only get used array entries after they have been + * exposed by host. */ + virtio_rmb(vq->weak_barriers); + vq->batch_last.id =3D virtio32_to_cpu(vq->vq.vdev, + vq->split.vring.used->ring[last_used].id); + vq->batch_last.len =3D virtio32_to_cpu(vq->vq.vdev, + vq->split.vring.used->ring[last_used].len); + } + + if (vq->batch_last.id =3D=3D last_used) { + vq->batch_last.id =3D num; + *len =3D vq->batch_last.len; + } else + *len =3D vq->split.desc_state[last_used].total_len; + + if (unlikely(last_used >=3D num)) { + BAD_RING(vq, "id %u out of range\n", last_used); + return NULL; + } + if (unlikely(!vq->split.desc_state[last_used].data)) { + BAD_RING(vq, "id %u is not a head!\n", last_used); + return NULL; + } + + /* detach_buf_split clears data, so grab it now. */ + ret =3D vq->split.desc_state[last_used].data; + detach_buf_split_in_order(vq, last_used, ctx); + + vq->last_used_idx++; + /* If we expect an interrupt for the next entry, tell host + * by writing event index and flush out the write before + * the read in the next get_buf call. */ + if (!(vq->split.avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT)) + virtio_store_mb(vq->weak_barriers, + &vring_used_event(&vq->split.vring), + cpu_to_virtio16(vq->vq.vdev, vq->last_used_idx)); + + LAST_ADD_TIME_INVALID(vq); + + END_USE(vq); + return ret; +} + static void virtqueue_disable_cb_split(struct vring_virtqueue *vq) { if (!(vq->split.avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT)) { @@ -1008,7 +1116,10 @@ static void *virtqueue_detach_unused_buf_split(struc= t vring_virtqueue *vq) continue; /* detach_buf_split clears data, so grab it now. */ buf =3D vq->split.desc_state[i].data; - detach_buf_split(vq, i, NULL); + if (virtqueue_is_in_order(vq)) + detach_buf_split_in_order(vq, i, NULL); + else + detach_buf_split(vq, i, NULL); vq->split.avail_idx_shadow--; vq->split.vring.avail->idx =3D cpu_to_virtio16(vq->vq.vdev, vq->split.avail_idx_shadow); @@ -1071,6 +1182,7 @@ static void virtqueue_vring_attach_split(struct vring= _virtqueue *vq, =20 /* Put everything in free lists. */ vq->free_head =3D 0; + vq->batch_last.id =3D vq->split.vring.num; } =20 static int vring_alloc_state_extra_split(struct vring_virtqueue_split *vri= ng_split) @@ -1182,7 +1294,6 @@ static struct virtqueue *__vring_new_virtqueue_split(= unsigned int index, if (!vq) return NULL; =20 - vq->layout =3D SPLIT; vq->vq.callback =3D callback; vq->vq.vdev =3D vdev; vq->vq.name =3D name; @@ -1202,6 +1313,8 @@ static struct virtqueue *__vring_new_virtqueue_split(= unsigned int index, vq->indirect =3D virtio_has_feature(vdev, VIRTIO_RING_F_INDIRECT_DESC) && !context; vq->event =3D virtio_has_feature(vdev, VIRTIO_RING_F_EVENT_IDX); + vq->layout =3D virtio_has_feature(vdev, VIRTIO_F_IN_ORDER) ? + SPLIT_IN_ORDER : SPLIT; =20 if (virtio_has_feature(vdev, VIRTIO_F_ORDER_PLATFORM)) vq->weak_barriers =3D false; @@ -1359,13 +1472,14 @@ static int virtqueue_add_indirect_packed(struct vri= ng_virtqueue *vq, unsigned int in_sgs, void *data, bool premapped, - gfp_t gfp) + gfp_t gfp, + u16 id) { struct vring_desc_extra *extra; struct vring_packed_desc *desc; struct scatterlist *sg; - unsigned int i, n, err_idx, len; - u16 head, id; + unsigned int i, n, err_idx, len, total_len =3D 0; + u16 head; dma_addr_t addr; =20 head =3D vq->packed.next_avail_idx; @@ -1383,8 +1497,6 @@ static int virtqueue_add_indirect_packed(struct vring= _virtqueue *vq, } =20 i =3D 0; - id =3D vq->free_head; - BUG_ON(id =3D=3D vq->packed.vring.num); =20 for (n =3D 0; n < out_sgs + in_sgs; n++) { for (sg =3D sgs[n]; sg; sg =3D sg_next(sg)) { @@ -1404,6 +1516,7 @@ static int virtqueue_add_indirect_packed(struct vring= _virtqueue *vq, extra[i].flags =3D n < out_sgs ? 0 : VRING_DESC_F_WRITE; } =20 + total_len +=3D len; i++; } } @@ -1450,13 +1563,15 @@ static int virtqueue_add_indirect_packed(struct vri= ng_virtqueue *vq, 1 << VRING_PACKED_DESC_F_USED; } vq->packed.next_avail_idx =3D n; - vq->free_head =3D vq->packed.desc_extra[id].next; + if (!virtqueue_is_in_order(vq)) + vq->free_head =3D vq->packed.desc_extra[id].next; =20 /* Store token and indirect buffer state. */ vq->packed.desc_state[id].num =3D 1; vq->packed.desc_state[id].data =3D data; vq->packed.desc_state[id].indir_desc =3D desc; vq->packed.desc_state[id].last =3D id; + vq->packed.desc_state[id].total_len =3D total_len; =20 vq->num_added +=3D 1; =20 @@ -1509,8 +1624,11 @@ static inline int virtqueue_add_packed(struct vring_= virtqueue *vq, BUG_ON(total_sg =3D=3D 0); =20 if (virtqueue_use_indirect(vq, total_sg)) { + id =3D vq->free_head; + BUG_ON(id =3D=3D vq->packed.vring.num); err =3D virtqueue_add_indirect_packed(vq, sgs, total_sg, out_sgs, - in_sgs, data, premapped, gfp); + in_sgs, data, premapped, + gfp, id); if (err !=3D -ENOMEM) { END_USE(vq); return err; @@ -1631,6 +1749,152 @@ static inline int virtqueue_add_packed(struct vring= _virtqueue *vq, return -EIO; } =20 +static inline int virtqueue_add_packed_in_order(struct vring_virtqueue *vq, + struct scatterlist *sgs[], + unsigned int total_sg, + unsigned int out_sgs, + unsigned int in_sgs, + void *data, + void *ctx, + bool premapped, + gfp_t gfp) +{ + struct vring_packed_desc *desc; + struct scatterlist *sg; + unsigned int i, n, c, err_idx, total_len =3D 0; + __le16 head_flags, flags; + u16 head, avail_used_flags; + int err; + + START_USE(vq); + + BUG_ON(data =3D=3D NULL); + BUG_ON(ctx && vq->indirect); + + if (unlikely(vq->broken)) { + END_USE(vq); + return -EIO; + } + + LAST_ADD_TIME_UPDATE(vq); + + BUG_ON(total_sg =3D=3D 0); + + if (virtqueue_use_indirect(vq, total_sg)) { + err =3D virtqueue_add_indirect_packed(vq, sgs, total_sg, out_sgs, + in_sgs, data, premapped, gfp, + vq->packed.next_avail_idx); + if (err !=3D -ENOMEM) { + END_USE(vq); + return err; + } + + /* fall back on direct */ + } + + head =3D vq->packed.next_avail_idx; + avail_used_flags =3D vq->packed.avail_used_flags; + + WARN_ON_ONCE(total_sg > vq->packed.vring.num && !vq->indirect); + + desc =3D vq->packed.vring.desc; + i =3D head; + + if (unlikely(vq->vq.num_free < total_sg)) { + pr_debug("Can't add buf len %i - avail =3D %i\n", + total_sg, vq->vq.num_free); + END_USE(vq); + return -ENOSPC; + } + + c =3D 0; + for (n =3D 0; n < out_sgs + in_sgs; n++) { + for (sg =3D sgs[n]; sg; sg =3D sg_next(sg)) { + dma_addr_t addr; + u32 len; + + if (vring_map_one_sg(vq, sg, n < out_sgs ? + DMA_TO_DEVICE : DMA_FROM_DEVICE, + &addr, &len, premapped)) + goto unmap_release; + + flags =3D cpu_to_le16(vq->packed.avail_used_flags | + (++c =3D=3D total_sg ? 0 : VRING_DESC_F_NEXT) | + (n < out_sgs ? 0 : VRING_DESC_F_WRITE)); + if (i =3D=3D head) + head_flags =3D flags; + else + desc[i].flags =3D flags; + + + desc[i].addr =3D cpu_to_le64(addr); + desc[i].len =3D cpu_to_le32(len); + desc[i].id =3D cpu_to_le16(head); + + if (unlikely(vq->use_map_api)) { + vq->packed.desc_extra[i].addr =3D premapped ? + DMA_MAPPING_ERROR: addr; + vq->packed.desc_extra[i].len =3D len; + vq->packed.desc_extra[i].flags =3D + le16_to_cpu(flags); + } + + if ((unlikely(++i >=3D vq->packed.vring.num))) { + i =3D 0; + vq->packed.avail_used_flags ^=3D + 1 << VRING_PACKED_DESC_F_AVAIL | + 1 << VRING_PACKED_DESC_F_USED; + vq->packed.avail_wrap_counter ^=3D 1; + } + + total_len +=3D len; + } + } + + /* We're using some buffers from the free list. */ + vq->vq.num_free -=3D total_sg; + + /* Update free pointer */ + vq->packed.next_avail_idx =3D i; + + /* Store token. */ + vq->packed.desc_state[head].num =3D total_sg; + vq->packed.desc_state[head].data =3D data; + vq->packed.desc_state[head].indir_desc =3D ctx; + vq->packed.desc_state[head].total_len =3D total_len; + + /* + * A driver MUST NOT make the first descriptor in the list + * available before all subsequent descriptors comprising + * the list are made available. + */ + virtio_wmb(vq->weak_barriers); + vq->packed.vring.desc[head].flags =3D head_flags; + vq->num_added +=3D total_sg; + + pr_debug("Added buffer head %i to %p\n", head, vq); + END_USE(vq); + + return 0; + +unmap_release: + err_idx =3D i; + i =3D head; + vq->packed.avail_used_flags =3D avail_used_flags; + + for (n =3D 0; n < total_sg; n++) { + if (i =3D=3D err_idx) + break; + vring_unmap_extra_packed(vq, &vq->packed.desc_extra[i]); + i++; + if (i >=3D vq->packed.vring.num) + i =3D 0; + } + + END_USE(vq); + return -EIO; +} + static bool virtqueue_kick_prepare_packed(struct vring_virtqueue *vq) { u16 new, old, off_wrap, flags, wrap_counter, event_idx; @@ -1792,10 +2056,81 @@ static void update_last_used_idx_packed(struct vrin= g_virtqueue *vq, cpu_to_le16(vq->last_used_idx)); } =20 +static bool more_used_packed_in_order(const struct vring_virtqueue *vq) +{ + if (vq->batch_last.id !=3D vq->packed.vring.num) + return true; + + return virtqueue_poll_packed(vq, READ_ONCE(vq->last_used_idx)); +} + +static void *virtqueue_get_buf_ctx_packed_in_order(struct vring_virtqueue = *vq, + unsigned int *len, + void **ctx) +{ + unsigned int num =3D vq->packed.vring.num; + u16 last_used, last_used_idx; + bool used_wrap_counter; + void *ret; + + START_USE(vq); + + if (unlikely(vq->broken)) { + END_USE(vq); + return NULL; + } + + last_used_idx =3D vq->last_used_idx; + used_wrap_counter =3D packed_used_wrap_counter(last_used_idx); + last_used =3D packed_last_used(last_used_idx); + + if (vq->batch_last.id =3D=3D num) { + if (!more_used_packed(vq)) { + pr_debug("No more buffers in queue\n"); + END_USE(vq); + return NULL; + } + /* Only get used elements after they have been exposed by host. */ + virtio_rmb(vq->weak_barriers); + vq->batch_last.id =3D + le16_to_cpu(vq->packed.vring.desc[last_used].id); + vq->batch_last.len =3D + le32_to_cpu(vq->packed.vring.desc[last_used].len); + } + + if (vq->batch_last.id =3D=3D last_used) { + vq->batch_last.id =3D num; + *len =3D vq->batch_last.len; + } else + *len =3D vq->packed.desc_state[last_used].total_len; + + if (unlikely(last_used >=3D num)) { + BAD_RING(vq, "id %u out of range\n", last_used); + return NULL; + } + if (unlikely(!vq->packed.desc_state[last_used].data)) { + BAD_RING(vq, "id %u is not a head!\n", last_used); + return NULL; + } + + /* detach_buf_packed clears data, so grab it now. */ + ret =3D vq->packed.desc_state[last_used].data; + detach_buf_packed_in_order(vq, last_used, ctx); + + update_last_used_idx_packed(vq, last_used, last_used, + used_wrap_counter); + + LAST_ADD_TIME_INVALID(vq); + + END_USE(vq); + return ret; +} + static void *virtqueue_get_buf_ctx_packed(struct vring_virtqueue *vq, unsigned int *len, void **ctx) { + unsigned int num =3D vq->packed.vring.num; u16 last_used, id, last_used_idx; bool used_wrap_counter; void *ret; @@ -1822,7 +2157,7 @@ static void *virtqueue_get_buf_ctx_packed(struct vrin= g_virtqueue *vq, id =3D le16_to_cpu(vq->packed.vring.desc[last_used].id); *len =3D le32_to_cpu(vq->packed.vring.desc[last_used].len); =20 - if (unlikely(id >=3D vq->packed.vring.num)) { + if (unlikely(id >=3D num)) { BAD_RING(vq, "id %u out of range\n", id); return NULL; } @@ -1963,7 +2298,10 @@ static void *virtqueue_detach_unused_buf_packed(stru= ct vring_virtqueue *vq) continue; /* detach_buf clears data, so grab it now. */ buf =3D vq->packed.desc_state[i].data; - detach_buf_packed(vq, i, NULL); + if (virtqueue_is_in_order(vq)) + detach_buf_packed_in_order(vq, i, NULL); + else + detach_buf_packed(vq, i, NULL); END_USE(vq); return buf; } @@ -1989,6 +2327,8 @@ static struct vring_desc_extra *vring_alloc_desc_extr= a(unsigned int num) for (i =3D 0; i < num - 1; i++) desc_extra[i].next =3D i + 1; =20 + desc_extra[num - 1].next =3D 0; + return desc_extra; } =20 @@ -2120,10 +2460,17 @@ static void virtqueue_vring_attach_packed(struct vr= ing_virtqueue *vq, { vq->packed =3D *vring_packed; =20 - /* Put everything in free lists. */ - vq->free_head =3D 0; + if (virtqueue_is_in_order(vq)) + vq->batch_last.id =3D vq->packed.vring.num; + else { + /* + * Put everything in free lists. Note that + * next_avail_idx is sufficient with IN_ORDER so + * free_head is unused. + */ + vq->free_head =3D 0 ; + } } - static void virtqueue_reset_packed(struct vring_virtqueue *vq) { memset(vq->packed.vring.device, 0, vq->packed.event_size_in_bytes); @@ -2168,13 +2515,14 @@ static struct virtqueue *__vring_new_virtqueue_pack= ed(unsigned int index, #else vq->broken =3D false; #endif - vq->layout =3D PACKED; vq->map =3D map; vq->use_map_api =3D vring_use_map_api(vdev); =20 vq->indirect =3D virtio_has_feature(vdev, VIRTIO_RING_F_INDIRECT_DESC) && !context; vq->event =3D virtio_has_feature(vdev, VIRTIO_RING_F_EVENT_IDX); + vq->layout =3D virtio_has_feature(vdev, VIRTIO_F_IN_ORDER) ? + PACKED_IN_ORDER : PACKED; =20 if (virtio_has_feature(vdev, VIRTIO_F_ORDER_PLATFORM)) vq->weak_barriers =3D false; @@ -2284,9 +2632,39 @@ static const struct virtqueue_ops packed_ops =3D { .reset =3D virtqueue_reset_packed, }; =20 +static const struct virtqueue_ops split_in_order_ops =3D { + .add =3D virtqueue_add_split, + .get =3D virtqueue_get_buf_ctx_split_in_order, + .kick_prepare =3D virtqueue_kick_prepare_split, + .disable_cb =3D virtqueue_disable_cb_split, + .enable_cb_delayed =3D virtqueue_enable_cb_delayed_split, + .enable_cb_prepare =3D virtqueue_enable_cb_prepare_split, + .poll =3D virtqueue_poll_split, + .detach_unused_buf =3D virtqueue_detach_unused_buf_split, + .more_used =3D more_used_split_in_order, + .resize =3D virtqueue_resize_split, + .reset =3D virtqueue_reset_split, +}; + +static const struct virtqueue_ops packed_in_order_ops =3D { + .add =3D virtqueue_add_packed_in_order, + .get =3D virtqueue_get_buf_ctx_packed_in_order, + .kick_prepare =3D virtqueue_kick_prepare_packed, + .disable_cb =3D virtqueue_disable_cb_packed, + .enable_cb_delayed =3D virtqueue_enable_cb_delayed_packed, + .enable_cb_prepare =3D virtqueue_enable_cb_prepare_packed, + .poll =3D virtqueue_poll_packed, + .detach_unused_buf =3D virtqueue_detach_unused_buf_packed, + .more_used =3D more_used_packed_in_order, + .resize =3D virtqueue_resize_packed, + .reset =3D virtqueue_reset_packed, +}; + static const struct virtqueue_ops *const all_ops[VQ_TYPE_MAX] =3D { [SPLIT] =3D &split_ops, - [PACKED] =3D &packed_ops + [PACKED] =3D &packed_ops, + [SPLIT_IN_ORDER] =3D &split_in_order_ops, + [PACKED_IN_ORDER] =3D &packed_in_order_ops, }; =20 static int virtqueue_disable_and_recycle(struct virtqueue *_vq, @@ -2342,6 +2720,12 @@ static int virtqueue_enable_after_reset(struct virtq= ueue *_vq) case PACKED: \ ret =3D all_ops[PACKED]->op(vq, ##__VA_ARGS__); \ break; \ + case SPLIT_IN_ORDER: \ + ret =3D all_ops[SPLIT_IN_ORDER]->op(vq, ##__VA_ARGS__); \ + break; \ + case PACKED_IN_ORDER: \ + ret =3D all_ops[PACKED_IN_ORDER]->op(vq, ##__VA_ARGS__); \ + break; \ default: \ BUG(); \ break; \ @@ -2358,10 +2742,16 @@ static int virtqueue_enable_after_reset(struct virt= queue *_vq) case PACKED: \ all_ops[PACKED]->op(vq, ##__VA_ARGS__); \ break; \ - default: \ - BUG(); \ - break; \ - } \ + case SPLIT_IN_ORDER: \ + all_ops[SPLIT_IN_ORDER]->op(vq, ##__VA_ARGS__); \ + break; \ + case PACKED_IN_ORDER: \ + all_ops[PACKED_IN_ORDER]->op(vq, ##__VA_ARGS__); \ + break; \ + default: \ + BUG(); \ + break; \ + } \ }) =20 static inline int virtqueue_add(struct virtqueue *_vq, @@ -3078,6 +3468,8 @@ void vring_transport_features(struct virtio_device *v= dev) break; case VIRTIO_F_NOTIFICATION_DATA: break; + case VIRTIO_F_IN_ORDER: + break; default: /* We don't understand this bit. */ __virtio_clear_bit(vdev, i); --=20 2.31.1