From nobody Mon Oct 6 06:40:55 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2F0AE204F8C for ; Thu, 24 Jul 2025 06:40:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339239; cv=none; b=CkhDcDkHy4xrqF4SEOsCNr2K7KsOfupxff0xfnQN8bM1qHVfj62lsGWPahN6cegHWazpXQSSkfnhZopi7V/6rk6ZyHoIq83PXzyXH2/x6pV9bNmpVkOMD1IsK+oeqZp5a74ZbX+dVOBPv5eCcWLFdGHQM0BVmf8remD59MHrVFY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339239; c=relaxed/simple; bh=Rfi3OcXIONn1RnF7f3k0VTEzUy6uTbi2+OHu9nbph9U=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=XkIwDLMOzxYNQ5uQTBktOYPrRpN8C8Cl0vt4CldlvFHnpCEasZqU3ouu+z3m08k0tuN530zmqvfbG4uHqBydSR8X05LV7T14bgQgUngrM97A8h0PuCu+quNE/mj14mjqRA4etedpgdCHFpM3ZKCbVntY+v5WXRPr3Vx6hVvuhQo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=KBK7sbFh; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="KBK7sbFh" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1753339234; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5iLSNS/4EUts+wMGIjSO1bWgp4tWcUWSbpWKLGtAJvY=; b=KBK7sbFh8bCfVnIcr8eLKzAW5r2eMerECrHq8LENcbB/xThs9eQVBScqcSMHaQ+ieKKmU+ 3wQZCLq+jl/LbbmqTA4n5wR1Q464tSGg37IKduRAxeyjMKtjirlbyWjgc2zbd2xQYLczg/ pSs3oxZroD54bfIpfZ4+2BXl+6SXHMg= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-561-GaRPeuoaMyORL7w43Xxa0A-1; Thu, 24 Jul 2025 02:40:32 -0400 X-MC-Unique: GaRPeuoaMyORL7w43Xxa0A-1 X-Mimecast-MFC-AGG-ID: GaRPeuoaMyORL7w43Xxa0A_1753339231 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 074931956087; Thu, 24 Jul 2025 06:40:31 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.231]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 690F818002AF; Thu, 24 Jul 2025 06:40:27 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V4 01/19] virtio_ring: rename virtqueue_reinit_xxx to virtqueue_reset_xxx() Date: Thu, 24 Jul 2025 14:39:59 +0800 Message-ID: <20250724064017.26058-2-jasowang@redhat.com> In-Reply-To: <20250724064017.26058-1-jasowang@redhat.com> References: <20250724064017.26058-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 To be consistent with virtqueue_reset(). Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index f5062061c408..02df3c9ab337 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -1005,7 +1005,7 @@ static void virtqueue_vring_init_split(struct vring_v= irtqueue_split *vring_split } } =20 -static void virtqueue_reinit_split(struct vring_virtqueue *vq) +static void virtqueue_reset_split(struct vring_virtqueue *vq) { int num; =20 @@ -1248,7 +1248,7 @@ static int virtqueue_resize_split(struct virtqueue *_= vq, u32 num) err_state_extra: vring_free_split(&vring_split, vdev, vring_dma_dev(vq)); err: - virtqueue_reinit_split(vq); + virtqueue_reset_split(vq); return -ENOMEM; } =20 @@ -2092,7 +2092,7 @@ static void virtqueue_vring_attach_packed(struct vrin= g_virtqueue *vq, vq->free_head =3D 0; } =20 -static void virtqueue_reinit_packed(struct vring_virtqueue *vq) +static void virtqueue_reset_packed(struct vring_virtqueue *vq) { memset(vq->packed.vring.device, 0, vq->packed.event_size_in_bytes); memset(vq->packed.vring.driver, 0, vq->packed.event_size_in_bytes); @@ -2219,7 +2219,7 @@ static int virtqueue_resize_packed(struct virtqueue *= _vq, u32 num) err_state_extra: vring_free_packed(&vring_packed, vdev, vring_dma_dev(vq)); err_ring: - virtqueue_reinit_packed(vq); + virtqueue_reset_packed(vq); return -ENOMEM; } =20 @@ -2860,9 +2860,9 @@ int virtqueue_reset(struct virtqueue *_vq, recycle_done(_vq); =20 if (vq->packed_ring) - virtqueue_reinit_packed(vq); + virtqueue_reset_packed(vq); else - virtqueue_reinit_split(vq); + virtqueue_reset_split(vq); =20 return virtqueue_enable_after_reset(_vq); } --=20 2.39.5 From nobody Mon Oct 6 06:40:55 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 171CF2236FD for ; Thu, 24 Jul 2025 06:40:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339242; cv=none; b=WUq40FprQJkKfGgZkaP6C75qxmSKYGQ8BCH2FO9D2qtnvZaCJyqu82+ZYFgJ3bFVFwmHmtmrn5RAR87m9IxSDnkVycDWvDKp/bjePguNnVNybeYeko+ZdCBcHd9B3eYmuuaIvYNUsDigtuochkNW9gV5DL3tAJ1JN0s5zgO3TMc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339242; c=relaxed/simple; bh=uwOmbujZkWwi6D/4k0/v8CyYvQ0s2kR+uX1sT0+5Cto=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=GVWy3V58U97nfQl+qEVWoN6OkEydcoyn1B/0uK2mKJjMM8GYEKzdKfpORQg6ZzcM+msQhfkvWRJE0YCX+ehzBy764SPfTov1SVpGfUtEmkXmIS7OtTGTgQgnM82oR2i4lo3fsWidMKFktQYoqQyu+aqIosqwhpc6x7r3ps8LgvM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=bROK3MID; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="bROK3MID" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1753339240; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mnKH3qFNhW0egQn6fG6AuCnYHUyntbTarBkYRfvYpGA=; b=bROK3MIDGs86+2pTXz2aMUtLB8BmTZI4qzAyG26JIYGveRLaBVYrVPQ9IwwmTEBQ0fz6i2 HP4rRgPXwVHy+Bm0RIgXsOV/ZWufiELXOQeHYHmN8AQCSjbyAygLdzeoUdKtUx6mE78e8K sy1/3nGr/aYST9jQO5/DPNx51ddnzV0= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-595-jJRPRtOaM2-buU408YXjNg-1; Thu, 24 Jul 2025 02:40:35 -0400 X-MC-Unique: jJRPRtOaM2-buU408YXjNg-1 X-Mimecast-MFC-AGG-ID: jJRPRtOaM2-buU408YXjNg_1753339235 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id E28D319560A5; Thu, 24 Jul 2025 06:40:34 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.231]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id B975F180035E; Thu, 24 Jul 2025 06:40:31 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V4 02/19] virtio_ring: switch to use vring_virtqueue in virtqueue_poll variants Date: Thu, 24 Jul 2025 14:40:00 +0800 Message-ID: <20250724064017.26058-3-jasowang@redhat.com> In-Reply-To: <20250724064017.26058-1-jasowang@redhat.com> References: <20250724064017.26058-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Those variants are used internally so let's switch to use vring_virtqueue as parameter to be consistent with other internal virtqueue helpers. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 02df3c9ab337..3eb1449d13b3 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -915,11 +915,10 @@ static unsigned int virtqueue_enable_cb_prepare_split= (struct virtqueue *_vq) return last_used_idx; } =20 -static bool virtqueue_poll_split(struct virtqueue *_vq, unsigned int last_= used_idx) +static bool virtqueue_poll_split(struct vring_virtqueue *vq, + unsigned int last_used_idx) { - struct vring_virtqueue *vq =3D to_vvq(_vq); - - return (u16)last_used_idx !=3D virtio16_to_cpu(_vq->vdev, + return (u16)last_used_idx !=3D virtio16_to_cpu(vq->vq.vdev, vq->split.vring.used->idx); } =20 @@ -1845,9 +1844,8 @@ static unsigned int virtqueue_enable_cb_prepare_packe= d(struct virtqueue *_vq) return vq->last_used_idx; } =20 -static bool virtqueue_poll_packed(struct virtqueue *_vq, u16 off_wrap) +static bool virtqueue_poll_packed(struct vring_virtqueue *vq, u16 off_wrap) { - struct vring_virtqueue *vq =3D to_vvq(_vq); bool wrap_counter; u16 used_idx; =20 @@ -2612,8 +2610,8 @@ bool virtqueue_poll(struct virtqueue *_vq, unsigned i= nt last_used_idx) return false; =20 virtio_mb(vq->weak_barriers); - return vq->packed_ring ? virtqueue_poll_packed(_vq, last_used_idx) : - virtqueue_poll_split(_vq, last_used_idx); + return vq->packed_ring ? virtqueue_poll_packed(vq, last_used_idx) : + virtqueue_poll_split(vq, last_used_idx); } EXPORT_SYMBOL_GPL(virtqueue_poll); =20 --=20 2.39.5 From nobody Mon Oct 6 06:40:55 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 327E0227E8A for ; Thu, 24 Jul 2025 06:40:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339247; cv=none; b=W9UZO6hPO3u9ielBa4WlbdTh5NscPhuL6cut2dxO4Zshs1G5y2byK/4zK9x9fViIHpKzXFS0xrwp6xw2V9oDIC4oxXJmBj9+CleRr2YMaMp11pbPvAay9XaHfwyQI+/dtQklJE0P6aPDTDi/uXK9nRS7OiEkOOhiDeR/PJ/2WgQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339247; c=relaxed/simple; bh=2tMfp9N5CvxGx703ZPmr0F/kSqDjTU32XFYL0oVAb5A=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=ff25KglT+GBAyoI/r+01DiDkXlGhvJr6cPHbAak948MxBfp2EPGq24sGSEtjo6bf+T51nitXINk3StAeEpGIpEPujzbUD10UcDw9Klm8j0gNDEge62VE60F0pyRCahwUKz59iLOoPBKERsFy8zyJ/M9GK1HMKtgu6794B7SedaU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=NDYxTtJN; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="NDYxTtJN" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1753339244; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1PyVKJ9gVTBwaZQPQwEgcjlTiZr8Nc+8prtl23yNdoA=; b=NDYxTtJNa2y6Xf75pIM1QUtD2G620O1ZsPBYUq4q599/uW/YJvUUUw4EktZb/k7j8oXYSk UZkgKEnbG2/wQZ0q2VbT3wjZwQltUN+nbcxRCAnqQn0wwAgB9kUNyyfcIwEwPwI3pmissT Oo6hG2w8A7NAhbpVsOYn9WgzdU7SnYA= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-183-wYD_saJ5NBqs-Kqlnjo1bQ-1; Thu, 24 Jul 2025 02:40:40 -0400 X-MC-Unique: wYD_saJ5NBqs-Kqlnjo1bQ-1 X-Mimecast-MFC-AGG-ID: wYD_saJ5NBqs-Kqlnjo1bQ_1753339239 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 23AE1180034E; Thu, 24 Jul 2025 06:40:39 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.231]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id AFE8318002AF; Thu, 24 Jul 2025 06:40:35 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V4 03/19] virtio_ring: unify logic of virtqueue_poll() and more_used() Date: Thu, 24 Jul 2025 14:40:01 +0800 Message-ID: <20250724064017.26058-4-jasowang@redhat.com> In-Reply-To: <20250724064017.26058-1-jasowang@redhat.com> References: <20250724064017.26058-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 This patch unifies the logic of virtqueue_poll() and more_used() for better code reusing and ease the future in order implementation. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 48 +++++++++++++++--------------------- 1 file changed, 20 insertions(+), 28 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 3eb1449d13b3..daa63fd54693 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -802,12 +802,18 @@ static void detach_buf_split(struct vring_virtqueue *= vq, unsigned int head, } } =20 -static bool more_used_split(const struct vring_virtqueue *vq) +static bool virtqueue_poll_split(const struct vring_virtqueue *vq, + unsigned int last_used_idx) { - return vq->last_used_idx !=3D virtio16_to_cpu(vq->vq.vdev, + return (u16)last_used_idx !=3D virtio16_to_cpu(vq->vq.vdev, vq->split.vring.used->idx); } =20 +static bool more_used_split(const struct vring_virtqueue *vq) +{ + return virtqueue_poll_split(vq, vq->last_used_idx); +} + static void *virtqueue_get_buf_ctx_split(struct virtqueue *_vq, unsigned int *len, void **ctx) @@ -915,13 +921,6 @@ static unsigned int virtqueue_enable_cb_prepare_split(= struct virtqueue *_vq) return last_used_idx; } =20 -static bool virtqueue_poll_split(struct vring_virtqueue *vq, - unsigned int last_used_idx) -{ - return (u16)last_used_idx !=3D virtio16_to_cpu(vq->vq.vdev, - vq->split.vring.used->idx); -} - static bool virtqueue_enable_cb_delayed_split(struct virtqueue *_vq) { struct vring_virtqueue *vq =3D to_vvq(_vq); @@ -1711,16 +1710,20 @@ static inline bool is_used_desc_packed(const struct= vring_virtqueue *vq, return avail =3D=3D used && used =3D=3D used_wrap_counter; } =20 -static bool more_used_packed(const struct vring_virtqueue *vq) +static bool virtqueue_poll_packed(const struct vring_virtqueue *vq, u16 of= f_wrap) { - u16 last_used; - u16 last_used_idx; - bool used_wrap_counter; + bool wrap_counter; + u16 used_idx; =20 - last_used_idx =3D READ_ONCE(vq->last_used_idx); - last_used =3D packed_last_used(last_used_idx); - used_wrap_counter =3D packed_used_wrap_counter(last_used_idx); - return is_used_desc_packed(vq, last_used, used_wrap_counter); + wrap_counter =3D off_wrap >> VRING_PACKED_EVENT_F_WRAP_CTR; + used_idx =3D off_wrap & ~(1 << VRING_PACKED_EVENT_F_WRAP_CTR); + + return is_used_desc_packed(vq, used_idx, wrap_counter); +} + +static bool more_used_packed(const struct vring_virtqueue *vq) +{ + return virtqueue_poll_packed(vq, READ_ONCE(vq->last_used_idx)); } =20 static void *virtqueue_get_buf_ctx_packed(struct virtqueue *_vq, @@ -1844,17 +1847,6 @@ static unsigned int virtqueue_enable_cb_prepare_pack= ed(struct virtqueue *_vq) return vq->last_used_idx; } =20 -static bool virtqueue_poll_packed(struct vring_virtqueue *vq, u16 off_wrap) -{ - bool wrap_counter; - u16 used_idx; - - wrap_counter =3D off_wrap >> VRING_PACKED_EVENT_F_WRAP_CTR; - used_idx =3D off_wrap & ~(1 << VRING_PACKED_EVENT_F_WRAP_CTR); - - return is_used_desc_packed(vq, used_idx, wrap_counter); -} - static bool virtqueue_enable_cb_delayed_packed(struct virtqueue *_vq) { struct vring_virtqueue *vq =3D to_vvq(_vq); --=20 2.39.5 From nobody Mon Oct 6 06:40:55 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B7A1D229B13 for ; Thu, 24 Jul 2025 06:40:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339250; cv=none; b=PxDsNpVIWOrbZDnu01TtvkE1yLicke8InucdGhS0MlisO2hfeR9Aa+uK+9RvYLVpJjE2VOGYYz+z1coal8AoG6KCfjjUEDjHom+Y1mwthusc613Y7UFgmiKL2BAGD04/6EuWCHXlRcPyKKBcqhRJkMMfA6CrwzhFP+GoJbC4M2I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339250; c=relaxed/simple; bh=KBR7BgN5zstd1g6lnmFptBjPEUSLbbxgPm6fTJwF4vs=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Jh1ncPw+XYEN9P2IQBoWD4lLF8Z9sU016S330NWM3Lljn0ohWZMopQkiHDOwyuIdA2qz2PIG6LKjgS4mDk2xWNTV0c0/jOAK8MRQgaahmwpK+FDD6K4NsFLL+mJtNOZtt9K6Osya5xsLL71djf48zRvM7bxxicZkuAZ5SjZ8VG0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=BF5JHc6i; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="BF5JHc6i" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1753339247; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eYNF07NdjCbeScRweEDKjdTIYj1lDxvOzhF/Gj+8+yk=; b=BF5JHc6iffnpOARF3jTp8F9NnVrpPh1yZ1Eph1AnD4ra9GgzbtJ+nsyT9N9YENexV5gwop ofwmKdGgfnh+07M+J21E/SDFjSPJ50Es139QN2cGf78QQMS6D1D9Cz+3UtgImQzRsteafb 1zu79Ll7ePMgbmq+0lWh3f/nBEvHG60= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-675-vutYopfVPh6oiZdBtDOWWQ-1; Thu, 24 Jul 2025 02:40:44 -0400 X-MC-Unique: vutYopfVPh6oiZdBtDOWWQ-1 X-Mimecast-MFC-AGG-ID: vutYopfVPh6oiZdBtDOWWQ_1753339243 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 5444718004A7; Thu, 24 Jul 2025 06:40:43 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.231]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 2271F18002AF; Thu, 24 Jul 2025 06:40:39 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V4 04/19] virtio_ring: switch to use vring_virtqueue for virtqueue resize variants Date: Thu, 24 Jul 2025 14:40:02 +0800 Message-ID: <20250724064017.26058-5-jasowang@redhat.com> In-Reply-To: <20250724064017.26058-1-jasowang@redhat.com> References: <20250724064017.26058-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Those variants are used internally so let's switch to use vring_virtqueue as parameter to be consistent with other internal virtqueue helpers. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index daa63fd54693..f30d491678a5 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -1216,11 +1216,10 @@ static struct virtqueue *vring_create_virtqueue_spl= it( return vq; } =20 -static int virtqueue_resize_split(struct virtqueue *_vq, u32 num) +static int virtqueue_resize_split(struct vring_virtqueue *vq, u32 num) { struct vring_virtqueue_split vring_split =3D {}; - struct vring_virtqueue *vq =3D to_vvq(_vq); - struct virtio_device *vdev =3D _vq->vdev; + struct virtio_device *vdev =3D vq->vq.vdev; int err; =20 err =3D vring_alloc_queue_split(&vring_split, vdev, num, @@ -2183,11 +2182,10 @@ static struct virtqueue *vring_create_virtqueue_pac= ked( return vq; } =20 -static int virtqueue_resize_packed(struct virtqueue *_vq, u32 num) +static int virtqueue_resize_packed(struct vring_virtqueue *vq, u32 num) { struct vring_virtqueue_packed vring_packed =3D {}; - struct vring_virtqueue *vq =3D to_vvq(_vq); - struct virtio_device *vdev =3D _vq->vdev; + struct virtio_device *vdev =3D vq->vq.vdev; int err; =20 if (vring_alloc_queue_packed(&vring_packed, vdev, num, vring_dma_dev(vq))) @@ -2809,9 +2807,9 @@ int virtqueue_resize(struct virtqueue *_vq, u32 num, recycle_done(_vq); =20 if (vq->packed_ring) - err =3D virtqueue_resize_packed(_vq, num); + err =3D virtqueue_resize_packed(vq, num); else - err =3D virtqueue_resize_split(_vq, num); + err =3D virtqueue_resize_split(vq, num); =20 err_reset =3D virtqueue_enable_after_reset(_vq); if (err_reset) --=20 2.39.5 From nobody Mon Oct 6 06:40:55 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 04C0122F755 for ; Thu, 24 Jul 2025 06:40:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339252; cv=none; b=Ju3yQcULu4nlPxNwyB3oot8mtYGTjOyH4dtug6BBi/AobkexeeBlM9TiBhB9jNJw2r+5S8NKcluXtrhdDL4SBeshrvLMdY+5YdaAglu/afxw8SM4oYh1rwLbxWj5YRNWZU7p3whNicThShNVaQ++imKEG8ULyCJ714+CFQaT3fI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339252; c=relaxed/simple; bh=f8Hay0T5RLtr2APW8jx5BDxUXw+HF7929uZ5d5RBo3c=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=TqCTrqGgj32FVFGNhPATqEfMZc1sUQzkifPQY2pxjeQey/mdUxNPB1WPkazBBY6tj+JSruJ0zaxqBnF5nSxkD7571CnV5F2ddvdMHJQ8taCkQyRgaaWbfVP0oJXbY0ad5tTVuj6JCoNFvifQwjEQz8Uub5NZxz5K19NZOSD7HRM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=CMQcH59O; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="CMQcH59O" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1753339250; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=P2xXvEXElQU9pxX1JjWEG3fP5Z6meiP0S2R7oUbAZzk=; b=CMQcH59OZ55JCMDbPEDYmM0t6Y2Xvokw2neXeAjqE/VSiXRkezvamimlGl8ZYLSeS7+Onn KAX0ZqgeQePwWvREkIfc2SyiujQgTdRVmJjMrHJcQVI992LdHcJD/ceYM1XNBUoPh3PRbK 0xXcqXXYQMBWPw4TDkQvHKnFn/WAgdY= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-340-i-6P8BeYMfKDk7ADFKlTeg-1; Thu, 24 Jul 2025 02:40:48 -0400 X-MC-Unique: i-6P8BeYMfKDk7ADFKlTeg-1 X-Mimecast-MFC-AGG-ID: i-6P8BeYMfKDk7ADFKlTeg_1753339247 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 4619D195608B; Thu, 24 Jul 2025 06:40:47 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.231]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 105EE180035E; Thu, 24 Jul 2025 06:40:43 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V4 05/19] virtio_ring: switch to use vring_virtqueue for virtqueue_kick_prepare variants Date: Thu, 24 Jul 2025 14:40:03 +0800 Message-ID: <20250724064017.26058-6-jasowang@redhat.com> In-Reply-To: <20250724064017.26058-1-jasowang@redhat.com> References: <20250724064017.26058-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Those variants are used internally so let's switch to use vring_virtqueue as parameter to be consistent with other internal virtqueue helpers. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index f30d491678a5..ac14324588bb 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -713,9 +713,8 @@ static inline int virtqueue_add_split(struct virtqueue = *_vq, return -ENOMEM; } =20 -static bool virtqueue_kick_prepare_split(struct virtqueue *_vq) +static bool virtqueue_kick_prepare_split(struct vring_virtqueue *vq) { - struct vring_virtqueue *vq =3D to_vvq(_vq); u16 new, old; bool needs_kick; =20 @@ -732,12 +731,12 @@ static bool virtqueue_kick_prepare_split(struct virtq= ueue *_vq) LAST_ADD_TIME_INVALID(vq); =20 if (vq->event) { - needs_kick =3D vring_need_event(virtio16_to_cpu(_vq->vdev, + needs_kick =3D vring_need_event(virtio16_to_cpu(vq->vq.vdev, vring_avail_event(&vq->split.vring)), new, old); } else { needs_kick =3D !(vq->split.vring.used->flags & - cpu_to_virtio16(_vq->vdev, + cpu_to_virtio16(vq->vq.vdev, VRING_USED_F_NO_NOTIFY)); } END_USE(vq); @@ -1597,9 +1596,8 @@ static inline int virtqueue_add_packed(struct virtque= ue *_vq, return -EIO; } =20 -static bool virtqueue_kick_prepare_packed(struct virtqueue *_vq) +static bool virtqueue_kick_prepare_packed(struct vring_virtqueue *vq) { - struct vring_virtqueue *vq =3D to_vvq(_vq); u16 new, old, off_wrap, flags, wrap_counter, event_idx; bool needs_kick; union { @@ -2458,8 +2456,8 @@ bool virtqueue_kick_prepare(struct virtqueue *_vq) { struct vring_virtqueue *vq =3D to_vvq(_vq); =20 - return vq->packed_ring ? virtqueue_kick_prepare_packed(_vq) : - virtqueue_kick_prepare_split(_vq); + return vq->packed_ring ? virtqueue_kick_prepare_packed(vq) : + virtqueue_kick_prepare_split(vq); } EXPORT_SYMBOL_GPL(virtqueue_kick_prepare); =20 --=20 2.39.5 From nobody Mon Oct 6 06:40:55 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0CB8821C9F5 for ; Thu, 24 Jul 2025 06:40:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339257; cv=none; b=tyHWzjoKcAQQFRlImD+nHvfNsCJMZf8PYZ5zctfcY0Lv1uWkl1RlGh9UnpEu6jhMdlHG72zvU5vcUgIqT0qLgo2hIMGjiv0Vdb5U+jMMjepKCK8A3/c6N/191nCBpoSrEippyAnjZRl9cKkEVLEWwwYrcHAERYv3BWzLz2vqVHU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339257; c=relaxed/simple; bh=mybsbMJf9XWfYlQRyil4sH4TitgQh7iRuM8qzpD62m4=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=DcvmkK4g47E0geY0MsvRlqbRjSH74sdw2kaHZ7ap26dRBR2PtSQs+nj/Y2l21HkfYYHHmU254+ZMgf50RJD7+TAnsmZ6AM1f8PWnkaZf2DyiTJFgVadClSJZ8TXhhDgTQ8kOAXIlNgryq2kSm8wIbWqk+kWdRACKjAWvuTePUxA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=ALeiZMHz; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="ALeiZMHz" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1753339255; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Y9OqrbFE0wKx4lqeLJtA8rj6ozKgcP0h080xh9UuVZo=; b=ALeiZMHzhPUa0V3VMzUDr/4UCFJmnq+Fz3gCGFenJZ046XOtZT0ac6/Rt6ON44RKsbDlrH YOX44yH8O6XGumGUy3e+0m9EZozQjm860QnyI3R0kDmBy3afUG0kUvoX8A7o5p2LZVzbJy SkfEFPiruCm53zph4SHdaAM1Pl7PAL0= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-548-wPrSeQrlPya-3q-ZGfUK6g-1; Thu, 24 Jul 2025 02:40:53 -0400 X-MC-Unique: wPrSeQrlPya-3q-ZGfUK6g-1 X-Mimecast-MFC-AGG-ID: wPrSeQrlPya-3q-ZGfUK6g_1753339252 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 10BFF1956088; Thu, 24 Jul 2025 06:40:52 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.231]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id EB1C218002AF; Thu, 24 Jul 2025 06:40:47 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V4 06/19] virtio_ring: switch to use vring_virtqueue for virtqueue_add variants Date: Thu, 24 Jul 2025 14:40:04 +0800 Message-ID: <20250724064017.26058-7-jasowang@redhat.com> In-Reply-To: <20250724064017.26058-1-jasowang@redhat.com> References: <20250724064017.26058-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Those variants are used internally so let's switch to use vring_virtqueue as parameter to be consistent with other internal virtqueue helpers. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 40 +++++++++++++++++------------------- 1 file changed, 19 insertions(+), 21 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index ac14324588bb..b7489d19d8ae 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -472,7 +472,7 @@ static unsigned int vring_unmap_one_split(const struct = vring_virtqueue *vq, return extra->next; } =20 -static struct vring_desc *alloc_indirect_split(struct virtqueue *_vq, +static struct vring_desc *alloc_indirect_split(struct vring_virtqueue *vq, unsigned int total_sg, gfp_t gfp) { @@ -501,7 +501,7 @@ static struct vring_desc *alloc_indirect_split(struct v= irtqueue *_vq, return desc; } =20 -static inline unsigned int virtqueue_add_desc_split(struct virtqueue *vq, +static inline unsigned int virtqueue_add_desc_split(struct vring_virtqueue= *vq, struct vring_desc *desc, struct vring_desc_extra *extra, unsigned int i, @@ -509,11 +509,12 @@ static inline unsigned int virtqueue_add_desc_split(s= truct virtqueue *vq, unsigned int len, u16 flags, bool premapped) { + struct virtio_device *vdev =3D vq->vq.vdev; u16 next; =20 - desc[i].flags =3D cpu_to_virtio16(vq->vdev, flags); - desc[i].addr =3D cpu_to_virtio64(vq->vdev, addr); - desc[i].len =3D cpu_to_virtio32(vq->vdev, len); + desc[i].flags =3D cpu_to_virtio16(vdev, flags); + desc[i].addr =3D cpu_to_virtio64(vdev, addr); + desc[i].len =3D cpu_to_virtio32(vdev, len); =20 extra[i].addr =3D premapped ? DMA_MAPPING_ERROR : addr; extra[i].len =3D len; @@ -521,12 +522,12 @@ static inline unsigned int virtqueue_add_desc_split(s= truct virtqueue *vq, =20 next =3D extra[i].next; =20 - desc[i].next =3D cpu_to_virtio16(vq->vdev, next); + desc[i].next =3D cpu_to_virtio16(vdev, next); =20 return next; } =20 -static inline int virtqueue_add_split(struct virtqueue *_vq, +static inline int virtqueue_add_split(struct vring_virtqueue *vq, struct scatterlist *sgs[], unsigned int total_sg, unsigned int out_sgs, @@ -536,7 +537,6 @@ static inline int virtqueue_add_split(struct virtqueue = *_vq, bool premapped, gfp_t gfp) { - struct vring_virtqueue *vq =3D to_vvq(_vq); struct vring_desc_extra *extra; struct scatterlist *sg; struct vring_desc *desc; @@ -561,7 +561,7 @@ static inline int virtqueue_add_split(struct virtqueue = *_vq, head =3D vq->free_head; =20 if (virtqueue_use_indirect(vq, total_sg)) - desc =3D alloc_indirect_split(_vq, total_sg, gfp); + desc =3D alloc_indirect_split(vq, total_sg, gfp); else { desc =3D NULL; WARN_ON_ONCE(total_sg > vq->split.vring.num && !vq->indirect); @@ -608,7 +608,7 @@ static inline int virtqueue_add_split(struct virtqueue = *_vq, /* Note that we trust indirect descriptor * table since it use stream DMA mapping. */ - i =3D virtqueue_add_desc_split(_vq, desc, extra, i, addr, len, + i =3D virtqueue_add_desc_split(vq, desc, extra, i, addr, len, VRING_DESC_F_NEXT, premapped); } @@ -625,14 +625,14 @@ static inline int virtqueue_add_split(struct virtqueu= e *_vq, /* Note that we trust indirect descriptor * table since it use stream DMA mapping. */ - i =3D virtqueue_add_desc_split(_vq, desc, extra, i, addr, len, + i =3D virtqueue_add_desc_split(vq, desc, extra, i, addr, len, VRING_DESC_F_NEXT | VRING_DESC_F_WRITE, premapped); } } /* Last one doesn't continue. */ - desc[prev].flags &=3D cpu_to_virtio16(_vq->vdev, ~VRING_DESC_F_NEXT); + desc[prev].flags &=3D cpu_to_virtio16(vq->vq.vdev, ~VRING_DESC_F_NEXT); if (!indirect && vring_need_unmap_buffer(vq, &extra[prev])) vq->split.desc_extra[prev & (vq->split.vring.num - 1)].flags &=3D ~VRING_DESC_F_NEXT; @@ -645,7 +645,7 @@ static inline int virtqueue_add_split(struct virtqueue = *_vq, if (vring_mapping_error(vq, addr)) goto unmap_release; =20 - virtqueue_add_desc_split(_vq, vq->split.vring.desc, + virtqueue_add_desc_split(vq, vq->split.vring.desc, vq->split.desc_extra, head, addr, total_sg * sizeof(struct vring_desc), @@ -671,13 +671,13 @@ static inline int virtqueue_add_split(struct virtqueu= e *_vq, /* Put entry in available array (but don't update avail->idx until they * do sync). */ avail =3D vq->split.avail_idx_shadow & (vq->split.vring.num - 1); - vq->split.vring.avail->ring[avail] =3D cpu_to_virtio16(_vq->vdev, head); + vq->split.vring.avail->ring[avail] =3D cpu_to_virtio16(vq->vq.vdev, head); =20 /* Descriptors and available array need to be set before we expose the * new available array entries. */ virtio_wmb(vq->weak_barriers); vq->split.avail_idx_shadow++; - vq->split.vring.avail->idx =3D cpu_to_virtio16(_vq->vdev, + vq->split.vring.avail->idx =3D cpu_to_virtio16(vq->vq.vdev, vq->split.avail_idx_shadow); vq->num_added++; =20 @@ -687,7 +687,7 @@ static inline int virtqueue_add_split(struct virtqueue = *_vq, /* This is very unlikely, but theoretically possible. Kick * just in case. */ if (unlikely(vq->num_added =3D=3D (1 << 16) - 1)) - virtqueue_kick(_vq); + virtqueue_kick(&vq->vq); =20 return 0; =20 @@ -702,7 +702,6 @@ static inline int virtqueue_add_split(struct virtqueue = *_vq, for (n =3D 0; n < total_sg; n++) { if (i =3D=3D err_idx) break; - i =3D vring_unmap_one_split(vq, &extra[i]); } =20 @@ -1441,7 +1440,7 @@ static int virtqueue_add_indirect_packed(struct vring= _virtqueue *vq, return -ENOMEM; } =20 -static inline int virtqueue_add_packed(struct virtqueue *_vq, +static inline int virtqueue_add_packed(struct vring_virtqueue *vq, struct scatterlist *sgs[], unsigned int total_sg, unsigned int out_sgs, @@ -1451,7 +1450,6 @@ static inline int virtqueue_add_packed(struct virtque= ue *_vq, bool premapped, gfp_t gfp) { - struct vring_virtqueue *vq =3D to_vvq(_vq); struct vring_packed_desc *desc; struct scatterlist *sg; unsigned int i, n, c, descs_used, err_idx, len; @@ -2263,9 +2261,9 @@ static inline int virtqueue_add(struct virtqueue *_vq, { struct vring_virtqueue *vq =3D to_vvq(_vq); =20 - return vq->packed_ring ? virtqueue_add_packed(_vq, sgs, total_sg, + return vq->packed_ring ? virtqueue_add_packed(vq, sgs, total_sg, out_sgs, in_sgs, data, ctx, premapped, gfp) : - virtqueue_add_split(_vq, sgs, total_sg, + virtqueue_add_split(vq, sgs, total_sg, out_sgs, in_sgs, data, ctx, premapped, gfp); } =20 --=20 2.39.5 From nobody Mon Oct 6 06:40:55 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1B6AC21C9F5 for ; Thu, 24 Jul 2025 06:41:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339263; cv=none; b=samRvflK2JcGz6TAD3Pbix5Dmq4Q5Mgn4Sl8TzF5W46Vd93xeImYWSkEGtFsh8Gjpa5Ch67K3GEIvBa6e/sLdtWCqSHFrIx3KzTPze9ztrqvKmVsQ31loEkli1wczH+e/4+HKp6C4ETCo7crhmHmT2ytyKsewilPAJKxECzenwU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339263; c=relaxed/simple; bh=s+CQBYZNnaJ8oGWROxcoiNNgZmeBC+LHXQTaDibu7js=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=DrvYkCRZUdE0tnqW94vmKsfY+WrkjfoMjPNeu4/nHWsPuCAkZNzAxYw9MRMt4sloEg8x7j2WMdleoy6F3ghUlvt4w1AM2XG43lz6Y3kvRMsQ/TV2Z+OAkfNWonT0R/pHQloQUNK6K9ye5ThsJar8mCrvWI5tYjAlzqNu5/oDmKI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=fg/Io4x5; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="fg/Io4x5" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1753339261; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4IWMeoU2L7LbWevh/6kUc/KaIDdl28mkB1hOWnyA09c=; b=fg/Io4x5woVH9SlYf6Pbi4H2jUEXX2ch0ii68XWqnhihgpoXj6ess7lJSYWCpUnex+KEJb bxgZPpIfe0tSrwr/utTnVkCaefMHJ6u+TvWIx9HTLhGZFa8IQjhZR80BRiFpArT38s3YuV fx5wgUlP7I2GpygiWc7j6w81XRW6Hq4= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-503-P_bIubVyMQeDgjpqzI-wwg-1; Thu, 24 Jul 2025 02:40:57 -0400 X-MC-Unique: P_bIubVyMQeDgjpqzI-wwg-1 X-Mimecast-MFC-AGG-ID: P_bIubVyMQeDgjpqzI-wwg_1753339256 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 1C386180028C; Thu, 24 Jul 2025 06:40:56 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.231]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id C2AC0180035E; Thu, 24 Jul 2025 06:40:52 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V4 07/19] virtio: switch to use vring_virtqueue for virtqueue_add variants Date: Thu, 24 Jul 2025 14:40:05 +0800 Message-ID: <20250724064017.26058-8-jasowang@redhat.com> In-Reply-To: <20250724064017.26058-1-jasowang@redhat.com> References: <20250724064017.26058-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Those variants are used internally so let's switch to use vring_virtqueue as parameter to be consistent with other internal virtqueue helpers. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index b7489d19d8ae..f66ba25a50b1 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -812,11 +812,10 @@ static bool more_used_split(const struct vring_virtqu= eue *vq) return virtqueue_poll_split(vq, vq->last_used_idx); } =20 -static void *virtqueue_get_buf_ctx_split(struct virtqueue *_vq, +static void *virtqueue_get_buf_ctx_split(struct vring_virtqueue *vq, unsigned int *len, void **ctx) { - struct vring_virtqueue *vq =3D to_vvq(_vq); void *ret; unsigned int i; u16 last_used; @@ -838,9 +837,9 @@ static void *virtqueue_get_buf_ctx_split(struct virtque= ue *_vq, virtio_rmb(vq->weak_barriers); =20 last_used =3D (vq->last_used_idx & (vq->split.vring.num - 1)); - i =3D virtio32_to_cpu(_vq->vdev, + i =3D virtio32_to_cpu(vq->vq.vdev, vq->split.vring.used->ring[last_used].id); - *len =3D virtio32_to_cpu(_vq->vdev, + *len =3D virtio32_to_cpu(vq->vq.vdev, vq->split.vring.used->ring[last_used].len); =20 if (unlikely(i >=3D vq->split.vring.num)) { @@ -862,7 +861,7 @@ static void *virtqueue_get_buf_ctx_split(struct virtque= ue *_vq, if (!(vq->split.avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT)) virtio_store_mb(vq->weak_barriers, &vring_used_event(&vq->split.vring), - cpu_to_virtio16(_vq->vdev, vq->last_used_idx)); + cpu_to_virtio16(vq->vq.vdev, vq->last_used_idx)); =20 LAST_ADD_TIME_INVALID(vq); =20 @@ -1721,11 +1720,10 @@ static bool more_used_packed(const struct vring_vir= tqueue *vq) return virtqueue_poll_packed(vq, READ_ONCE(vq->last_used_idx)); } =20 -static void *virtqueue_get_buf_ctx_packed(struct virtqueue *_vq, +static void *virtqueue_get_buf_ctx_packed(struct vring_virtqueue *vq, unsigned int *len, void **ctx) { - struct vring_virtqueue *vq =3D to_vvq(_vq); u16 last_used, id, last_used_idx; bool used_wrap_counter; void *ret; @@ -2525,8 +2523,8 @@ void *virtqueue_get_buf_ctx(struct virtqueue *_vq, un= signed int *len, { struct vring_virtqueue *vq =3D to_vvq(_vq); =20 - return vq->packed_ring ? virtqueue_get_buf_ctx_packed(_vq, len, ctx) : - virtqueue_get_buf_ctx_split(_vq, len, ctx); + return vq->packed_ring ? virtqueue_get_buf_ctx_packed(vq, len, ctx) : + virtqueue_get_buf_ctx_split(vq, len, ctx); } EXPORT_SYMBOL_GPL(virtqueue_get_buf_ctx); =20 --=20 2.39.5 From nobody Mon Oct 6 06:40:55 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 224CE238C1B for ; Thu, 24 Jul 2025 06:41:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339267; cv=none; b=fs8l67TCaWNMSPsWRd0oRnzDHkfxQ6yC5IxKBqpFyr+I+NhzeZP0JgzrhkBCrsLWBXhXdJlQsm02YUi6F5t0aTrCpbXuWMTF4zDtG00z2T0Pxixe8r8qssu8wcqua1Cu+oDOGrqb++9F7htIBpNe3LNWWzEQIaHT0vCZGUqei5w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339267; c=relaxed/simple; bh=e9cz4fPspKJlc/1MR/bizpn9l2cVvyFX7vC5ULqOX1c=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Kx7KJ2OdzlrEShZ4EW+LAU044xLQQwdBoVzPPPa3qgasH1L+gXNNOC/JTP/CbL1fQyNIdYjIAZsOfE4FPzMJmqMrzNH/O+H+hYQg+2kqzbE5Sswip5YL4NmVDIBb5nyk63uYB7fJLgQYmk7SNcphSnMTGOO12qlOVITh3Ppa2dg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=WjJ4MR7a; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="WjJ4MR7a" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1753339265; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=66yYxDS1mtYV0qB82OZU4piVLOScJngkEdFnKp8w3HQ=; b=WjJ4MR7aJmznv4+fwTkuev6h51Gd/QFiHssmMLdE20eMDerW1VdESuzOJfRkDbidb6Vtwx ZWjqxtwJtcEojTOyUDgoDHr7WooGa4kdtT6ftbhWAnhFpIZ+2sYfBqANHCxnPbo6QAflcl Syzf2AGKMplSUVeznIS2ptIMn2TpALE= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-640-PNhFN8WmMJOnJf3s2yYWZQ-1; Thu, 24 Jul 2025 02:41:01 -0400 X-MC-Unique: PNhFN8WmMJOnJf3s2yYWZQ-1 X-Mimecast-MFC-AGG-ID: PNhFN8WmMJOnJf3s2yYWZQ_1753339260 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 4EEC418002A2; Thu, 24 Jul 2025 06:41:00 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.231]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 20E40180035E; Thu, 24 Jul 2025 06:40:56 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V4 08/19] virtio_ring: switch to use vring_virtqueue for enable_cb_prepare variants Date: Thu, 24 Jul 2025 14:40:06 +0800 Message-ID: <20250724064017.26058-9-jasowang@redhat.com> In-Reply-To: <20250724064017.26058-1-jasowang@redhat.com> References: <20250724064017.26058-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Those variants are used internally so let's switch to use vring_virtqueue as parameter to be consistent with other internal virtqueue helpers. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index f66ba25a50b1..fb0690890a39 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -893,9 +893,8 @@ static void virtqueue_disable_cb_split(struct virtqueue= *_vq) } } =20 -static unsigned int virtqueue_enable_cb_prepare_split(struct virtqueue *_v= q) +static unsigned int virtqueue_enable_cb_prepare_split(struct vring_virtque= ue *vq) { - struct vring_virtqueue *vq =3D to_vvq(_vq); u16 last_used_idx; =20 START_USE(vq); @@ -909,10 +908,10 @@ static unsigned int virtqueue_enable_cb_prepare_split= (struct virtqueue *_vq) vq->split.avail_flags_shadow &=3D ~VRING_AVAIL_F_NO_INTERRUPT; if (!vq->event) vq->split.vring.avail->flags =3D - cpu_to_virtio16(_vq->vdev, + cpu_to_virtio16(vq->vq.vdev, vq->split.avail_flags_shadow); } - vring_used_event(&vq->split.vring) =3D cpu_to_virtio16(_vq->vdev, + vring_used_event(&vq->split.vring) =3D cpu_to_virtio16(vq->vq.vdev, last_used_idx =3D vq->last_used_idx); END_USE(vq); return last_used_idx; @@ -1807,10 +1806,8 @@ static void virtqueue_disable_cb_packed(struct virtq= ueue *_vq) } } =20 -static unsigned int virtqueue_enable_cb_prepare_packed(struct virtqueue *_= vq) +static unsigned int virtqueue_enable_cb_prepare_packed(struct vring_virtqu= eue *vq) { - struct vring_virtqueue *vq =3D to_vvq(_vq); - START_USE(vq); =20 /* @@ -2572,8 +2569,8 @@ unsigned int virtqueue_enable_cb_prepare(struct virtq= ueue *_vq) if (vq->event_triggered) vq->event_triggered =3D false; =20 - return vq->packed_ring ? virtqueue_enable_cb_prepare_packed(_vq) : - virtqueue_enable_cb_prepare_split(_vq); + return vq->packed_ring ? virtqueue_enable_cb_prepare_packed(vq) : + virtqueue_enable_cb_prepare_split(vq); } EXPORT_SYMBOL_GPL(virtqueue_enable_cb_prepare); =20 --=20 2.39.5 From nobody Mon Oct 6 06:40:55 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D8DFA239581 for ; Thu, 24 Jul 2025 06:41:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339269; cv=none; b=ZTy34hbK6D3AVCaO8ko/9bxLf5vH1XMS0jPQZnCXpEEfLAStJ2SlPWvSswHx9PQ8q21YbrOjmH8y4rHPnc89J0EXtpt1ifeM5QEul9KNsuUXHV+8PhpEvqabM1EWQn/3ThsZq/Y/ch73icMZ0z9mO8lAmdXSxG7gsIrAbs4F5BU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339269; c=relaxed/simple; bh=kDVF6mjyGQZduywqf17UgF9c5I8+yYT5p6KWhbk3a5k=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=AZGwvBRMe79fo2B1+qdWLy+WCkecDab1Uf9D1lUXtosbRjoY4t/+Ms4O8QGJRX62EpOHl2CC/eOIvKUW3rS2sqDazUgot7QEWCUDOXYkdRN/bTtlWYw3at0ZlW3fKgWrMN8AuMaZUF9n4PbqbitbLx2zsoUAj15nKlEgdp4utJA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=VvkgoozO; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="VvkgoozO" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1753339266; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=y7+NRqTNelY4QQ1QKCpA21fcpe0mrEiUI7rbLdY7J+c=; b=VvkgoozOgAEIInUSOVbpzRFBE1zQrs0+ZZaFzozmn8WvJ/6vZmGdfiiVX5Tqxe2GiRtEo+ j6HeaJStqMrZCdVjvHk2WgZrWI/8gKtDAA9y2RHuOOghcMrPpISW6untyDKxrfBPJSMm7X HikwforMwSgH0Sji0alT8MjZqJ0rDyY= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-138-ELKblT-HOfyGSy1cKC0xGw-1; Thu, 24 Jul 2025 02:41:05 -0400 X-MC-Unique: ELKblT-HOfyGSy1cKC0xGw-1 X-Mimecast-MFC-AGG-ID: ELKblT-HOfyGSy1cKC0xGw_1753339264 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 2A90E19560A7; Thu, 24 Jul 2025 06:41:04 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.231]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 1C42418002AF; Thu, 24 Jul 2025 06:41:00 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V4 09/19] virtio_ring: use vring_virtqueue for enable_cb_delayed variants Date: Thu, 24 Jul 2025 14:40:07 +0800 Message-ID: <20250724064017.26058-10-jasowang@redhat.com> In-Reply-To: <20250724064017.26058-1-jasowang@redhat.com> References: <20250724064017.26058-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Those variants are used internally so let's switch to use vring_virtqueue as parameter to be consistent with other internal virtqueue helpers. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index fb0690890a39..08c74337f704 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -917,9 +917,8 @@ static unsigned int virtqueue_enable_cb_prepare_split(s= truct vring_virtqueue *vq return last_used_idx; } =20 -static bool virtqueue_enable_cb_delayed_split(struct virtqueue *_vq) +static bool virtqueue_enable_cb_delayed_split(struct vring_virtqueue *vq) { - struct vring_virtqueue *vq =3D to_vvq(_vq); u16 bufs; =20 START_USE(vq); @@ -933,7 +932,7 @@ static bool virtqueue_enable_cb_delayed_split(struct vi= rtqueue *_vq) vq->split.avail_flags_shadow &=3D ~VRING_AVAIL_F_NO_INTERRUPT; if (!vq->event) vq->split.vring.avail->flags =3D - cpu_to_virtio16(_vq->vdev, + cpu_to_virtio16(vq->vq.vdev, vq->split.avail_flags_shadow); } /* TODO: tune this threshold */ @@ -941,9 +940,9 @@ static bool virtqueue_enable_cb_delayed_split(struct vi= rtqueue *_vq) =20 virtio_store_mb(vq->weak_barriers, &vring_used_event(&vq->split.vring), - cpu_to_virtio16(_vq->vdev, vq->last_used_idx + bufs)); + cpu_to_virtio16(vq->vq.vdev, vq->last_used_idx + bufs)); =20 - if (unlikely((u16)(virtio16_to_cpu(_vq->vdev, vq->split.vring.used->idx) + if (unlikely((u16)(virtio16_to_cpu(vq->vq.vdev, vq->split.vring.used->idx) - vq->last_used_idx) > bufs)) { END_USE(vq); return false; @@ -1837,9 +1836,8 @@ static unsigned int virtqueue_enable_cb_prepare_packe= d(struct vring_virtqueue *v return vq->last_used_idx; } =20 -static bool virtqueue_enable_cb_delayed_packed(struct virtqueue *_vq) +static bool virtqueue_enable_cb_delayed_packed(struct vring_virtqueue *vq) { - struct vring_virtqueue *vq =3D to_vvq(_vq); u16 used_idx, wrap_counter, last_used_idx; u16 bufs; =20 @@ -2635,8 +2633,8 @@ bool virtqueue_enable_cb_delayed(struct virtqueue *_v= q) if (vq->event_triggered) data_race(vq->event_triggered =3D false); =20 - return vq->packed_ring ? virtqueue_enable_cb_delayed_packed(_vq) : - virtqueue_enable_cb_delayed_split(_vq); + return vq->packed_ring ? virtqueue_enable_cb_delayed_packed(vq) : + virtqueue_enable_cb_delayed_split(vq); } EXPORT_SYMBOL_GPL(virtqueue_enable_cb_delayed); =20 --=20 2.39.5 From nobody Mon Oct 6 06:40:55 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A561423F429 for ; Thu, 24 Jul 2025 06:41:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339275; cv=none; b=Ox3+4IQ0Md6XkeYPp3JAtBs/vg5TVOeY42JuKruWD6z66ezEvJXa2lw/CpRkMhlvv9iFJTFwcpBRiuoz8NjvHIoqLNTLyFjbeH5XZK0i55/9k4UbBS6Kt1QZwMioJ8F22A8hlVTcDwAL0o9Ks69c0HyG0TSyGDm8hE/vtt0yQ3w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339275; c=relaxed/simple; bh=1fcgHBUgA2IgC4Nm3niU78UzHiBHsDnHvPI7e4JEyUY=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=qhS7EWJd61wLhfmbmYtfiXROCSuAd5vLsK4CEhfYvuImgHZiFe5MKd0Q5NeWQO5QSkVIR+4kFQtg9J2/dalk7pmqURBOEIMKpDIxHMribaIPTMmmf1R9fvVByPU9WMr7baxHM/jlyunX5iZVtcyUP9E2UDfjF5K2n5B6dckaQE0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=OHBv7Cij; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="OHBv7Cij" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1753339272; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=u22cjOh4T7psPxv5YQPhWhYBdZ5P+0VH1zRwbXsZph8=; b=OHBv7Cij9ketcUD0Cw0sa4201ZUdCVd19DVW6vW2beVSg2AUK7ijzCMgVuSSmJBtz/kYvu FdsZzFXGHRLhvzeaXpsMp+XUe0EEqmCM638AYgGqBhJtl7/IddRPPMpIFBPwr0S3cdnjM7 VqZQYXAIHNTRy7bBbo7XO+1y5uNoWl0= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-339-WtIxvwpBNCmjZ75gC02mbw-1; Thu, 24 Jul 2025 02:41:09 -0400 X-MC-Unique: WtIxvwpBNCmjZ75gC02mbw-1 X-Mimecast-MFC-AGG-ID: WtIxvwpBNCmjZ75gC02mbw_1753339268 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 3395219560A3; Thu, 24 Jul 2025 06:41:08 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.231]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 02F5A180035E; Thu, 24 Jul 2025 06:41:04 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V4 10/19] virtio_ring: switch to use vring_virtqueue for disable_cb variants Date: Thu, 24 Jul 2025 14:40:08 +0800 Message-ID: <20250724064017.26058-11-jasowang@redhat.com> In-Reply-To: <20250724064017.26058-1-jasowang@redhat.com> References: <20250724064017.26058-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Those variants are used internally so let's switch to use vring_virtqueue as parameter to be consistent with other internal virtqueue helpers. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 08c74337f704..76375fa3379d 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -869,10 +869,8 @@ static void *virtqueue_get_buf_ctx_split(struct vring_= virtqueue *vq, return ret; } =20 -static void virtqueue_disable_cb_split(struct virtqueue *_vq) +static void virtqueue_disable_cb_split(struct vring_virtqueue *vq) { - struct vring_virtqueue *vq =3D to_vvq(_vq); - if (!(vq->split.avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT)) { vq->split.avail_flags_shadow |=3D VRING_AVAIL_F_NO_INTERRUPT; =20 @@ -888,7 +886,7 @@ static void virtqueue_disable_cb_split(struct virtqueue= *_vq) vring_used_event(&vq->split.vring) =3D 0x0; else vq->split.vring.avail->flags =3D - cpu_to_virtio16(_vq->vdev, + cpu_to_virtio16(vq->vq.vdev, vq->split.avail_flags_shadow); } } @@ -1786,10 +1784,8 @@ static void *virtqueue_get_buf_ctx_packed(struct vri= ng_virtqueue *vq, return ret; } =20 -static void virtqueue_disable_cb_packed(struct virtqueue *_vq) +static void virtqueue_disable_cb_packed(struct vring_virtqueue *vq) { - struct vring_virtqueue *vq =3D to_vvq(_vq); - if (vq->packed.event_flags_shadow !=3D VRING_PACKED_EVENT_FLAG_DISABLE) { vq->packed.event_flags_shadow =3D VRING_PACKED_EVENT_FLAG_DISABLE; =20 @@ -2542,9 +2538,9 @@ void virtqueue_disable_cb(struct virtqueue *_vq) struct vring_virtqueue *vq =3D to_vvq(_vq); =20 if (vq->packed_ring) - virtqueue_disable_cb_packed(_vq); + virtqueue_disable_cb_packed(vq); else - virtqueue_disable_cb_split(_vq); + virtqueue_disable_cb_split(vq); } EXPORT_SYMBOL_GPL(virtqueue_disable_cb); =20 --=20 2.39.5 From nobody Mon Oct 6 06:40:55 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B625724169E for ; Thu, 24 Jul 2025 06:41:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339278; cv=none; b=TM0RNcCUjkX0tKaA66e/lT4PufDhAtiK1VOVUAbPFX8mZrbRkVAkDhaTeVu69FhMkCMeDIb5waJl76ODIesHlgmisidjOcsPf6/qSrCbrZ81dxNjcnSeqFQgQSNB0rUYKUyj+c4fIhexKoYFSSTKnyDRHBgwLXHdQ7gRa4Fqxqs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339278; c=relaxed/simple; bh=znMkjthQQqU3xlnHwN1/lZbxfygC3bedemTsvG3xTl4=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=VFFvzW4Yok15RJw6ZFSkDlXFhIRUtk1Yi3zfNaa0kT88w/e5ylVQ9mi//9BdCKQbDYOUbPS/HNlaiEWF8S3FlZZI8M/E5cec03gOo6Nk1MYXxOqClbJd3RvHoyhe8y/orhUVV+W1+qBOU/AJZ766Q44OBkTUIMcB3EFeuruMwFU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=gasq74qu; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="gasq74qu" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1753339275; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=alnGVuz23aU/lBX/KwG8+ef7d5gPo+i+QbZaVoGOCdU=; b=gasq74quxk1vlW1vHcuGK4JjTszcAa9laX3XTx7xXnTKA2LV59HVD3rr6jv0G2hl1acCRA qX3Xv/WEorWXVH0v8JAPFGFzc1ho2TUtPgX+QRoY2FSF9OUfBbe0EkBv6n4Ev/sAsuXriz ztMpHfMubd6dD+XIS2/mTW0mOIH6dPQ= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-144-fm8-yAK6MIiWUK8Lfl391Q-1; Thu, 24 Jul 2025 02:41:12 -0400 X-MC-Unique: fm8-yAK6MIiWUK8Lfl391Q-1 X-Mimecast-MFC-AGG-ID: fm8-yAK6MIiWUK8Lfl391Q_1753339272 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id F31D01800C2F; Thu, 24 Jul 2025 06:41:11 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.231]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id E4DF218002AF; Thu, 24 Jul 2025 06:41:08 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V4 11/19] virtio_ring: switch to use vring_virtqueue for detach_unused_buf variants Date: Thu, 24 Jul 2025 14:40:09 +0800 Message-ID: <20250724064017.26058-12-jasowang@redhat.com> In-Reply-To: <20250724064017.26058-1-jasowang@redhat.com> References: <20250724064017.26058-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Those variants are used internally so let's switch to use vring_virtqueue as parameter to be consistent with other internal virtqueue helpers. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 76375fa3379d..27d1239705bd 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -950,9 +950,8 @@ static bool virtqueue_enable_cb_delayed_split(struct vr= ing_virtqueue *vq) return true; } =20 -static void *virtqueue_detach_unused_buf_split(struct virtqueue *_vq) +static void *virtqueue_detach_unused_buf_split(struct vring_virtqueue *vq) { - struct vring_virtqueue *vq =3D to_vvq(_vq); unsigned int i; void *buf; =20 @@ -965,7 +964,7 @@ static void *virtqueue_detach_unused_buf_split(struct v= irtqueue *_vq) buf =3D vq->split.desc_state[i].data; detach_buf_split(vq, i, NULL); vq->split.avail_idx_shadow--; - vq->split.vring.avail->idx =3D cpu_to_virtio16(_vq->vdev, + vq->split.vring.avail->idx =3D cpu_to_virtio16(vq->vq.vdev, vq->split.avail_idx_shadow); END_USE(vq); return buf; @@ -1892,9 +1891,8 @@ static bool virtqueue_enable_cb_delayed_packed(struct= vring_virtqueue *vq) return true; } =20 -static void *virtqueue_detach_unused_buf_packed(struct virtqueue *_vq) +static void *virtqueue_detach_unused_buf_packed(struct vring_virtqueue *vq) { - struct vring_virtqueue *vq =3D to_vvq(_vq); unsigned int i; void *buf; =20 @@ -2646,8 +2644,8 @@ void *virtqueue_detach_unused_buf(struct virtqueue *_= vq) { struct vring_virtqueue *vq =3D to_vvq(_vq); =20 - return vq->packed_ring ? virtqueue_detach_unused_buf_packed(_vq) : - virtqueue_detach_unused_buf_split(_vq); + return vq->packed_ring ? virtqueue_detach_unused_buf_packed(vq) : + virtqueue_detach_unused_buf_split(vq); } EXPORT_SYMBOL_GPL(virtqueue_detach_unused_buf); =20 --=20 2.39.5 From nobody Mon Oct 6 06:40:55 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6C79F24169E for ; Thu, 24 Jul 2025 06:41:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339283; cv=none; b=TI7cF9d66VY5K3jW+ynOIs3WMbA8ySebdciK+lRkwlJcFh9xo33NL+p58Uv/KLPRYQB+zz68HMF6Y4h0tjGZ0iyT5Oyd/S/rjUO+H+nDB3dK1VfTB78q0d0I1ig3sd00vhSZinJcpTXIPQDFDvkDnrdFbxZ+ITzMNWALGIfgp2Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339283; c=relaxed/simple; bh=BrUp54+Nw89iAStDLuvi4fujbgr5ptH8J/GZDCWHtGI=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Sa1RYQDPNupwHAbM/1mNQPbWGUlWiwa9OejxZ9ucf/PrkB24evg0sMfkDliSGgg2kRZOIxKWOK59XRrkdY2fZyoc6HAxrq06V/xyviluyZCoGO0r3zxPmNdBNxtA7OtUaG18h3AezioiWORMQ6nmNNPeei7cP+FueYH9sUDa/Gc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=K1YILXqL; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="K1YILXqL" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1753339280; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yCK9EXlCNR+D1rOETqxNqaof4DwL+sBnRuTMJtuZIeg=; b=K1YILXqL+eDjpUwvWfmVCef7VtxZN7CGClyF/LbkeIL1knZXL4jhOKLh0cDDJW0eC0lJfA WjfVlE3JmQYJWEwn6btBC7k/g9f/26/yLDJwp3yGMDn839iUc8JtdUeb95FKaLt/BTZaek 5iPRfv8owD+pL+LB13UInZk+tSx6d24= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-599-1EJXPPRtO_ijHj0Le9bu-Q-1; Thu, 24 Jul 2025 02:41:17 -0400 X-MC-Unique: 1EJXPPRtO_ijHj0Le9bu-Q-1 X-Mimecast-MFC-AGG-ID: 1EJXPPRtO_ijHj0Le9bu-Q_1753339276 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 14D131800165; Thu, 24 Jul 2025 06:41:16 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.231]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id B602D180035E; Thu, 24 Jul 2025 06:41:12 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V4 12/19] virtio_ring: use u16 for last_used_idx in virtqueue_poll_split() Date: Thu, 24 Jul 2025 14:40:10 +0800 Message-ID: <20250724064017.26058-13-jasowang@redhat.com> In-Reply-To: <20250724064017.26058-1-jasowang@redhat.com> References: <20250724064017.26058-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Use u16 for last_used_idx in virtqueue_poll_split() to align with the spec. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 27d1239705bd..103e807249b6 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -801,7 +801,7 @@ static void detach_buf_split(struct vring_virtqueue *vq= , unsigned int head, } =20 static bool virtqueue_poll_split(const struct vring_virtqueue *vq, - unsigned int last_used_idx) + u16 last_used_idx) { return (u16)last_used_idx !=3D virtio16_to_cpu(vq->vq.vdev, vq->split.vring.used->idx); --=20 2.39.5 From nobody Mon Oct 6 06:40:55 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 29CF3247298 for ; Thu, 24 Jul 2025 06:41:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339288; cv=none; b=VYC+lwlhTpkop2C7hQ4Ks3eR3Z9QoQrGCaFiJvmS2TwzN+1QNDiMdhTx5MqCMfa4QmDDB1GL35I/9trJrKllhDTQh9FAI7SRc/UFtUx1+RqBKMhJNI0ojbmY7E9vJ+Mskb+/Dr+bJnOgcuT2b63Kkw6lyifoS5l5cXCmr5odohg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339288; c=relaxed/simple; bh=f6E8NPihnHZwv8Y3/vhn/WffdqwwWxUUse+KXT04YiM=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=E5TGUzXclCGll+VK1oM7EVXTt3e90PSbn5SHTqfYVo1wQQPRDNMJ2zux6ayKzfbwOuFWpcucPvSZqlLHIQmHd1fSrqujbR3Rg68pA7OiGt/kGx4sw1dvxCB1P9wCi3sIPL0nAaBkLMr5M3jWILdN4fnLozQHn6S8CoAGhV/qpTA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=QEsPfOoB; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="QEsPfOoB" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1753339285; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NBQfQPqanUIVgdz82yClH7kZ6uGY2ao6bQ0FmHu88sA=; b=QEsPfOoB2ReoIbsYjIf0bp1JRa2Lp8ISj3NsixVbUjSc5u16udn5hjwKnVGQIvy1FXwGVd tbQZTqTZVu9pP4piLOo45BsdTRuciSRGC45W9WCMeT0Q0vHexWVg8D4fCxPtVwER0Ro7tW F/uak8IY9hhDvxwEnMTMHQw5f8FnR8I= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-180-rTduFjAaPAq3a3R53pGGuQ-1; Thu, 24 Jul 2025 02:41:21 -0400 X-MC-Unique: rTduFjAaPAq3a3R53pGGuQ-1 X-Mimecast-MFC-AGG-ID: rTduFjAaPAq3a3R53pGGuQ_1753339280 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 48033195608F; Thu, 24 Jul 2025 06:41:20 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.231]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 152A51801712; Thu, 24 Jul 2025 06:41:16 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V4 13/19] virtio_ring: introduce virtqueue ops Date: Thu, 24 Jul 2025 14:40:11 +0800 Message-ID: <20250724064017.26058-14-jasowang@redhat.com> In-Reply-To: <20250724064017.26058-1-jasowang@redhat.com> References: <20250724064017.26058-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 This patch introduces virtqueue ops which is a set of the callbacks that will be called for different queue layout or features. This would help to avoid branches for split/packed and will ease the future implementation like in order. Note that in order to eliminate the indirect calls this patch uses global array of const ops to allow compiler to avoid indirect branches. Tested with CONFIG_MITIGATION_RETPOLINE, no performance differences were noticed. Acked-by: Eugenio P=C3=A9rez Suggested-by: Michael S. Tsirkin Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 173 ++++++++++++++++++++++++++--------- 1 file changed, 130 insertions(+), 43 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 103e807249b6..2f0b14940e05 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -67,6 +67,12 @@ #define LAST_ADD_TIME_INVALID(vq) #endif =20 +enum vq_layout { + SPLIT =3D 0, + PACKED, + VQ_TYPE_MAX, +}; + struct vring_desc_state_split { void *data; /* Data for callback. */ =20 @@ -159,12 +165,28 @@ struct vring_virtqueue_packed { size_t event_size_in_bytes; }; =20 +struct vring_virtqueue; + +struct virtqueue_ops { + int (*add)(struct vring_virtqueue *_vq, struct scatterlist *sgs[], + unsigned int total_sg, unsigned int out_sgs, + unsigned int in_sgs, void *data, + void *ctx, bool premapped, gfp_t gfp); + void *(*get)(struct vring_virtqueue *vq, unsigned int *len, void **ctx); + bool (*kick_prepare)(struct vring_virtqueue *vq); + void (*disable_cb)(struct vring_virtqueue *vq); + bool (*enable_cb_delayed)(struct vring_virtqueue *vq); + unsigned int (*enable_cb_prepare)(struct vring_virtqueue *vq); + bool (*poll)(const struct vring_virtqueue *vq, u16 last_used_idx); + void *(*detach_unused_buf)(struct vring_virtqueue *vq); + bool (*more_used)(const struct vring_virtqueue *vq); + int (*resize)(struct vring_virtqueue *vq, u32 num); + void (*reset)(struct vring_virtqueue *vq); +}; + struct vring_virtqueue { struct virtqueue vq; =20 - /* Is this a packed ring? */ - bool packed_ring; - /* Is DMA API used? */ bool use_dma_api; =20 @@ -180,6 +202,8 @@ struct vring_virtqueue { /* Host publishes avail event idx */ bool event; =20 + enum vq_layout layout; + /* Head of free buffer list. */ unsigned int free_head; /* Number we've added since last sync. */ @@ -232,6 +256,12 @@ static void vring_free(struct virtqueue *_vq); =20 #define to_vvq(_vq) container_of_const(_vq, struct vring_virtqueue, vq) =20 + +static inline bool virtqueue_is_packed(const struct vring_virtqueue *vq) +{ + return vq->layout =3D=3D PACKED; +} + static bool virtqueue_use_indirect(const struct vring_virtqueue *vq, unsigned int total_sg) { @@ -422,7 +452,7 @@ static void virtqueue_init(struct vring_virtqueue *vq, = u32 num) { vq->vq.num_free =3D num; =20 - if (vq->packed_ring) + if (virtqueue_is_packed(vq)) vq->last_used_idx =3D 0 | (1 << VRING_PACKED_EVENT_F_WRAP_CTR); else vq->last_used_idx =3D 0; @@ -1116,6 +1146,8 @@ static int vring_alloc_queue_split(struct vring_virtq= ueue_split *vring_split, return 0; } =20 +static const struct virtqueue_ops split_ops; + static struct virtqueue *__vring_new_virtqueue_split(unsigned int index, struct vring_virtqueue_split *vring_split, struct virtio_device *vdev, @@ -1133,7 +1165,7 @@ static struct virtqueue *__vring_new_virtqueue_split(= unsigned int index, if (!vq) return NULL; =20 - vq->packed_ring =3D false; + vq->layout =3D SPLIT; vq->vq.callback =3D callback; vq->vq.vdev =3D vdev; vq->vq.name =3D name; @@ -2076,6 +2108,8 @@ static void virtqueue_reset_packed(struct vring_virtq= ueue *vq) virtqueue_vring_init_packed(&vq->packed, !!vq->vq.callback); } =20 +static const struct virtqueue_ops packed_ops; + static struct virtqueue *__vring_new_virtqueue_packed(unsigned int index, struct vring_virtqueue_packed *vring_packed, struct virtio_device *vdev, @@ -2106,7 +2140,7 @@ static struct virtqueue *__vring_new_virtqueue_packed= (unsigned int index, #else vq->broken =3D false; #endif - vq->packed_ring =3D true; + vq->layout =3D PACKED; vq->dma_dev =3D dma_dev; vq->use_dma_api =3D vring_use_dma_api(vdev); =20 @@ -2194,6 +2228,39 @@ static int virtqueue_resize_packed(struct vring_virt= queue *vq, u32 num) return -ENOMEM; } =20 +static const struct virtqueue_ops split_ops =3D { + .add =3D virtqueue_add_split, + .get =3D virtqueue_get_buf_ctx_split, + .kick_prepare =3D virtqueue_kick_prepare_split, + .disable_cb =3D virtqueue_disable_cb_split, + .enable_cb_delayed =3D virtqueue_enable_cb_delayed_split, + .enable_cb_prepare =3D virtqueue_enable_cb_prepare_split, + .poll =3D virtqueue_poll_split, + .detach_unused_buf =3D virtqueue_detach_unused_buf_split, + .more_used =3D more_used_split, + .resize =3D virtqueue_resize_split, + .reset =3D virtqueue_reset_split, +}; + +static const struct virtqueue_ops packed_ops =3D { + .add =3D virtqueue_add_packed, + .get =3D virtqueue_get_buf_ctx_packed, + .kick_prepare =3D virtqueue_kick_prepare_packed, + .disable_cb =3D virtqueue_disable_cb_packed, + .enable_cb_delayed =3D virtqueue_enable_cb_delayed_packed, + .enable_cb_prepare =3D virtqueue_enable_cb_prepare_packed, + .poll =3D virtqueue_poll_packed, + .detach_unused_buf =3D virtqueue_detach_unused_buf_packed, + .more_used =3D more_used_packed, + .resize =3D virtqueue_resize_packed, + .reset =3D virtqueue_reset_packed, +}; + +static const struct virtqueue_ops *const all_ops[VQ_TYPE_MAX] =3D { + [SPLIT] =3D &split_ops, + [PACKED] =3D &packed_ops +}; + static int virtqueue_disable_and_recycle(struct virtqueue *_vq, void (*recycle)(struct virtqueue *vq, void *buf)) { @@ -2236,6 +2303,39 @@ static int virtqueue_enable_after_reset(struct virtq= ueue *_vq) * Generic functions and exported symbols. */ =20 +#define VIRTQUEUE_CALL(vq, op, ...) \ + ({ \ + typeof(all_ops[SPLIT]->op(vq, ##__VA_ARGS__)) ret; \ + \ + switch (vq->layout) { \ + case SPLIT: \ + ret =3D all_ops[SPLIT]->op(vq, ##__VA_ARGS__); \ + break; \ + case PACKED: \ + ret =3D all_ops[PACKED]->op(vq, ##__VA_ARGS__); \ + break; \ + default: \ + BUG(); \ + break; \ + } \ + ret; \ +}) + +#define VOID_VIRTQUEUE_CALL(vq, op, ...) \ + ({ \ + switch ((vq)->layout) { \ + case SPLIT: \ + all_ops[SPLIT]->op(vq, ##__VA_ARGS__); \ + break; \ + case PACKED: \ + all_ops[PACKED]->op(vq, ##__VA_ARGS__); \ + break; \ + default: \ + BUG(); \ + break; \ + } \ +}) + static inline int virtqueue_add(struct virtqueue *_vq, struct scatterlist *sgs[], unsigned int total_sg, @@ -2248,10 +2348,9 @@ static inline int virtqueue_add(struct virtqueue *_v= q, { struct vring_virtqueue *vq =3D to_vvq(_vq); =20 - return vq->packed_ring ? virtqueue_add_packed(vq, sgs, total_sg, - out_sgs, in_sgs, data, ctx, premapped, gfp) : - virtqueue_add_split(vq, sgs, total_sg, - out_sgs, in_sgs, data, ctx, premapped, gfp); + return VIRTQUEUE_CALL(vq, add, sgs, total_sg, + out_sgs, in_sgs, data, + ctx, premapped, gfp); } =20 /** @@ -2441,8 +2540,7 @@ bool virtqueue_kick_prepare(struct virtqueue *_vq) { struct vring_virtqueue *vq =3D to_vvq(_vq); =20 - return vq->packed_ring ? virtqueue_kick_prepare_packed(vq) : - virtqueue_kick_prepare_split(vq); + return VIRTQUEUE_CALL(vq, kick_prepare); } EXPORT_SYMBOL_GPL(virtqueue_kick_prepare); =20 @@ -2512,8 +2610,7 @@ void *virtqueue_get_buf_ctx(struct virtqueue *_vq, un= signed int *len, { struct vring_virtqueue *vq =3D to_vvq(_vq); =20 - return vq->packed_ring ? virtqueue_get_buf_ctx_packed(vq, len, ctx) : - virtqueue_get_buf_ctx_split(vq, len, ctx); + return VIRTQUEUE_CALL(vq, get, len, ctx); } EXPORT_SYMBOL_GPL(virtqueue_get_buf_ctx); =20 @@ -2535,10 +2632,7 @@ void virtqueue_disable_cb(struct virtqueue *_vq) { struct vring_virtqueue *vq =3D to_vvq(_vq); =20 - if (vq->packed_ring) - virtqueue_disable_cb_packed(vq); - else - virtqueue_disable_cb_split(vq); + VOID_VIRTQUEUE_CALL(vq, disable_cb); } EXPORT_SYMBOL_GPL(virtqueue_disable_cb); =20 @@ -2561,8 +2655,7 @@ unsigned int virtqueue_enable_cb_prepare(struct virtq= ueue *_vq) if (vq->event_triggered) vq->event_triggered =3D false; =20 - return vq->packed_ring ? virtqueue_enable_cb_prepare_packed(vq) : - virtqueue_enable_cb_prepare_split(vq); + return VIRTQUEUE_CALL(vq, enable_cb_prepare); } EXPORT_SYMBOL_GPL(virtqueue_enable_cb_prepare); =20 @@ -2583,8 +2676,8 @@ bool virtqueue_poll(struct virtqueue *_vq, unsigned i= nt last_used_idx) return false; =20 virtio_mb(vq->weak_barriers); - return vq->packed_ring ? virtqueue_poll_packed(vq, last_used_idx) : - virtqueue_poll_split(vq, last_used_idx); + + return VIRTQUEUE_CALL(vq, poll, last_used_idx); } EXPORT_SYMBOL_GPL(virtqueue_poll); =20 @@ -2627,8 +2720,7 @@ bool virtqueue_enable_cb_delayed(struct virtqueue *_v= q) if (vq->event_triggered) data_race(vq->event_triggered =3D false); =20 - return vq->packed_ring ? virtqueue_enable_cb_delayed_packed(vq) : - virtqueue_enable_cb_delayed_split(vq); + return VIRTQUEUE_CALL(vq, enable_cb_delayed); } EXPORT_SYMBOL_GPL(virtqueue_enable_cb_delayed); =20 @@ -2644,14 +2736,13 @@ void *virtqueue_detach_unused_buf(struct virtqueue = *_vq) { struct vring_virtqueue *vq =3D to_vvq(_vq); =20 - return vq->packed_ring ? virtqueue_detach_unused_buf_packed(vq) : - virtqueue_detach_unused_buf_split(vq); + return VIRTQUEUE_CALL(vq, detach_unused_buf); } EXPORT_SYMBOL_GPL(virtqueue_detach_unused_buf); =20 static inline bool more_used(const struct vring_virtqueue *vq) { - return vq->packed_ring ? more_used_packed(vq) : more_used_split(vq); + return VIRTQUEUE_CALL(vq, more_used); } =20 /** @@ -2780,7 +2871,8 @@ int virtqueue_resize(struct virtqueue *_vq, u32 num, if (!num) return -EINVAL; =20 - if ((vq->packed_ring ? vq->packed.vring.num : vq->split.vring.num) =3D=3D= num) + if ((virtqueue_is_packed(vq) ? vq->packed.vring.num : + vq->split.vring.num) =3D=3D num) return 0; =20 err =3D virtqueue_disable_and_recycle(_vq, recycle); @@ -2789,10 +2881,7 @@ int virtqueue_resize(struct virtqueue *_vq, u32 num, if (recycle_done) recycle_done(_vq); =20 - if (vq->packed_ring) - err =3D virtqueue_resize_packed(vq, num); - else - err =3D virtqueue_resize_split(vq, num); + err =3D VIRTQUEUE_CALL(vq, resize, num); =20 err_reset =3D virtqueue_enable_after_reset(_vq); if (err_reset) @@ -2830,10 +2919,7 @@ int virtqueue_reset(struct virtqueue *_vq, if (recycle_done) recycle_done(_vq); =20 - if (vq->packed_ring) - virtqueue_reset_packed(vq); - else - virtqueue_reset_split(vq); + VOID_VIRTQUEUE_CALL(vq, reset); =20 return virtqueue_enable_after_reset(_vq); } @@ -2875,7 +2961,7 @@ static void vring_free(struct virtqueue *_vq) struct vring_virtqueue *vq =3D to_vvq(_vq); =20 if (vq->we_own_ring) { - if (vq->packed_ring) { + if (virtqueue_is_packed(vq)) { vring_free_queue(vq->vq.vdev, vq->packed.ring_size_in_bytes, vq->packed.vring.desc, @@ -2904,7 +2990,7 @@ static void vring_free(struct virtqueue *_vq) vring_dma_dev(vq)); } } - if (!vq->packed_ring) { + if (!virtqueue_is_packed(vq)) { kfree(vq->split.desc_state); kfree(vq->split.desc_extra); } @@ -2929,7 +3015,7 @@ u32 vring_notification_data(struct virtqueue *_vq) struct vring_virtqueue *vq =3D to_vvq(_vq); u16 next; =20 - if (vq->packed_ring) + if (virtqueue_is_packed(vq)) next =3D (vq->packed.next_avail_idx & ~(-(1 << VRING_PACKED_EVENT_F_WRAP_CTR))) | vq->packed.avail_wrap_counter << @@ -2982,7 +3068,8 @@ unsigned int virtqueue_get_vring_size(const struct vi= rtqueue *_vq) =20 const struct vring_virtqueue *vq =3D to_vvq(_vq); =20 - return vq->packed_ring ? vq->packed.vring.num : vq->split.vring.num; + return virtqueue_is_packed(vq) ? vq->packed.vring.num : + vq->split.vring.num; } EXPORT_SYMBOL_GPL(virtqueue_get_vring_size); =20 @@ -3065,7 +3152,7 @@ dma_addr_t virtqueue_get_desc_addr(const struct virtq= ueue *_vq) =20 BUG_ON(!vq->we_own_ring); =20 - if (vq->packed_ring) + if (virtqueue_is_packed(vq)) return vq->packed.ring_dma_addr; =20 return vq->split.queue_dma_addr; @@ -3078,7 +3165,7 @@ dma_addr_t virtqueue_get_avail_addr(const struct virt= queue *_vq) =20 BUG_ON(!vq->we_own_ring); =20 - if (vq->packed_ring) + if (virtqueue_is_packed(vq)) return vq->packed.driver_event_dma_addr; =20 return vq->split.queue_dma_addr + @@ -3092,7 +3179,7 @@ dma_addr_t virtqueue_get_used_addr(const struct virtq= ueue *_vq) =20 BUG_ON(!vq->we_own_ring); =20 - if (vq->packed_ring) + if (virtqueue_is_packed(vq)) return vq->packed.device_event_dma_addr; =20 return vq->split.queue_dma_addr + --=20 2.39.5 From nobody Mon Oct 6 06:40:55 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5460722FAF4 for ; Thu, 24 Jul 2025 06:41:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339290; cv=none; b=CCv0IySxSH4InJH2C9CpJigSV2G32nwUPySqPnRzABw7wGtWq/ywk+xUqNUzfl2/ybOPvTz9Q0lzkxxbuCY+y1jg9zRqEp2EL6ICOjVVnu6/vneY7HJmWRhalF+lETg3Vw09nj7XG3GEN5HxUKHPfDII0dUdrV2gbziUG8spUqk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339290; c=relaxed/simple; bh=CeiTqtpT3BCdJoDDAqMKZFHC7DJJaHTsQtRK8FEcu70=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=QUJu0DDGDJzrrfK/0NzsFfQKqfxLxYHgIJLjh4XXBvRotfKhtUst1ukET1FuEc2cW0viatUQsguCcCsmvEKgViwmobDWKQQkmjwi9Tp6FYUsP+ZMxEm50iIhDZ5xLr9TGxvTXu12CPBtVYTLYSeLBihe9kmI6BBkYtzxFdTIp9A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=iY3rW/5c; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="iY3rW/5c" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1753339287; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qc6FYNQBKIB+acS4K5VrBDZprHD+upxJfELmeFU0+t4=; b=iY3rW/5c82Wp2337cByulnTkQsDUfst76r3OUg0rYDZj6wVEcGnpA4wR5M51ipy1VM9Ws8 i8GOULLxIRz1dOxxW+9bEM4RyOT0Vx7/UHI1+al0zMJ3WSQQdB6TlzdrLVfjRSvT5Rc01K sCViA1/wZiA9gqmVNc7xx9dAN38rQXo= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-14-pKXsCnHKNHu9qGqTFX86ww-1; Thu, 24 Jul 2025 02:41:25 -0400 X-MC-Unique: pKXsCnHKNHu9qGqTFX86ww-1 X-Mimecast-MFC-AGG-ID: pKXsCnHKNHu9qGqTFX86ww_1753339284 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 2ACA91800294; Thu, 24 Jul 2025 06:41:24 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.231]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 070B5180035E; Thu, 24 Jul 2025 06:41:20 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V4 14/19] virtio_ring: determine descriptor flags at one time Date: Thu, 24 Jul 2025 14:40:12 +0800 Message-ID: <20250724064017.26058-15-jasowang@redhat.com> In-Reply-To: <20250724064017.26058-1-jasowang@redhat.com> References: <20250724064017.26058-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Let's determine the last descriptor by counting the number of sg. This would be consistent with packed virtqueue implementation and ease the future in-order implementation. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 2f0b14940e05..01bbbcd65417 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -570,7 +570,7 @@ static inline int virtqueue_add_split(struct vring_virt= queue *vq, struct vring_desc_extra *extra; struct scatterlist *sg; struct vring_desc *desc; - unsigned int i, n, avail, descs_used, prev, err_idx; + unsigned int i, n, avail, descs_used, err_idx, c =3D 0; int head; bool indirect; =20 @@ -627,6 +627,7 @@ static inline int virtqueue_add_split(struct vring_virt= queue *vq, } =20 for (n =3D 0; n < out_sgs; n++) { + sg =3D sgs[n]; for (sg =3D sgs[n]; sg; sg =3D sg_next(sg)) { dma_addr_t addr; u32 len; @@ -634,12 +635,12 @@ static inline int virtqueue_add_split(struct vring_vi= rtqueue *vq, if (vring_map_one_sg(vq, sg, DMA_TO_DEVICE, &addr, &len, premapped)) goto unmap_release; =20 - prev =3D i; /* Note that we trust indirect descriptor * table since it use stream DMA mapping. */ i =3D virtqueue_add_desc_split(vq, desc, extra, i, addr, len, - VRING_DESC_F_NEXT, + ++c =3D=3D total_sg ? + 0 : VRING_DESC_F_NEXT, premapped); } } @@ -651,21 +652,15 @@ static inline int virtqueue_add_split(struct vring_vi= rtqueue *vq, if (vring_map_one_sg(vq, sg, DMA_FROM_DEVICE, &addr, &len, premapped)) goto unmap_release; =20 - prev =3D i; /* Note that we trust indirect descriptor * table since it use stream DMA mapping. */ - i =3D virtqueue_add_desc_split(vq, desc, extra, i, addr, len, - VRING_DESC_F_NEXT | - VRING_DESC_F_WRITE, - premapped); + i =3D virtqueue_add_desc_split(vq, desc, extra, + i, addr, len, + (++c =3D=3D total_sg ? 0 : VRING_DESC_F_NEXT) | + VRING_DESC_F_WRITE, premapped); } } - /* Last one doesn't continue. */ - desc[prev].flags &=3D cpu_to_virtio16(vq->vq.vdev, ~VRING_DESC_F_NEXT); - if (!indirect && vring_need_unmap_buffer(vq, &extra[prev])) - vq->split.desc_extra[prev & (vq->split.vring.num - 1)].flags &=3D - ~VRING_DESC_F_NEXT; =20 if (indirect) { /* Now that the indirect table is filled in, map it. */ --=20 2.39.5 From nobody Mon Oct 6 06:40:55 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4D97B24DD11 for ; Thu, 24 Jul 2025 06:41:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339296; cv=none; b=sg/pO0lKISnP9V0KEoH20GXrr6aYrhUBqD4lDLPDkg3j40aOo2B1KLovbuTcfIs8306KFMuuDZmKqzPnGzMdY7qEO6piHeaG2iiJnqsL/bnAfURgXpWHNgBbQNVOMy0fqIBKEpSZHYzbQ65+p5r2Hb6QhFwa6UQd0MsjTkio5Uo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339296; c=relaxed/simple; bh=h2Qp87PI0h1xjsXyVE7VBnmnVfNlPemFMuexCczlqIk=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=hh341PEdWr97ovSQ/36XyLQZBJ62gxkfuLS6Bh/swVrpYv8t4yKUwBYvFqxD+rpaAfF2W8hXzr3Vzo13ctfb08taUkDV43IkuXTbsEAm+bCbaws0qWo3MPkTmsi6vYO6GY6YNVOQthdYXfyeXnoQ/YAoARXrfCMitamF8UKeOyM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=guBW8RDc; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="guBW8RDc" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1753339292; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uLj4vQ76k5rdjrHpOoONiDq/brYcC+kkPUOFiNuUnfI=; b=guBW8RDceqKVvN5ixeyV+x+ppZWO9k1L9aweZjckt0ivhtYTexl9OTidBuFi8+QgOhDFIL HXz8cRw63eNzv0GM7H3s+jxKQ4VcFMqJHFc3yX1Og3gm9xAKtndI3WQaV8vciz75IqTkSp wpnsZPq+oBhGEUEAuHcQqm2fFZ0Mrbk= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-298-BF6AXSlwPv-Pkf0uhWb2-A-1; Thu, 24 Jul 2025 02:41:29 -0400 X-MC-Unique: BF6AXSlwPv-Pkf0uhWb2-A-1 X-Mimecast-MFC-AGG-ID: BF6AXSlwPv-Pkf0uhWb2-A_1753339288 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 28EF5180028C; Thu, 24 Jul 2025 06:41:28 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.231]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id EE40F180035E; Thu, 24 Jul 2025 06:41:24 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V4 15/19] virtio_ring: factor out core logic of buffer detaching Date: Thu, 24 Jul 2025 14:40:13 +0800 Message-ID: <20250724064017.26058-16-jasowang@redhat.com> In-Reply-To: <20250724064017.26058-1-jasowang@redhat.com> References: <20250724064017.26058-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Factor out core logic of buffer detaching and leave the id population to the caller so in order can just call the core logic. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 01bbbcd65417..b0eb68efb535 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -1663,8 +1663,8 @@ static bool virtqueue_kick_prepare_packed(struct vrin= g_virtqueue *vq) return needs_kick; } =20 -static void detach_buf_packed(struct vring_virtqueue *vq, - unsigned int id, void **ctx) +static void detach_buf_packed_in_order(struct vring_virtqueue *vq, + unsigned int id, void **ctx) { struct vring_desc_state_packed *state =3D NULL; struct vring_packed_desc *desc; @@ -1675,8 +1675,6 @@ static void detach_buf_packed(struct vring_virtqueue = *vq, /* Clear data ptr. */ state->data =3D NULL; =20 - vq->packed.desc_extra[state->last].next =3D vq->free_head; - vq->free_head =3D id; vq->vq.num_free +=3D state->num; =20 if (unlikely(vq->use_dma_api)) { @@ -1713,6 +1711,17 @@ static void detach_buf_packed(struct vring_virtqueue= *vq, } } =20 +static void detach_buf_packed(struct vring_virtqueue *vq, + unsigned int id, void **ctx) +{ + struct vring_desc_state_packed *state =3D &vq->packed.desc_state[id]; + + vq->packed.desc_extra[state->last].next =3D vq->free_head; + vq->free_head =3D id; + + return detach_buf_packed_in_order(vq, id, ctx); +} + static inline bool is_used_desc_packed(const struct vring_virtqueue *vq, u16 idx, bool used_wrap_counter) { --=20 2.39.5 From nobody Mon Oct 6 06:40:55 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6C0CC231825 for ; Thu, 24 Jul 2025 06:41:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339299; cv=none; b=VuMj72eYYievmKqc0F/Kn8p37Tg7BqayBm7l7GH8ecUJlfgoeKmJN3cYeSCNq7y1ulAf8zO9H0hpi0qIjUeTH+/TytHJLSpi33ZKk4P6kLDIvRf3y6UFomzCKvwG1tQTw+3bv9Sv12wS8dQJDUGahXDM8qk7ILgfF+wkCz21mu4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339299; c=relaxed/simple; bh=1lcDz66T2LOOPaKmYda3h5h/WU+JB/ib0C6rjWve8X8=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=DEVzZyeMe/treUqoMdABQLCDzKF7istEFd80AS9PWj7p8n8Cf57eMjRPDiMde94PuFP9BtVujFIFMAbhVo4v99KJuYQ9vcGbErAeSMjgtf8NWWpxy20SLNuH9LRPH44oRGoYL93KENOwfDyszSkzmWRc5To2ZF4wcreES1pNHT0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=PTjH7fRS; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="PTjH7fRS" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1753339296; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=blHJ84TlmhiM7nfcBxEl9nONPuNo8diNGupJxM7h4DA=; b=PTjH7fRSETdzJAFFMOzzUXm6gWVN1/ARH4uHjlQKOpx+watdF2ej0XuVNVeZxrbDL6akAY 7BdjuOJ0UYzKVq2PmkWY7+ZF0jlDfJpD2W92dm0GKlpVGM31GIr5FwrRmGm94axZRAiayd XLf1tUgX919M5tSFAKHvYBKOhf/O2DA= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-482-63-_I0ixN3iYCuo9H9KIzw-1; Thu, 24 Jul 2025 02:41:33 -0400 X-MC-Unique: 63-_I0ixN3iYCuo9H9KIzw-1 X-Mimecast-MFC-AGG-ID: 63-_I0ixN3iYCuo9H9KIzw_1753339292 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 12CB819560B5; Thu, 24 Jul 2025 06:41:32 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.231]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id EC1E218002AF; Thu, 24 Jul 2025 06:41:28 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V4 16/19] virtio_ring: factor out core logic for updating last_used_idx Date: Thu, 24 Jul 2025 14:40:14 +0800 Message-ID: <20250724064017.26058-17-jasowang@redhat.com> In-Reply-To: <20250724064017.26058-1-jasowang@redhat.com> References: <20250724064017.26058-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Factor out the core logic for updating last_used_idx to be reused by the packed in order implementation. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 43 +++++++++++++++++++++--------------- 1 file changed, 25 insertions(+), 18 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index b0eb68efb535..271508203443 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -1751,6 +1751,30 @@ static bool more_used_packed(const struct vring_virt= queue *vq) return virtqueue_poll_packed(vq, READ_ONCE(vq->last_used_idx)); } =20 +static void update_last_used_idx_packed(struct vring_virtqueue *vq, + u16 id, u16 last_used, + u16 used_wrap_counter) +{ + last_used +=3D vq->packed.desc_state[id].num; + if (unlikely(last_used >=3D vq->packed.vring.num)) { + last_used -=3D vq->packed.vring.num; + used_wrap_counter ^=3D 1; + } + + last_used =3D (last_used | (used_wrap_counter << VRING_PACKED_EVENT_F_WRA= P_CTR)); + WRITE_ONCE(vq->last_used_idx, last_used); + + /* + * If we expect an interrupt for the next entry, tell host + * by writing event index and flush out the write before + * the read in the next get_buf call. + */ + if (vq->packed.event_flags_shadow =3D=3D VRING_PACKED_EVENT_FLAG_DESC) + virtio_store_mb(vq->weak_barriers, + &vq->packed.vring.driver->off_wrap, + cpu_to_le16(vq->last_used_idx)); +} + static void *virtqueue_get_buf_ctx_packed(struct vring_virtqueue *vq, unsigned int *len, void **ctx) @@ -1794,24 +1818,7 @@ static void *virtqueue_get_buf_ctx_packed(struct vri= ng_virtqueue *vq, ret =3D vq->packed.desc_state[id].data; detach_buf_packed(vq, id, ctx); =20 - last_used +=3D vq->packed.desc_state[id].num; - if (unlikely(last_used >=3D vq->packed.vring.num)) { - last_used -=3D vq->packed.vring.num; - used_wrap_counter ^=3D 1; - } - - last_used =3D (last_used | (used_wrap_counter << VRING_PACKED_EVENT_F_WRA= P_CTR)); - WRITE_ONCE(vq->last_used_idx, last_used); - - /* - * If we expect an interrupt for the next entry, tell host - * by writing event index and flush out the write before - * the read in the next get_buf call. - */ - if (vq->packed.event_flags_shadow =3D=3D VRING_PACKED_EVENT_FLAG_DESC) - virtio_store_mb(vq->weak_barriers, - &vq->packed.vring.driver->off_wrap, - cpu_to_le16(vq->last_used_idx)); + update_last_used_idx_packed(vq, id, last_used, used_wrap_counter); =20 LAST_ADD_TIME_INVALID(vq); =20 --=20 2.39.5 From nobody Mon Oct 6 06:40:55 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 89A332528E1 for ; Thu, 24 Jul 2025 06:41:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339303; cv=none; b=DP6wJ8CaqFFUNuJtOLmvtznDCLYzElLcGYKbqCQd4TQPqKcDS8YqlU78Rh8PBuDEiaNTQvajFP+qWLnko8pIFLVfAvGF5Rdc+Sp0vNeGBkECAfI8nKhsnnucK8BDWhsTtfThzOrFe/jEPV6PXGjaXk2lxSSIG+yJ4fx3/WVfyno= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339303; c=relaxed/simple; bh=dLizkOvbLZ0mXHp9i9D8E8wkmqtzwgQr/6qBHnhNpsY=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=KCp59QiCuJO+dOSPxNSprRW0w7K22+9KQaqb791QWFXSx2uwaWJ7gaIOvSjJW6HGnYKF7u2YBi3f11+ppNEbih9zaMWqZXUNBXeYCX5I4wgx80WNO2pTcc4uBj3BEaJm8QQPHx6BEVF5mMdQzVjPzxNQ+0EeoAXPctegKp0+cTY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=VbncWMY8; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="VbncWMY8" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1753339300; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JC1BBeyGidUOJtM3T/uuRMFcZBP3h9yiKcbrcYifepA=; b=VbncWMY8IkbXksMgI95lK33CPRehZwdRz282OYCgE8Ww3qgf2GaEi6i2bsT8GxatW54Cv9 eEy7za1oiyUWyVI8pvMRpLpd0eYhW6extwUTmIZBO0xNybARBwm+gOk5836EB1QIsua8CD vZ/j7LJ8VcawrWQ+SaCSGJ3Qxhe81+U= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-204-FuHJxoEHPgerZvV63UMNZA-1; Thu, 24 Jul 2025 02:41:36 -0400 X-MC-Unique: FuHJxoEHPgerZvV63UMNZA-1 X-Mimecast-MFC-AGG-ID: FuHJxoEHPgerZvV63UMNZA_1753339296 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id F37CE1956096; Thu, 24 Jul 2025 06:41:35 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.231]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id E4A0518016F9; Thu, 24 Jul 2025 06:41:32 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V4 17/19] virtio_ring: factor out split indirect detaching logic Date: Thu, 24 Jul 2025 14:40:15 +0800 Message-ID: <20250724064017.26058-18-jasowang@redhat.com> In-Reply-To: <20250724064017.26058-1-jasowang@redhat.com> References: <20250724064017.26058-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Factor out the split indirect descriptor detaching logic in order to make it be reused by the in order support. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 63 ++++++++++++++++++++---------------- 1 file changed, 35 insertions(+), 28 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 271508203443..b1f99f8f05e6 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -767,11 +767,42 @@ static bool virtqueue_kick_prepare_split(struct vring= _virtqueue *vq) return needs_kick; } =20 +static void detach_indirect_split(struct vring_virtqueue *vq, + unsigned int head) +{ + struct vring_desc_extra *extra =3D vq->split.desc_extra; + struct vring_desc *indir_desc =3D + vq->split.desc_state[head].indir_desc; + unsigned int j; + u32 len, num; + + /* Free the indirect table, if any, now that it's unmapped. */ + if (!indir_desc) + return; + len =3D vq->split.desc_extra[head].len; + + BUG_ON(!(vq->split.desc_extra[head].flags & + VRING_DESC_F_INDIRECT)); + BUG_ON(len =3D=3D 0 || len % sizeof(struct vring_desc)); + + num =3D len / sizeof(struct vring_desc); + + extra =3D (struct vring_desc_extra *)&indir_desc[num]; + + if (vq->use_dma_api) { + for (j =3D 0; j < num; j++) + vring_unmap_one_split(vq, &extra[j]); + } + + kfree(indir_desc); + vq->split.desc_state[head].indir_desc =3D NULL; +} + static void detach_buf_split(struct vring_virtqueue *vq, unsigned int head, void **ctx) { struct vring_desc_extra *extra; - unsigned int i, j; + unsigned int i; __virtio16 nextflag =3D cpu_to_virtio16(vq->vq.vdev, VRING_DESC_F_NEXT); =20 /* Clear data ptr. */ @@ -795,34 +826,10 @@ static void detach_buf_split(struct vring_virtqueue *= vq, unsigned int head, /* Plus final descriptor */ vq->vq.num_free++; =20 - if (vq->indirect) { - struct vring_desc *indir_desc =3D - vq->split.desc_state[head].indir_desc; - u32 len, num; - - /* Free the indirect table, if any, now that it's unmapped. */ - if (!indir_desc) - return; - len =3D vq->split.desc_extra[head].len; - - BUG_ON(!(vq->split.desc_extra[head].flags & - VRING_DESC_F_INDIRECT)); - BUG_ON(len =3D=3D 0 || len % sizeof(struct vring_desc)); - - num =3D len / sizeof(struct vring_desc); - - extra =3D (struct vring_desc_extra *)&indir_desc[num]; - - if (vq->use_dma_api) { - for (j =3D 0; j < num; j++) - vring_unmap_one_split(vq, &extra[j]); - } - - kfree(indir_desc); - vq->split.desc_state[head].indir_desc =3D NULL; - } else if (ctx) { + if (vq->indirect) + detach_indirect_split(vq, head); + else if (ctx) *ctx =3D vq->split.desc_state[head].indir_desc; - } } =20 static bool virtqueue_poll_split(const struct vring_virtqueue *vq, --=20 2.39.5 From nobody Mon Oct 6 06:40:55 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9462F253358 for ; Thu, 24 Jul 2025 06:41:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339305; cv=none; b=YF3g3C1Ia8w7qPYm+p1p5FLHxprrX2xGjxOSURCQf9MzwSa4Oq5gbO0aGk1TiD0DVahwmqEw7wKbUHMYmEUyyad12thb/2UvJixfuJFtT8oskY/SYCXK2yuaARVq7+F6DcxHdBd6b3qg+l+47GjiCUdmfCAwh3FvIY9njNuWXWI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339305; c=relaxed/simple; bh=Y/Dq2JaqvInWtobs4SIRiJ7Mq6QDiDuMa+IvLXLhY48=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=QxJf79/1PSvLBq0cTBIMX+ExxlxbRvYQn5WM+j70ZXU3tDK7yQUZTdPZ9nCBaBiOqnNlgUvP2wtq/gxcxJjw0YA4iIGWiezEd9h8I4bHaubASSx9OGrfFAYcRgXsLQYgprcPkChwt005F4Z73z11MZzM5kZqvciVmg3BTBuxva4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Tc8OMyqH; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Tc8OMyqH" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1753339302; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SpNQ0ibc27fat+Wmy9GL9edMrZ/wk2mEn1uvqmmwjA0=; b=Tc8OMyqHoIOJyeqC0MNo0WXHsDo5kDnyjRGIosvsr/yoanOeaiTjNbfwHzEPL6mhYp1fQs OHAvzhWDwbX/D5p4Y8HjXKwFEE6E5fDHgo88IRV+jYiCvosIDGENFbt2S9HAqIh7CkpiBY fAku3Cc1tLhTtApdVkvDYG0RCZWLHJQ= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-628-IhXiD3S1NDu-5tPbyFAEqA-1; Thu, 24 Jul 2025 02:41:41 -0400 X-MC-Unique: IhXiD3S1NDu-5tPbyFAEqA-1 X-Mimecast-MFC-AGG-ID: IhXiD3S1NDu-5tPbyFAEqA_1753339300 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 1C38E1800C38; Thu, 24 Jul 2025 06:41:40 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.231]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id D611B180035E; Thu, 24 Jul 2025 06:41:36 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V4 18/19] virtio_ring: factor out split detaching logic Date: Thu, 24 Jul 2025 14:40:16 +0800 Message-ID: <20250724064017.26058-19-jasowang@redhat.com> In-Reply-To: <20250724064017.26058-1-jasowang@redhat.com> References: <20250724064017.26058-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 This patch factors out the split core detaching logic that could be reused by in order feature into a dedicated function. Acked-by: Eugenio P=C3=A9rez Reviewed-by: Xuan Zhuo Signed-off-by: Jason Wang --- drivers/virtio/virtio_ring.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index b1f99f8f05e6..e675d8305dbf 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -798,8 +798,9 @@ static void detach_indirect_split(struct vring_virtqueu= e *vq, vq->split.desc_state[head].indir_desc =3D NULL; } =20 -static void detach_buf_split(struct vring_virtqueue *vq, unsigned int head, - void **ctx) +static unsigned detach_buf_split_in_order(struct vring_virtqueue *vq, + unsigned int head, + void **ctx) { struct vring_desc_extra *extra; unsigned int i; @@ -820,8 +821,6 @@ static void detach_buf_split(struct vring_virtqueue *vq= , unsigned int head, } =20 vring_unmap_one_split(vq, &extra[i]); - vq->split.desc_extra[i].next =3D vq->free_head; - vq->free_head =3D head; =20 /* Plus final descriptor */ vq->vq.num_free++; @@ -830,6 +829,17 @@ static void detach_buf_split(struct vring_virtqueue *v= q, unsigned int head, detach_indirect_split(vq, head); else if (ctx) *ctx =3D vq->split.desc_state[head].indir_desc; + + return i; +} + +static void detach_buf_split(struct vring_virtqueue *vq, unsigned int head, + void **ctx) +{ + unsigned int i =3D detach_buf_split_in_order(vq, head, ctx); + + vq->split.desc_extra[i].next =3D vq->free_head; + vq->free_head =3D head; } =20 static bool virtqueue_poll_split(const struct vring_virtqueue *vq, --=20 2.39.5 From nobody Mon Oct 6 06:40:55 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D23D2223DEF for ; Thu, 24 Jul 2025 06:41:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339312; cv=none; b=ApXgjEVzK0ZPIvdX7lFPyT29l9ldrkaSUmb+PaDxJRH/2gBUpKfpJRk2YYZRyFiM1Ouf2+M7bv+f4gFUIau8IRN0AJ/pxUBt85GWO2174aFxrPYuPBAqBv7a0seK/HHb50clWfR9R1dFHLnZlI4V9icclqzmobmsOJ2ESeS0l+I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753339312; c=relaxed/simple; bh=ZQHvErjoXrtkP8xXyueqiJajUwyaRHAsSF1QuxGe47s=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=VseTPyhKn2xb0mNlAfAx0xEfCwe261T1Tf0qutxaovUcNCfyU6ewHbBStM6sF+XzeKEv3BC3aIGCBisvhArmQroUomEN5yZpsbhrRnco7BPOcwbbvndk8q/TIWsvUp8akNyTT0L+ObuUrNuHyFMi0gq4TaDS0uZts3JvWO5sdOs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Dl93hK5B; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Dl93hK5B" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1753339309; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QeuvPqn22m6FznFbgF5dl1VeqYk03ekyNy8Uwlz8o8M=; b=Dl93hK5BDwm3PZEuCLK1I/utZVPV8y2V0Oq84RYELTOQpmpKq5NezEgNx2tfCDzE3z9hPD BIZPBj53r+JJp1/M2mTTBmeGlPpXsisE52E40ZN9PgUpLRJ+FWgzZSjMSCPhNBjWOGKxKA kD7urucgionl3UBcuuomhmWhlvUcccs= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-357-kstiuTsYMxmwp0x3MHVFGQ-1; Thu, 24 Jul 2025 02:41:44 -0400 X-MC-Unique: kstiuTsYMxmwp0x3MHVFGQ-1 X-Mimecast-MFC-AGG-ID: kstiuTsYMxmwp0x3MHVFGQ_1753339304 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id F15B31800297; Thu, 24 Jul 2025 06:41:43 +0000 (UTC) Received: from localhost.localdomain (unknown [10.72.112.231]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id E1A02180035E; Thu, 24 Jul 2025 06:41:40 +0000 (UTC) From: Jason Wang To: mst@redhat.com, jasowang@redhat.com, xuanzhuo@linux.alibaba.com, eperezma@redhat.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: [PATCH V4 19/19] virtio_ring: add in order support Date: Thu, 24 Jul 2025 14:40:17 +0800 Message-ID: <20250724064017.26058-20-jasowang@redhat.com> In-Reply-To: <20250724064017.26058-1-jasowang@redhat.com> References: <20250724064017.26058-1-jasowang@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Content-Type: text/plain; charset="utf-8" This patch implements in order support for both split virtqueue and packed virtqueue. Perfomance could be gained for the device where the memory access could be expensive (e.g vhost-net or a real PCI device): Benchmark with KVM guest: Vhost-net on the host: (pktgen + XDP_DROP): in_order=3Doff | in_order=3Don | +% TX: 5.20Mpps | 6.20Mpps | +19% RX: 3.47Mpps | 3.61Mpps | + 4% Vhost-user(testpmd) on the host: (pktgen/XDP_DROP): For split virtqueue: in_order=3Doff | in_order=3Don | +% TX: 5.60Mpps | 5.60Mpps | +0.0% RX: 9.16Mpps | 9.61Mpps | +4.9% For packed virtqueue: in_order=3Doff | in_order=3Don | +% TX: 5.60Mpps | 5.70Mpps | +1.7% RX: 10.6Mpps | 10.8Mpps | +1.8% Benchmark also shows no performance impact for in_order=3Doff for queue size with 256 and 1024. Signed-off-by: Jason Wang Acked-by: Eugenio P=C3=A9rez --- drivers/virtio/virtio_ring.c | 436 +++++++++++++++++++++++++++++++++-- 1 file changed, 414 insertions(+), 22 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index e675d8305dbf..8f9413acd4e2 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -70,11 +70,14 @@ enum vq_layout { SPLIT =3D 0, PACKED, + SPLIT_IN_ORDER, + PACKED_IN_ORDER, VQ_TYPE_MAX, }; =20 struct vring_desc_state_split { void *data; /* Data for callback. */ + u32 total_len; /* Buffer Length */ =20 /* Indirect desc table and extra table, if any. These two will be * allocated together. So we won't stress more to the memory allocator. @@ -84,6 +87,7 @@ struct vring_desc_state_split { =20 struct vring_desc_state_packed { void *data; /* Data for callback. */ + u32 total_len; /* Buffer Length */ =20 /* Indirect desc table and extra table, if any. These two will be * allocated together. So we won't stress more to the memory allocator. @@ -206,6 +210,17 @@ struct vring_virtqueue { =20 /* Head of free buffer list. */ unsigned int free_head; + + /* + * With IN_ORDER, devices write a single used ring entry with + * the id corresponding to the head entry of the descriptor chain + * describing the last buffer in the batch + */ + struct used_entry { + u32 id; + u32 len; + } batch_last; + /* Number we've added since last sync. */ unsigned int num_added; =20 @@ -259,7 +274,12 @@ static void vring_free(struct virtqueue *_vq); =20 static inline bool virtqueue_is_packed(const struct vring_virtqueue *vq) { - return vq->layout =3D=3D PACKED; + return vq->layout =3D=3D PACKED || vq->layout =3D=3D PACKED_IN_ORDER; +} + +static inline bool virtqueue_is_in_order(const struct vring_virtqueue *vq) +{ + return vq->layout =3D=3D SPLIT_IN_ORDER || vq->layout =3D=3D PACKED_IN_OR= DER; } =20 static bool virtqueue_use_indirect(const struct vring_virtqueue *vq, @@ -571,6 +591,8 @@ static inline int virtqueue_add_split(struct vring_virt= queue *vq, struct scatterlist *sg; struct vring_desc *desc; unsigned int i, n, avail, descs_used, err_idx, c =3D 0; + /* Total length for in-order */ + unsigned int total_len =3D 0; int head; bool indirect; =20 @@ -642,6 +664,7 @@ static inline int virtqueue_add_split(struct vring_virt= queue *vq, ++c =3D=3D total_sg ? 0 : VRING_DESC_F_NEXT, premapped); + total_len +=3D len; } } for (; n < (out_sgs + in_sgs); n++) { @@ -659,6 +682,7 @@ static inline int virtqueue_add_split(struct vring_virt= queue *vq, i, addr, len, (++c =3D=3D total_sg ? 0 : VRING_DESC_F_NEXT) | VRING_DESC_F_WRITE, premapped); + total_len +=3D len; } } =20 @@ -681,7 +705,12 @@ static inline int virtqueue_add_split(struct vring_vir= tqueue *vq, vq->vq.num_free -=3D descs_used; =20 /* Update free pointer */ - if (indirect) + if (virtqueue_is_in_order(vq)) { + vq->free_head +=3D descs_used; + if (vq->free_head >=3D vq->split.vring.num) + vq->free_head -=3D vq->split.vring.num; + vq->split.desc_state[head].total_len =3D total_len;; + } else if (indirect) vq->free_head =3D vq->split.desc_extra[head].next; else vq->free_head =3D i; @@ -854,6 +883,14 @@ static bool more_used_split(const struct vring_virtque= ue *vq) return virtqueue_poll_split(vq, vq->last_used_idx); } =20 +static bool more_used_split_in_order(const struct vring_virtqueue *vq) +{ + if (vq->batch_last.id !=3D vq->packed.vring.num) + return true; + + return virtqueue_poll_split(vq, vq->last_used_idx); +} + static void *virtqueue_get_buf_ctx_split(struct vring_virtqueue *vq, unsigned int *len, void **ctx) @@ -911,6 +948,73 @@ static void *virtqueue_get_buf_ctx_split(struct vring_= virtqueue *vq, return ret; } =20 +static void *virtqueue_get_buf_ctx_split_in_order(struct vring_virtqueue *= vq, + unsigned int *len, + void **ctx) +{ + void *ret; + unsigned int num =3D vq->split.vring.num; + u16 last_used; + + START_USE(vq); + + if (unlikely(vq->broken)) { + END_USE(vq); + return NULL; + } + + last_used =3D (vq->last_used_idx & (vq->split.vring.num - 1)); + + if (vq->batch_last.id =3D=3D num) { + if (!more_used_split(vq)) { + pr_debug("No more buffers in queue\n"); + END_USE(vq); + return NULL; + } + + /* Only get used array entries after they have been + * exposed by host. */ + virtio_rmb(vq->weak_barriers); + vq->batch_last.id =3D virtio32_to_cpu(vq->vq.vdev, + vq->split.vring.used->ring[last_used].id); + vq->batch_last.len =3D virtio32_to_cpu(vq->vq.vdev, + vq->split.vring.used->ring[last_used].len); + } + + if (vq->batch_last.id =3D=3D last_used) { + vq->batch_last.id =3D num; + *len =3D vq->batch_last.len; + } else + *len =3D vq->split.desc_state[last_used].total_len; + + if (unlikely(last_used >=3D num)) { + BAD_RING(vq, "id %u out of range\n", last_used); + return NULL; + } + if (unlikely(!vq->split.desc_state[last_used].data)) { + BAD_RING(vq, "id %u is not a head!\n", last_used); + return NULL; + } + + /* detach_buf_split clears data, so grab it now. */ + ret =3D vq->split.desc_state[last_used].data; + detach_buf_split_in_order(vq, last_used, ctx); + + vq->last_used_idx++; + /* If we expect an interrupt for the next entry, tell host + * by writing event index and flush out the write before + * the read in the next get_buf call. */ + if (!(vq->split.avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT)) + virtio_store_mb(vq->weak_barriers, + &vring_used_event(&vq->split.vring), + cpu_to_virtio16(vq->vq.vdev, vq->last_used_idx)); + + LAST_ADD_TIME_INVALID(vq); + + END_USE(vq); + return ret; +} + static void virtqueue_disable_cb_split(struct vring_virtqueue *vq) { if (!(vq->split.avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT)) { @@ -1004,7 +1108,10 @@ static void *virtqueue_detach_unused_buf_split(struc= t vring_virtqueue *vq) continue; /* detach_buf_split clears data, so grab it now. */ buf =3D vq->split.desc_state[i].data; - detach_buf_split(vq, i, NULL); + if (virtqueue_is_in_order(vq)) + detach_buf_split_in_order(vq, i, NULL); + else + detach_buf_split(vq, i, NULL); vq->split.avail_idx_shadow--; vq->split.vring.avail->idx =3D cpu_to_virtio16(vq->vq.vdev, vq->split.avail_idx_shadow); @@ -1067,6 +1174,7 @@ static void virtqueue_vring_attach_split(struct vring= _virtqueue *vq, =20 /* Put everything in free lists. */ vq->free_head =3D 0; + vq->batch_last.id =3D vq->split.vring.num; } =20 static int vring_alloc_state_extra_split(struct vring_virtqueue_split *vri= ng_split) @@ -1177,7 +1285,6 @@ static struct virtqueue *__vring_new_virtqueue_split(= unsigned int index, if (!vq) return NULL; =20 - vq->layout =3D SPLIT; vq->vq.callback =3D callback; vq->vq.vdev =3D vdev; vq->vq.name =3D name; @@ -1197,6 +1304,8 @@ static struct virtqueue *__vring_new_virtqueue_split(= unsigned int index, vq->indirect =3D virtio_has_feature(vdev, VIRTIO_RING_F_INDIRECT_DESC) && !context; vq->event =3D virtio_has_feature(vdev, VIRTIO_RING_F_EVENT_IDX); + vq->layout =3D virtio_has_feature(vdev, VIRTIO_F_IN_ORDER) ? + SPLIT_IN_ORDER : SPLIT; =20 if (virtio_has_feature(vdev, VIRTIO_F_ORDER_PLATFORM)) vq->weak_barriers =3D false; @@ -1360,13 +1469,14 @@ static int virtqueue_add_indirect_packed(struct vri= ng_virtqueue *vq, unsigned int in_sgs, void *data, bool premapped, - gfp_t gfp) + gfp_t gfp, + u16 id) { struct vring_desc_extra *extra; struct vring_packed_desc *desc; struct scatterlist *sg; - unsigned int i, n, err_idx, len; - u16 head, id; + unsigned int i, n, err_idx, len, total_len =3D 0; + u16 head; dma_addr_t addr; =20 head =3D vq->packed.next_avail_idx; @@ -1384,8 +1494,6 @@ static int virtqueue_add_indirect_packed(struct vring= _virtqueue *vq, } =20 i =3D 0; - id =3D vq->free_head; - BUG_ON(id =3D=3D vq->packed.vring.num); =20 for (n =3D 0; n < out_sgs + in_sgs; n++) { for (sg =3D sgs[n]; sg; sg =3D sg_next(sg)) { @@ -1405,6 +1513,7 @@ static int virtqueue_add_indirect_packed(struct vring= _virtqueue *vq, extra[i].flags =3D n < out_sgs ? 0 : VRING_DESC_F_WRITE; } =20 + total_len +=3D len; i++; } } @@ -1458,6 +1567,7 @@ static int virtqueue_add_indirect_packed(struct vring= _virtqueue *vq, vq->packed.desc_state[id].data =3D data; vq->packed.desc_state[id].indir_desc =3D desc; vq->packed.desc_state[id].last =3D id; + vq->packed.desc_state[id].total_len =3D total_len; =20 vq->num_added +=3D 1; =20 @@ -1510,8 +1620,11 @@ static inline int virtqueue_add_packed(struct vring_= virtqueue *vq, BUG_ON(total_sg =3D=3D 0); =20 if (virtqueue_use_indirect(vq, total_sg)) { + id =3D vq->free_head; + BUG_ON(id =3D=3D vq->packed.vring.num); err =3D virtqueue_add_indirect_packed(vq, sgs, total_sg, out_sgs, - in_sgs, data, premapped, gfp); + in_sgs, data, premapped, + gfp, id); if (err !=3D -ENOMEM) { END_USE(vq); return err; @@ -1632,6 +1745,152 @@ static inline int virtqueue_add_packed(struct vring= _virtqueue *vq, return -EIO; } =20 +static inline int virtqueue_add_packed_in_order(struct vring_virtqueue *vq, + struct scatterlist *sgs[], + unsigned int total_sg, + unsigned int out_sgs, + unsigned int in_sgs, + void *data, + void *ctx, + bool premapped, + gfp_t gfp) +{ + struct vring_packed_desc *desc; + struct scatterlist *sg; + unsigned int i, n, c, err_idx, total_len =3D 0; + __le16 head_flags, flags; + u16 head, avail_used_flags; + int err; + + START_USE(vq); + + BUG_ON(data =3D=3D NULL); + BUG_ON(ctx && vq->indirect); + + if (unlikely(vq->broken)) { + END_USE(vq); + return -EIO; + } + + LAST_ADD_TIME_UPDATE(vq); + + BUG_ON(total_sg =3D=3D 0); + + if (virtqueue_use_indirect(vq, total_sg)) { + err =3D virtqueue_add_indirect_packed(vq, sgs, total_sg, out_sgs, + in_sgs, data, premapped, gfp, + vq->packed.next_avail_idx); + if (err !=3D -ENOMEM) { + END_USE(vq); + return err; + } + + /* fall back on direct */ + } + + head =3D vq->packed.next_avail_idx; + avail_used_flags =3D vq->packed.avail_used_flags; + + WARN_ON_ONCE(total_sg > vq->packed.vring.num && !vq->indirect); + + desc =3D vq->packed.vring.desc; + i =3D head; + + if (unlikely(vq->vq.num_free < total_sg)) { + pr_debug("Can't add buf len %i - avail =3D %i\n", + total_sg, vq->vq.num_free); + END_USE(vq); + return -ENOSPC; + } + + c =3D 0; + for (n =3D 0; n < out_sgs + in_sgs; n++) { + for (sg =3D sgs[n]; sg; sg =3D sg_next(sg)) { + dma_addr_t addr; + u32 len; + + if (vring_map_one_sg(vq, sg, n < out_sgs ? + DMA_TO_DEVICE : DMA_FROM_DEVICE, + &addr, &len, premapped)) + goto unmap_release; + + flags =3D cpu_to_le16(vq->packed.avail_used_flags | + (++c =3D=3D total_sg ? 0 : VRING_DESC_F_NEXT) | + (n < out_sgs ? 0 : VRING_DESC_F_WRITE)); + if (i =3D=3D head) + head_flags =3D flags; + else + desc[i].flags =3D flags; + + + desc[i].addr =3D cpu_to_le64(addr); + desc[i].len =3D cpu_to_le32(len); + desc[i].id =3D cpu_to_le16(head); + + if (unlikely(vq->use_dma_api)) { + vq->packed.desc_extra[i].addr =3D premapped ? + DMA_MAPPING_ERROR: addr; + vq->packed.desc_extra[i].len =3D len; + vq->packed.desc_extra[i].flags =3D + le16_to_cpu(flags); + } + + if ((unlikely(++i >=3D vq->packed.vring.num))) { + i =3D 0; + vq->packed.avail_used_flags ^=3D + 1 << VRING_PACKED_DESC_F_AVAIL | + 1 << VRING_PACKED_DESC_F_USED; + vq->packed.avail_wrap_counter ^=3D 1; + } + + total_len +=3D len; + } + } + + /* We're using some buffers from the free list. */ + vq->vq.num_free -=3D total_sg; + + /* Update free pointer */ + vq->packed.next_avail_idx =3D i; + + /* Store token. */ + vq->packed.desc_state[head].num =3D total_sg; + vq->packed.desc_state[head].data =3D data; + vq->packed.desc_state[head].indir_desc =3D ctx; + vq->packed.desc_state[head].total_len =3D total_len; + + /* + * A driver MUST NOT make the first descriptor in the list + * available before all subsequent descriptors comprising + * the list are made available. + */ + virtio_wmb(vq->weak_barriers); + vq->packed.vring.desc[head].flags =3D head_flags; + vq->num_added +=3D total_sg; + + pr_debug("Added buffer head %i to %p\n", head, vq); + END_USE(vq); + + return 0; + +unmap_release: + err_idx =3D i; + i =3D head; + vq->packed.avail_used_flags =3D avail_used_flags; + + for (n =3D 0; n < total_sg; n++) { + if (i =3D=3D err_idx) + break; + vring_unmap_extra_packed(vq, &vq->packed.desc_extra[i]); + i++; + if (i >=3D vq->packed.vring.num) + i =3D 0; + } + + END_USE(vq); + return -EIO; +} + static bool virtqueue_kick_prepare_packed(struct vring_virtqueue *vq) { u16 new, old, off_wrap, flags, wrap_counter, event_idx; @@ -1752,7 +2011,7 @@ static inline bool is_used_desc_packed(const struct v= ring_virtqueue *vq, return avail =3D=3D used && used =3D=3D used_wrap_counter; } =20 -static bool virtqueue_poll_packed(const struct vring_virtqueue *vq, u16 of= f_wrap) +static bool __virtqueue_poll_packed(const struct vring_virtqueue *vq, u16 = off_wrap) { bool wrap_counter; u16 used_idx; @@ -1763,6 +2022,11 @@ static bool virtqueue_poll_packed(const struct vring= _virtqueue *vq, u16 off_wrap return is_used_desc_packed(vq, used_idx, wrap_counter); } =20 +static bool virtqueue_poll_packed(const struct vring_virtqueue *vq, u16 of= f_wrap) +{ + return __virtqueue_poll_packed(vq, off_wrap); +} + static bool more_used_packed(const struct vring_virtqueue *vq) { return virtqueue_poll_packed(vq, READ_ONCE(vq->last_used_idx)); @@ -1792,10 +2056,86 @@ static void update_last_used_idx_packed(struct vrin= g_virtqueue *vq, cpu_to_le16(vq->last_used_idx)); } =20 +static bool more_used_packed_in_order(const struct vring_virtqueue *vq) +{ + if (vq->batch_last.id !=3D vq->packed.vring.num) + return true; + + return virtqueue_poll_packed(vq, READ_ONCE(vq->last_used_idx)); +} + +static bool __more_used_packed(const struct vring_virtqueue *vq) +{ + return __virtqueue_poll_packed(vq, READ_ONCE(vq->last_used_idx)); +} + +static void *virtqueue_get_buf_ctx_packed_in_order(struct vring_virtqueue = *vq, + unsigned int *len, + void **ctx) +{ + unsigned int num =3D vq->packed.vring.num; + u16 last_used, id, last_used_idx; + bool used_wrap_counter; + void *ret; + + START_USE(vq); + + if (unlikely(vq->broken)) { + END_USE(vq); + return NULL; + } + + last_used_idx =3D vq->last_used_idx; + used_wrap_counter =3D packed_used_wrap_counter(last_used_idx); + last_used =3D packed_last_used(last_used_idx); + + if (vq->batch_last.id =3D=3D num) { + if (!__more_used_packed(vq)) { + pr_debug("No more buffers in queue\n"); + END_USE(vq); + return NULL; + } + /* Only get used elements after they have been exposed by host. */ + virtio_rmb(vq->weak_barriers); + vq->batch_last.id =3D + le16_to_cpu(vq->packed.vring.desc[last_used].id); + vq->batch_last.len =3D + le32_to_cpu(vq->packed.vring.desc[last_used].len); + } + + if (vq->batch_last.id =3D=3D last_used) { + vq->batch_last.id =3D num; + *len =3D vq->batch_last.len; + } else + *len =3D vq->packed.desc_state[last_used].total_len; + + if (unlikely(last_used >=3D num)) { + BAD_RING(vq, "id %u out of range\n", id); + return NULL; + } + if (unlikely(!vq->packed.desc_state[last_used].data)) { + BAD_RING(vq, "id %u is not a head!\n", id); + return NULL; + } + + /* detach_buf_packed clears data, so grab it now. */ + ret =3D vq->packed.desc_state[last_used].data; + detach_buf_packed_in_order(vq, last_used, ctx); + + update_last_used_idx_packed(vq, last_used, last_used, + used_wrap_counter); + + LAST_ADD_TIME_INVALID(vq); + + END_USE(vq); + return ret; +} + static void *virtqueue_get_buf_ctx_packed(struct vring_virtqueue *vq, unsigned int *len, void **ctx) { + unsigned int num =3D vq->packed.vring.num; u16 last_used, id, last_used_idx; bool used_wrap_counter; void *ret; @@ -1807,7 +2147,7 @@ static void *virtqueue_get_buf_ctx_packed(struct vrin= g_virtqueue *vq, return NULL; } =20 - if (!more_used_packed(vq)) { + if (!__more_used_packed(vq)) { pr_debug("No more buffers in queue\n"); END_USE(vq); return NULL; @@ -1822,7 +2162,7 @@ static void *virtqueue_get_buf_ctx_packed(struct vrin= g_virtqueue *vq, id =3D le16_to_cpu(vq->packed.vring.desc[last_used].id); *len =3D le32_to_cpu(vq->packed.vring.desc[last_used].len); =20 - if (unlikely(id >=3D vq->packed.vring.num)) { + if (unlikely(id >=3D num)) { BAD_RING(vq, "id %u out of range\n", id); return NULL; } @@ -1942,6 +2282,7 @@ static bool virtqueue_enable_cb_delayed_packed(struct= vring_virtqueue *vq) last_used_idx =3D READ_ONCE(vq->last_used_idx); wrap_counter =3D packed_used_wrap_counter(last_used_idx); used_idx =3D packed_last_used(last_used_idx); + if (is_used_desc_packed(vq, used_idx, wrap_counter)) { END_USE(vq); return false; @@ -1963,7 +2304,7 @@ static void *virtqueue_detach_unused_buf_packed(struc= t vring_virtqueue *vq) continue; /* detach_buf clears data, so grab it now. */ buf =3D vq->packed.desc_state[i].data; - detach_buf_packed(vq, i, NULL); + detach_buf_packed_in_order(vq, i, NULL); END_USE(vq); return buf; } @@ -1989,6 +2330,8 @@ static struct vring_desc_extra *vring_alloc_desc_extr= a(unsigned int num) for (i =3D 0; i < num - 1; i++) desc_extra[i].next =3D i + 1; =20 + desc_extra[num - 1].next =3D 0; + return desc_extra; } =20 @@ -2120,8 +2463,12 @@ static void virtqueue_vring_attach_packed(struct vri= ng_virtqueue *vq, { vq->packed =3D *vring_packed; =20 - /* Put everything in free lists. */ - vq->free_head =3D 0; + if (virtqueue_is_in_order(vq)) + vq->batch_last.id =3D vq->split.vring.num; + else { + /* Put everything in free lists. */ + vq->free_head =3D 0; + } } =20 static void virtqueue_reset_packed(struct vring_virtqueue *vq) @@ -2168,13 +2515,14 @@ static struct virtqueue *__vring_new_virtqueue_pack= ed(unsigned int index, #else vq->broken =3D false; #endif - vq->layout =3D PACKED; vq->dma_dev =3D dma_dev; vq->use_dma_api =3D vring_use_dma_api(vdev); =20 vq->indirect =3D virtio_has_feature(vdev, VIRTIO_RING_F_INDIRECT_DESC) && !context; vq->event =3D virtio_has_feature(vdev, VIRTIO_RING_F_EVENT_IDX); + vq->layout =3D virtio_has_feature(vdev, VIRTIO_F_IN_ORDER) ? + PACKED_IN_ORDER : PACKED; =20 if (virtio_has_feature(vdev, VIRTIO_F_ORDER_PLATFORM)) vq->weak_barriers =3D false; @@ -2284,9 +2632,39 @@ static const struct virtqueue_ops packed_ops =3D { .reset =3D virtqueue_reset_packed, }; =20 +static const struct virtqueue_ops split_in_order_ops =3D { + .add =3D virtqueue_add_split, + .get =3D virtqueue_get_buf_ctx_split_in_order, + .kick_prepare =3D virtqueue_kick_prepare_split, + .disable_cb =3D virtqueue_disable_cb_split, + .enable_cb_delayed =3D virtqueue_enable_cb_delayed_split, + .enable_cb_prepare =3D virtqueue_enable_cb_prepare_split, + .poll =3D virtqueue_poll_split, + .detach_unused_buf =3D virtqueue_detach_unused_buf_split, + .more_used =3D more_used_split_in_order, + .resize =3D virtqueue_resize_split, + .reset =3D virtqueue_reset_split, +}; + +static const struct virtqueue_ops packed_in_order_ops =3D { + .add =3D virtqueue_add_packed_in_order, + .get =3D virtqueue_get_buf_ctx_packed_in_order, + .kick_prepare =3D virtqueue_kick_prepare_packed, + .disable_cb =3D virtqueue_disable_cb_packed, + .enable_cb_delayed =3D virtqueue_enable_cb_delayed_packed, + .enable_cb_prepare =3D virtqueue_enable_cb_prepare_packed, + .poll =3D virtqueue_poll_packed, + .detach_unused_buf =3D virtqueue_detach_unused_buf_packed, + .more_used =3D more_used_packed_in_order, + .resize =3D virtqueue_resize_packed, + .reset =3D virtqueue_reset_packed, +}; + static const struct virtqueue_ops *const all_ops[VQ_TYPE_MAX] =3D { [SPLIT] =3D &split_ops, - [PACKED] =3D &packed_ops + [PACKED] =3D &packed_ops, + [SPLIT_IN_ORDER] =3D &split_in_order_ops, + [PACKED_IN_ORDER] =3D &packed_in_order_ops, }; =20 static int virtqueue_disable_and_recycle(struct virtqueue *_vq, @@ -2342,6 +2720,12 @@ static int virtqueue_enable_after_reset(struct virtq= ueue *_vq) case PACKED: \ ret =3D all_ops[PACKED]->op(vq, ##__VA_ARGS__); \ break; \ + case SPLIT_IN_ORDER: \ + ret =3D all_ops[SPLIT_IN_ORDER]->op(vq, ##__VA_ARGS__); \ + break; \ + case PACKED_IN_ORDER: \ + ret =3D all_ops[PACKED_IN_ORDER]->op(vq, ##__VA_ARGS__); \ + break; \ default: \ BUG(); \ break; \ @@ -2358,10 +2742,16 @@ static int virtqueue_enable_after_reset(struct virt= queue *_vq) case PACKED: \ all_ops[PACKED]->op(vq, ##__VA_ARGS__); \ break; \ - default: \ - BUG(); \ - break; \ - } \ + case SPLIT_IN_ORDER: \ + all_ops[SPLIT_IN_ORDER]->op(vq, ##__VA_ARGS__); \ + break; \ + case PACKED_IN_ORDER: \ + all_ops[PACKED_IN_ORDER]->op(vq, ##__VA_ARGS__); \ + break; \ + default: \ + BUG(); \ + break; \ + } \ }) =20 static inline int virtqueue_add(struct virtqueue *_vq, @@ -3076,6 +3466,8 @@ void vring_transport_features(struct virtio_device *v= dev) break; case VIRTIO_F_NOTIFICATION_DATA: break; + case VIRTIO_F_IN_ORDER: + break; default: /* We don't understand this bit. */ __virtio_clear_bit(vdev, i); --=20 2.39.5