From nobody Sat Nov 15 20:49:00 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1747221613; cv=none; d=zohomail.com; s=zohoarc; b=Krmv9LeIsxYMxQK+LjUra6QYkI88u0b6nNi5L2O3MlH+LS1G3wrYoB0n4X8g8ruS5SZV2SDiJbG5xDN3BjKoV2t7HiKIkMlAfZeQds9WGSXhqdHxQd3IFIlbOVKL1c5KtMU0IAr/CAO7AH9MV1VnpnayPbwIADdKa5myHIHgoIU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1747221613; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=N+ANctcqNUebqa6A05t2Bh5IOy6djTiWwSLGhOcQkKo=; b=MYxQxxlAnGs7EHV1L+rb2wFQBRLRtRYDkM7KXLFkvchzZPtmoH1pEHPzcUvRP+ZVZ2Z8p5Enr+Tl3eFWDSvJZ2SJfvApRFnqrMrgJOMiEgA8ymtxlSONX+Ar1SfpmpUHqtFYgzBp6UmEP5JuqzsuhqWDw3MOzeJpgamUXF2QKU8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 17472216135203.266527674659983; Wed, 14 May 2025 04:20:13 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1uFA9J-0005zO-5Z; Wed, 14 May 2025 07:19:45 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uFA9F-0005yp-76 for qemu-devel@nongnu.org; Wed, 14 May 2025 07:19:41 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uFA9D-0001Ll-Em for qemu-devel@nongnu.org; Wed, 14 May 2025 07:19:40 -0400 Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-493-rWzCaHOnPoyTcTwD-uU5XQ-1; Wed, 14 May 2025 07:19:36 -0400 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id E1E571800361 for ; Wed, 14 May 2025 11:19:35 +0000 (UTC) Received: from toolbx.redhat.com (unknown [10.42.28.147]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id C976D19560A3; Wed, 14 May 2025 11:19:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1747221578; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=N+ANctcqNUebqa6A05t2Bh5IOy6djTiWwSLGhOcQkKo=; b=D5aR6Gqx1g7V/nsq9rG/tuprZqoYngutjJ768VDzrbbm90uevJ9dPtXAwTZ2zsgT3iQTvL zN+cZ4KSQTZ8Xcqw2EW/xEkWdG6IM/3hVbkS+VNHGc8XecMlGGixswA7P9o9QDAm5eptFS oVhqClD0lntehBQae+D2DyjA73u79Yg= X-MC-Unique: rWzCaHOnPoyTcTwD-uU5XQ-1 X-Mimecast-MFC-AGG-ID: rWzCaHOnPoyTcTwD-uU5XQ_1747221576 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= To: qemu-devel@nongnu.org Cc: =?UTF-8?q?Marc-Andr=C3=A9=20Lureau?= , =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= Subject: [PATCH v2 1/3] ui/vnc.c: replace big endian flag with byte order value Date: Wed, 14 May 2025 12:19:29 +0100 Message-ID: <20250514111931.1711390-2-berrange@redhat.com> In-Reply-To: <20250514111931.1711390-1-berrange@redhat.com> References: <20250514111931.1711390-1-berrange@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=berrange@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -37 X-Spam_score: -3.8 X-Spam_bar: --- X-Spam_report: (-3.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.686, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1747221615185019000 It will make it easier to do certain comparisons in future if we store G_BIG_ENDIAN/G_LITTLE_ENDIAN directly, instead of a boolean flag, as we can then compare directly to the G_BYTE_ORDER constant. Signed-off-by: Daniel P. Berrang=C3=A9 Reviewed-by: BALATON Zoltan Reviewed-by: Philippe Mathieu-Daud=C3=A9 --- ui/vnc-enc-tight.c | 2 +- ui/vnc-enc-zrle.c | 2 +- ui/vnc-jobs.c | 2 +- ui/vnc.c | 6 +++--- ui/vnc.h | 2 +- 5 files changed, 7 insertions(+), 7 deletions(-) diff --git a/ui/vnc-enc-tight.c b/ui/vnc-enc-tight.c index 41f559eb83..f8aaa8f346 100644 --- a/ui/vnc-enc-tight.c +++ b/ui/vnc-enc-tight.c @@ -150,7 +150,7 @@ tight_detect_smooth_image24(VncState *vs, int w, int h) * If client is big-endian, color samples begin from the second * byte (offset 1) of a 32-bit pixel value. */ - off =3D vs->client_be; + off =3D vs->client_endian =3D=3D G_BIG_ENDIAN ? 1 : 0; =20 memset(stats, 0, sizeof (stats)); =20 diff --git a/ui/vnc-enc-zrle.c b/ui/vnc-enc-zrle.c index bd33b89063..97ec6c7119 100644 --- a/ui/vnc-enc-zrle.c +++ b/ui/vnc-enc-zrle.c @@ -255,7 +255,7 @@ static void zrle_write_u8(VncState *vs, uint8_t value) static int zrle_send_framebuffer_update(VncState *vs, int x, int y, int w, int h) { - bool be =3D vs->client_be; + bool be =3D vs->client_endian =3D=3D G_BIG_ENDIAN; size_t bytes; int zywrle_level; =20 diff --git a/ui/vnc-jobs.c b/ui/vnc-jobs.c index fcca7ec632..d3486af9e2 100644 --- a/ui/vnc-jobs.c +++ b/ui/vnc-jobs.c @@ -188,7 +188,7 @@ static void vnc_async_encoding_start(VncState *orig, Vn= cState *local) local->lossy_rect =3D orig->lossy_rect; local->write_pixels =3D orig->write_pixels; local->client_pf =3D orig->client_pf; - local->client_be =3D orig->client_be; + local->client_endian =3D orig->client_endian; local->tight =3D orig->tight; local->zlib =3D orig->zlib; local->hextile =3D orig->hextile; diff --git a/ui/vnc.c b/ui/vnc.c index 9e097dc4b4..ab18172c4d 100644 --- a/ui/vnc.c +++ b/ui/vnc.c @@ -891,7 +891,7 @@ void vnc_convert_pixel(VncState *vs, uint8_t *buf, uint= 32_t v) buf[0] =3D v; break; case 2: - if (vs->client_be) { + if (vs->client_endian =3D=3D G_BIG_ENDIAN) { buf[0] =3D v >> 8; buf[1] =3D v; } else { @@ -901,7 +901,7 @@ void vnc_convert_pixel(VncState *vs, uint8_t *buf, uint= 32_t v) break; default: case 4: - if (vs->client_be) { + if (vs->client_endian =3D=3D G_BIG_ENDIAN) { buf[0] =3D v >> 24; buf[1] =3D v >> 16; buf[2] =3D v >> 8; @@ -2312,7 +2312,7 @@ static void set_pixel_format(VncState *vs, int bits_p= er_pixel, vs->client_pf.bits_per_pixel =3D bits_per_pixel; vs->client_pf.bytes_per_pixel =3D bits_per_pixel / 8; vs->client_pf.depth =3D bits_per_pixel =3D=3D 32 ? 24 : bits_per_pixel; - vs->client_be =3D big_endian_flag; + vs->client_endian =3D big_endian_flag ? G_BIG_ENDIAN : G_LITTLE_ENDIAN; =20 if (!true_color_flag) { send_color_map(vs); diff --git a/ui/vnc.h b/ui/vnc.h index acc53a2cc1..02613aa63a 100644 --- a/ui/vnc.h +++ b/ui/vnc.h @@ -323,7 +323,7 @@ struct VncState VncWritePixels *write_pixels; PixelFormat client_pf; pixman_format_code_t client_format; - bool client_be; + int client_endian; /* G_LITTLE_ENDIAN or G_BIG_ENDIAN */ =20 CaptureVoiceOut *audio_cap; struct audsettings as; --=20 2.49.0 From nobody Sat Nov 15 20:49:00 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1747221631; cv=none; d=zohomail.com; s=zohoarc; b=K9bGuNe67pPpLMiDl2xM6A4gxcgmWHfokG4JH6lSdKcznfpt/Vj4cpakSFBJlZC+jDhHIsKuNC4KqUWXyA3bkvO4T8P1nkeSLLjLHiNfvizMbOMRTfG6iidEJZkWwOOOd7J2FZSAcl14ub0kZjgp+tD/9OKHFvXV+GyNvYeQB/M= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1747221631; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=Ap10gC6UPF4kfaH1XFvlSl7+azVePcG6UA51Fn1/tbg=; b=ce2yxA9OtkIoULJdQ7ZkX2/WMOU2YBzzwSUNCGULD3wqbmd8mLd20IpbMgdtm6EM4kesgqWn8nKizAkK0WJa4xmnkKjXqH4NrsMsw9TEAwIuYfVO8vklJmb1jEeTWG/mXoww9JY4uLyq59tDJi+B9KbzcjBK/XVrdpbLKCI0MN8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1747221631926370.6542724056918; Wed, 14 May 2025 04:20:31 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1uFA9K-0005zu-CC; Wed, 14 May 2025 07:19:46 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uFA9J-0005zS-1i for qemu-devel@nongnu.org; Wed, 14 May 2025 07:19:45 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uFA9H-0001ME-02 for qemu-devel@nongnu.org; Wed, 14 May 2025 07:19:44 -0400 Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-551-jJaplFhKPYqv81S7dQmivA-1; Wed, 14 May 2025 07:19:38 -0400 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id CC5E41955DC5; Wed, 14 May 2025 11:19:37 +0000 (UTC) Received: from toolbx.redhat.com (unknown [10.42.28.147]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 50D0319560A3; Wed, 14 May 2025 11:19:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1747221582; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ap10gC6UPF4kfaH1XFvlSl7+azVePcG6UA51Fn1/tbg=; b=bqtokI036YPw8t7Kk8NVmjNVQTiGSnrp3g3IEiu1dBgW1edHLsSUkgXup+FnKUiS8vAR8h 50R3eqGiQl25w49TwW4dGjLrMr0ZMAfrngtDof6wzT11Z/Bq10gFUszmwEwezhrVMTQNU2 oMJQQmfAonkIg7BkvCEOGc/l0Min2fw= X-MC-Unique: jJaplFhKPYqv81S7dQmivA-1 X-Mimecast-MFC-AGG-ID: jJaplFhKPYqv81S7dQmivA_1747221578 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= To: qemu-devel@nongnu.org Cc: =?UTF-8?q?Marc-Andr=C3=A9=20Lureau?= , =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= Subject: [PATCH v2 2/3] ui/vnc: take account of client byte order in pixman format Date: Wed, 14 May 2025 12:19:30 +0100 Message-ID: <20250514111931.1711390-3-berrange@redhat.com> In-Reply-To: <20250514111931.1711390-1-berrange@redhat.com> References: <20250514111931.1711390-1-berrange@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=berrange@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -37 X-Spam_score: -3.8 X-Spam_bar: --- X-Spam_report: (-3.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.686, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1747221634466116600 The set_pixel_conversion() method is responsible for determining whether the VNC client pixel format matches the server format, and thus whether we can use the fast path "copy" impl for sending pixels, or must use the generic impl with bit swizzling. The VNC server format is set at build time to VNC_SERVER_FB_FORMAT, which corresponds to PIXMAN_x8r8g8b8. The qemu_pixman_get_format() method is then responsible for converting the VNC pixel format into a pixman format. The VNC client pixel shifts are relative to the associated endianness. The pixman formats are always relative to the host native endianness. The qemu_pixman_get_format() method does not take into account the VNC client endianness, and is thus returning a pixman format that is only valid with the host endianness matches that of the VNC client. This has been broken since pixman was introduced to the VNC server: commit 9f64916da20eea67121d544698676295bbb105a7 Author: Gerd Hoffmann Date: Wed Oct 10 13:29:43 2012 +0200 pixman/vnc: use pixman images in vnc. The flaw can be demonstrated using the Tigervnc client by using vncviewer -AutoSelect=3D0 -PreferredEncoding=3Draw server:display connecting from a LE client to a QEMU on a BE server, or the reverse. The bug was masked, however, because almost all VNC clients will advertize support for the "tight" encoding and the QEMU VNC server will prefer "tight" if advertized. The tight_pack24 method is responsible for taking a set of pixels which have already been converted into client endianness and then repacking them into the TPIXEL format which the RFB spec defines as "TPIXEL is only 3 bytes long, where the first byte is the red component, the second byte is the green component, and the third byte is the blue component of the pixel color value" IOW, the TPIXEL format is fixed on the wire, regardless of what the VNC client declare as its endianness. Since the VNC pixel encoding code was failing to honour the endian flag of the client, the tight_pack24 method was always operating on data in native endianness. Its impl cancelled out the VNC pixel encoding bug. With the VNC pixel encoding code now fixed, the tight_pack24 method needs to take into account that it is operating on data in client endianness, not native endianness. It thus may need to invert the pixel shifts. Reviewed-by: Philippe Mathieu-Daud=C3=A9 Signed-off-by: Daniel P. Berrang=C3=A9 --- include/ui/qemu-pixman.h | 4 ++-- ui/qemu-pixman.c | 15 ++++++++------- ui/vnc-enc-tight.c | 2 +- ui/vnc.c | 3 ++- 4 files changed, 13 insertions(+), 11 deletions(-) diff --git a/include/ui/qemu-pixman.h b/include/ui/qemu-pixman.h index 193bc046d1..2ca0ed7029 100644 --- a/include/ui/qemu-pixman.h +++ b/include/ui/qemu-pixman.h @@ -75,12 +75,12 @@ PixelFormat qemu_pixelformat_from_pixman(pixman_format_= code_t format); pixman_format_code_t qemu_default_pixman_format(int bpp, bool native_endia= n); pixman_format_code_t qemu_drm_format_to_pixman(uint32_t drm_format); uint32_t qemu_pixman_to_drm_format(pixman_format_code_t pixman); -int qemu_pixman_get_type(int rshift, int gshift, int bshift); +int qemu_pixman_get_type(int rshift, int gshift, int bshift, int endian); bool qemu_pixman_check_format(DisplayChangeListener *dcl, pixman_format_code_t format); =20 #ifdef CONFIG_PIXMAN -pixman_format_code_t qemu_pixman_get_format(PixelFormat *pf); +pixman_format_code_t qemu_pixman_get_format(PixelFormat *pf, int endian); pixman_image_t *qemu_pixman_linebuf_create(pixman_format_code_t format, int width); void qemu_pixman_linebuf_fill(pixman_image_t *linebuf, pixman_image_t *fb, diff --git a/ui/qemu-pixman.c b/ui/qemu-pixman.c index 6ef4376f4e..ef4e71da11 100644 --- a/ui/qemu-pixman.c +++ b/ui/qemu-pixman.c @@ -126,33 +126,34 @@ uint32_t qemu_pixman_to_drm_format(pixman_format_code= _t pixman_format) return 0; } =20 -int qemu_pixman_get_type(int rshift, int gshift, int bshift) +int qemu_pixman_get_type(int rshift, int gshift, int bshift, int endian) { int type =3D PIXMAN_TYPE_OTHER; + bool native_endian =3D (endian =3D=3D G_BYTE_ORDER); =20 if (rshift > gshift && gshift > bshift) { if (bshift =3D=3D 0) { - type =3D PIXMAN_TYPE_ARGB; + type =3D native_endian ? PIXMAN_TYPE_ARGB : PIXMAN_TYPE_BGRA; } else { - type =3D PIXMAN_TYPE_RGBA; + type =3D native_endian ? PIXMAN_TYPE_RGBA : PIXMAN_TYPE_ABGR; } } else if (rshift < gshift && gshift < bshift) { if (rshift =3D=3D 0) { - type =3D PIXMAN_TYPE_ABGR; + type =3D native_endian ? PIXMAN_TYPE_ABGR : PIXMAN_TYPE_RGBA; } else { - type =3D PIXMAN_TYPE_BGRA; + type =3D native_endian ? PIXMAN_TYPE_BGRA : PIXMAN_TYPE_ARGB; } } return type; } =20 #ifdef CONFIG_PIXMAN -pixman_format_code_t qemu_pixman_get_format(PixelFormat *pf) +pixman_format_code_t qemu_pixman_get_format(PixelFormat *pf, int endian) { pixman_format_code_t format; int type; =20 - type =3D qemu_pixman_get_type(pf->rshift, pf->gshift, pf->bshift); + type =3D qemu_pixman_get_type(pf->rshift, pf->gshift, pf->bshift, endi= an); format =3D PIXMAN_FORMAT(pf->bits_per_pixel, type, pf->abits, pf->rbits, pf->gbits, pf->bbits); if (!pixman_format_supported_source(format)) { diff --git a/ui/vnc-enc-tight.c b/ui/vnc-enc-tight.c index f8aaa8f346..a5bdc19ebb 100644 --- a/ui/vnc-enc-tight.c +++ b/ui/vnc-enc-tight.c @@ -891,7 +891,7 @@ static void tight_pack24(VncState *vs, uint8_t *buf, si= ze_t count, size_t *ret) =20 buf8 =3D buf; =20 - if (1 /* FIXME */) { + if (vs->client_endian =3D=3D G_BYTE_ORDER) { rshift =3D vs->client_pf.rshift; gshift =3D vs->client_pf.gshift; bshift =3D vs->client_pf.bshift; diff --git a/ui/vnc.c b/ui/vnc.c index ab18172c4d..d095cd7da3 100644 --- a/ui/vnc.c +++ b/ui/vnc.c @@ -2240,7 +2240,8 @@ static void set_encodings(VncState *vs, int32_t *enco= dings, size_t n_encodings) =20 static void set_pixel_conversion(VncState *vs) { - pixman_format_code_t fmt =3D qemu_pixman_get_format(&vs->client_pf); + pixman_format_code_t fmt =3D qemu_pixman_get_format(&vs->client_pf, + vs->client_endian); =20 if (fmt =3D=3D VNC_SERVER_FB_FORMAT) { vs->write_pixels =3D vnc_write_pixels_copy; --=20 2.49.0 From nobody Sat Nov 15 20:49:00 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1747221642; cv=none; d=zohomail.com; s=zohoarc; b=Ne0EMOLiVqWNPoafyTU0QDbVnza+GONWTweY0F/YeI9ghOIqJrgKGum8EBNvIFrUr19xF9N7q+kVqpPkpT88OXMG1CTk9ILtA/lxOKl7jG5qV19RcD96XkxcV0w42s/WLFA5nouKbK9QLhPMFgCQctSoWzyEbHwPJbClNR9YiXQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1747221642; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=4J5CtSMgf/lbTPFeTGrU8Ks+Fea1QIrBlMcjYXKmsKI=; b=mkROGEFlSvkAyCFDWnuOFsKCjZKSTZPkrj4qwQ6Tq01lp57t1hlH/b0A/M/EHLEi01m7WXc6yGxLA5LPYAO6BZvZsLJsbZ9i8YR9ttjhei0OXTuFCcRX40xn91twcHUmLdj3jfpoSHtjCH3KBMKoBHrF77BfavxLotUbyNPgHvI= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1747221642428900.7972737726766; Wed, 14 May 2025 04:20:42 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1uFA9L-00060J-Pf; Wed, 14 May 2025 07:19:47 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uFA9K-0005zv-BF for qemu-devel@nongnu.org; Wed, 14 May 2025 07:19:46 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uFA9I-0001MK-IM for qemu-devel@nongnu.org; Wed, 14 May 2025 07:19:45 -0400 Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-552-mq43wVQcOEWlToZpyPLPPQ-1; Wed, 14 May 2025 07:19:40 -0400 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id C6D53180048E; Wed, 14 May 2025 11:19:39 +0000 (UTC) Received: from toolbx.redhat.com (unknown [10.42.28.147]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 6CF8019560A3; Wed, 14 May 2025 11:19:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1747221583; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4J5CtSMgf/lbTPFeTGrU8Ks+Fea1QIrBlMcjYXKmsKI=; b=Cad1W1xvkEBRxEMMk8+Ft+5KtdAQ422SE31wVgGKjlQnyn0ZbumO8lO2s3swB0J70iL3k0 yd0jSKSFXBTupnpaTsS5yVZdx1AwXWRY4YgBUosP+0lZJnrTyq/FBckZ2YLHZTFDhN6OYD Ib7mQ/fv+j9Tk4Xls6MP6JrrjVj6xFg= X-MC-Unique: mq43wVQcOEWlToZpyPLPPQ-1 X-Mimecast-MFC-AGG-ID: mq43wVQcOEWlToZpyPLPPQ_1747221579 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= To: qemu-devel@nongnu.org Cc: =?UTF-8?q?Marc-Andr=C3=A9=20Lureau?= , =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= Subject: [PATCH v2 3/3] ui/vnc: fix tight palette pixel encoding for 8/16-bpp formats Date: Wed, 14 May 2025 12:19:31 +0100 Message-ID: <20250514111931.1711390-4-berrange@redhat.com> In-Reply-To: <20250514111931.1711390-1-berrange@redhat.com> References: <20250514111931.1711390-1-berrange@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=berrange@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -37 X-Spam_score: -3.8 X-Spam_bar: --- X-Spam_report: (-3.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.686, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1747221642997019000 When sending a tight rectangle with the palette filter, if the client format was 8/16bpp, the colours on big endian hosts are not set as we're sending the wrong bytes. We must first cast the 32-bit colour to a 16/8-bit value, and then send the result. Reviewed-by: Philippe Mathieu-Daud=C3=A9 Signed-off-by: Daniel P. Berrang=C3=A9 --- ui/vnc-enc-tight.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/ui/vnc-enc-tight.c b/ui/vnc-enc-tight.c index a5bdc19ebb..705dcbb429 100644 --- a/ui/vnc-enc-tight.c +++ b/ui/vnc-enc-tight.c @@ -1001,13 +1001,17 @@ static int send_mono_rect(VncState *vs, int x, int = y, break; } case 2: - vnc_write(vs, &bg, 2); - vnc_write(vs, &fg, 2); + uint16_t bg16 =3D bg; + uint16_t fg16 =3D fg; + vnc_write(vs, &bg16, 2); + vnc_write(vs, &fg16, 2); tight_encode_mono_rect16(vs->tight->tight.buffer, w, h, bg, fg); break; default: - vnc_write_u8(vs, bg); - vnc_write_u8(vs, fg); + uint8_t bg8 =3D bg; + uint8_t fg8 =3D fg; + vnc_write_u8(vs, bg8); + vnc_write_u8(vs, fg8); tight_encode_mono_rect8(vs->tight->tight.buffer, w, h, bg, fg); break; } --=20 2.49.0