From nobody Mon Jun 15 10:46:33 2026 Received: from mail-qt1-f181.google.com (mail-qt1-f181.google.com [209.85.160.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CCE3926A1AC for ; Thu, 9 Apr 2026 16:41:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775752921; cv=none; b=ICfFJfFKEEWIf8g4CaqYJDrcbhjG/iCF+xT7wtlt8H78vAuTk8ob33ktgWBGeqs+x9EinZHHsNLzL4FpdRfoGQhd7dnBtxRVNmk5oqLNdCGjlHtYUuhFJfFkBWUVeB0pKJyHG+dnP6tZz0mriPYacXpK0WPIwRyAC9ocOhO2nAE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775752921; c=relaxed/simple; bh=UiiB/dvJmtTT5bw8VptLNHrZqJ7OV1e/VxOu3A2sTNc=; h=From:To:Cc:Subject:Date:Message-Id:MIME-Version; b=I8Kt2kobi1mjmtIxsiaMu3EFk9iPbIKcOxCBAdzClENoP7j+VobvfAsAUwcrW8R5RpTolOFuPsMll4T4//XqeTleyl77aK9+CT0hBE5xZJ1PNV63+IK5ql7WpcC0M1EW2O5CExg8RSQkneAp8Qi+jDxsu8SMsD9c9sFdf7EDYhk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=r/Fu1E6W; arc=none smtp.client-ip=209.85.160.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="r/Fu1E6W" Received: by mail-qt1-f181.google.com with SMTP id d75a77b69052e-50d59d249fbso11873541cf.0 for ; Thu, 09 Apr 2026 09:41:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1775752919; x=1776357719; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=uih4IkEdnJDVtHqKmPWCYNbfi2V3Eeb0IfkSkkn7OLw=; b=r/Fu1E6WNDGwx5B1zLESz7qklEQBeU+tnvYxdncp+zwp1YzCoiW5OPceIJVnYpNQnc Oft5pmEHFpEbsILizvv5ALllmB8YLBPrtd8R7JWRhhaHd6KDCG34yVi53/qyyH0I0bfc PWV23V8u2nXWczNGRx0vUtbWQsmiMWoDBdY8c06TWqybFQfHXeBc1i759HBTTPvAbXcA Q7UJfsSQehtNLOzelOLw9A7ooJUHqdPFAWHU1FTXOk35xLzoLERdX1SN9iNfpqn+h+nS 9iT8HSbXoTkyP3qJJYpqfsOOoPH5oOlR+eIZbDxzWRnUOKMKKgkOVBcUcK0Gk0yBBxky cAOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1775752919; x=1776357719; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=uih4IkEdnJDVtHqKmPWCYNbfi2V3Eeb0IfkSkkn7OLw=; b=fpuDu889spkwSh//rubqeHAkSqsmQahOFVze8+VHqtMxaBJclMYBHSBTm7XOJqijKp b7LxzsB6fjR2d1kLfc4JSGjGvwejIJS3s9tb427YnZf3dDXfskXqc+aTb4SZc13EX9Ld 6DNwunSX+8EmOfu69CDXDQzynIGJaB1UyvxceNHOFhwVWDt5Dpe13RpWDRnliUajGPzc p8DwgbnoszFOrk7wglQYUDNe7JKxdFWpenwbiKm7ozgy3/FKxYj6JGERvIW6o85wTsV9 8s3mzN5zOIpYadsIy2rl367AxfkyCcdb6myEZ36YnAau+lHpHXyXZOZTbp0hrlvwbBvQ IOUQ== X-Forwarded-Encrypted: i=1; AJvYcCX8M41e74swvdFnmHaoHJtbEM82m/kZ4aPf6c20SnJ38KG6d7UOZvvaDxM2fTxLNRJNTyVrk0SWD1+PkkU=@vger.kernel.org X-Gm-Message-State: AOJu0YyaY/qH6GaUbPXme0QpdETi8/vjG6eOSCd5PM/X06jC2Pmv0rRU 7eDl0oWR6QmY5AfDxHobLMOrrgmVtZu4IhgkHS28EcgO/s2Gev1LKeJN X-Gm-Gg: AeBDiesQuZXX+8TKe6SwvRD4kwiUzeCFeV0IfSNqEL0Cqe7FdzUSOj+L4vCiBICM0KI 3bUZTMXWq6YGzLla0Rv1rSYzV4Tkm8vmZRDHQ29HQSd/R9S8AoX/SnU8cHj+WCCmGxgJJPi9FFm 4l51gK4HaqSIH00D7gVYjzjaB4pt1/oGwJ4clAubYWzLpx4Q/cL+el0xD3MEI0f0voTeu8fdG4z ywgj1DFdmvkU5JWUVxi/XjsP0QL/VMHebg5MZgjhpurBspV1aBpLyOcqOZsWRb2n/7E/0WOWvfL doIpZPiALSm5yIcXGM/qwM1JCUIR0V/vwJ8SajGBZRqwct/Iaq3ar3UOzAGrPkEgwtLXox7h/su CmQC3BI36XGtF32Ye+U3cEgV9MywraejW1JuI0FKCemLuyOX36w9I2XQ8UggeJ1Euo0hO6H49aO 9ulabsmsLc54Lo5OkaXk9TLKe2dIuskSI/FkzP5P/TgyxFqoZOdKMvGGHrHuxstiaPS1IfkL+Gx BHH33UlDDHBd2p+k4LOorO09BxQ X-Received: by 2002:a05:622a:5d0f:b0:50d:bc78:5ff6 with SMTP id d75a77b69052e-50dbc78892dmr70800001cf.61.1775752918681; Thu, 09 Apr 2026 09:41:58 -0700 (PDT) Received: from TDC4045031631.e0cglfehwr0e5gttmepj3hi3hf.ux.internal.cloudapp.net ([20.63.37.123]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-8ac849e2891sm622556d6.5.2026.04.09.09.41.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 09 Apr 2026 09:41:58 -0700 (PDT) From: Ashutosh Desai To: Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Simona Vetter Cc: dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org, Ashutosh Desai Subject: [PATCH] drm/gem: Fix inconsistent plane dimension calculation in drm_gem_fb_init_with_funcs() Date: Thu, 9 Apr 2026 16:41:56 +0000 Message-Id: <20260409164156.2235189-1-ashutoshdesai993@gmail.com> X-Mailer: git-send-email 2.34.1 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" drm_gem_fb_init_with_funcs() computes sub-sampled plane dimensions using plain integer division: unsigned int width =3D mode_cmd->width / (i ? info->hsub : 1); unsigned int height =3D mode_cmd->height / (i ? info->vsub : 1); However, the ioctl-level framebuffer_check() in drm_framebuffer.c uses drm_format_info_plane_width/height() which round up dimensions via DIV_ROUND_UP(). This inconsistency corrupts the subsequent GEM object size check for certain pixel format and dimension combinations. For example, with NV12 (vsub=3D2) and a 1-pixel-tall framebuffer the GEM size validation path sees height=3D0 instead of height=3D1. The expression (height - 1) then wraps to UINT_MAX as an unsigned int, causing min_size to overflow and wrap back to a small value. A tiny GEM object therefore passes the size guard, yet when the GPU accesses the chroma plane it will read or write memory beyond the object's bounds. Fix by replacing the open-coded divisions with drm_format_info_plane_width() and drm_format_info_plane_height(), which use DIV_ROUND_UP() and match the calculation already used in framebuffer_check(). Signed-off-by: Ashutosh Desai Reviewed-by: Thomas Zimmermann --- drivers/gpu/drm/drm_gem_framebuffer_helper.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/drm_gem_framebuffer_helper.c b/drivers/gpu/drm= /drm_gem_framebuffer_helper.c index 9166c353f..88808e972 100644 --- a/drivers/gpu/drm/drm_gem_framebuffer_helper.c +++ b/drivers/gpu/drm/drm_gem_framebuffer_helper.c @@ -172,8 +172,8 @@ int drm_gem_fb_init_with_funcs(struct drm_device *dev, } =20 for (i =3D 0; i < info->num_planes; i++) { - unsigned int width =3D mode_cmd->width / (i ? info->hsub : 1); - unsigned int height =3D mode_cmd->height / (i ? info->vsub : 1); + unsigned int width =3D drm_format_info_plane_width(info, mode_cmd->width= , i); + unsigned int height =3D drm_format_info_plane_height(info, mode_cmd->hei= ght, i); unsigned int min_size; =20 objs[i] =3D drm_gem_object_lookup(file, mode_cmd->handles[i]); --=20 2.34.1