The ro variable, being of type bool, caused the -EROFS return value from
mnt_want_write() to be implicitly converted to 1. This prevented the file
from being correctly acquired, thus triggering the issue reported by
syzbot [1].

Changing the type of ro to int allows the system to correctly identify
the reason for the file open failure.

[1]
KASAN: null-ptr-deref in range [0x0000000000000040-0x0000000000000047]
Call Trace:
 do_mq_open+0x5a0/0x770 ipc/mqueue.c:932
 __do_sys_mq_open ipc/mqueue.c:945 [inline]
 __se_sys_mq_open ipc/mqueue.c:938 [inline]
 __x64_sys_mq_open+0x16a/0x1c0 ipc/mqueue.c:938

Fixes: f2573685bd0c ("ipc: convert do_mq_open() to FD_ADD()")
Reported-by: syzbot+40f42779048f7476e2e0@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=40f42779048f7476e2e0
Tested-by: syzbot+40f42779048f7476e2e0@syzkaller.appspotmail.com
Signed-off-by: Edward Adam Davis <eadavis@qq.com>
---
 ipc/mqueue.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ipc/mqueue.c b/ipc/mqueue.c
index 56e811f9e5fa..90664d26ec07 100644
--- a/ipc/mqueue.c
+++ b/ipc/mqueue.c
@@ -893,7 +893,7 @@ static int prepare_open(struct dentry *dentry, int oflag, int ro,
 }
 
 static struct file *mqueue_file_open(struct filename *name,
-				     struct vfsmount *mnt, int oflag, bool ro,
+				     struct vfsmount *mnt, int oflag, int ro,
 				     umode_t mode, struct mq_attr *attr)
 {
 	struct dentry *dentry;
-- 
2.43.0

On Thu, 04 Dec 2025 21:16:22 +0800, Edward Adam Davis wrote:
> The ro variable, being of type bool, caused the -EROFS return value from
> mnt_want_write() to be implicitly converted to 1. This prevented the file
> from being correctly acquired, thus triggering the issue reported by
> syzbot [1].
> 
> Changing the type of ro to int allows the system to correctly identify
> the reason for the file open failure.
> 
> [...]

Applied to the vfs.fixes branch of the vfs/vfs.git tree.
Patches in the vfs.fixes branch should appear in linux-next soon.

Please report any outstanding bugs that were missed during review in a
new review to the original patch series allowing us to drop it.

It's encouraged to provide Acked-bys and Reviewed-bys even though the
patch has now been applied. If possible patch trailers will be updated.

Note that commit hashes shown below are subject to change due to rebase,
trailer updates or similar. If in doubt, please check the listed branch.

tree:   https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git
branch: vfs.fixes

[1/1] mqueue: correct the type of ro to int
      https://git.kernel.org/vfs/vfs/c/1d98eeadbc10

On Thu 04-12-25 21:16:22, Edward Adam Davis wrote:
> The ro variable, being of type bool, caused the -EROFS return value from
> mnt_want_write() to be implicitly converted to 1. This prevented the file
> from being correctly acquired, thus triggering the issue reported by
> syzbot [1].
> 
> Changing the type of ro to int allows the system to correctly identify
> the reason for the file open failure.
> 
> [1]
> KASAN: null-ptr-deref in range [0x0000000000000040-0x0000000000000047]
> Call Trace:
>  do_mq_open+0x5a0/0x770 ipc/mqueue.c:932
>  __do_sys_mq_open ipc/mqueue.c:945 [inline]
>  __se_sys_mq_open ipc/mqueue.c:938 [inline]
>  __x64_sys_mq_open+0x16a/0x1c0 ipc/mqueue.c:938
> 
> Fixes: f2573685bd0c ("ipc: convert do_mq_open() to FD_ADD()")
> Reported-by: syzbot+40f42779048f7476e2e0@syzkaller.appspotmail.com
> Closes: https://syzkaller.appspot.com/bug?extid=40f42779048f7476e2e0
> Tested-by: syzbot+40f42779048f7476e2e0@syzkaller.appspotmail.com
> Signed-off-by: Edward Adam Davis <eadavis@qq.com>

Ah, indeed. mqueue_file_open() was returning ERR_PTR(1) which was confusing
FD_ADD error handling. Thanks for debugging this! Feel free to add:

Reviewed-by: Jan Kara <jack@suse.cz>

								Honza

> ---
>  ipc/mqueue.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/ipc/mqueue.c b/ipc/mqueue.c
> index 56e811f9e5fa..90664d26ec07 100644
> --- a/ipc/mqueue.c
> +++ b/ipc/mqueue.c
> @@ -893,7 +893,7 @@ static int prepare_open(struct dentry *dentry, int oflag, int ro,
>  }
>  
>  static struct file *mqueue_file_open(struct filename *name,
> -				     struct vfsmount *mnt, int oflag, bool ro,
> +				     struct vfsmount *mnt, int oflag, int ro,
>  				     umode_t mode, struct mq_attr *attr)
>  {
>  	struct dentry *dentry;
> -- 
> 2.43.0
> 
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR