1. Patchew
  2. linux
  3. View series

[PATCH 0/1] Fix an imbalanced file ref count in the mmap syscall

Liav Albani posted 1 patch 6 days, 15 hours ago 
include/linux/mm_types.h |  1 +
mm/vma.c                 | 24 +++++++++++++-----------
2 files changed, 14 insertions(+), 11 deletions(-)
[PATCH 0/1] Fix an imbalanced file ref count in the mmap syscall
Posted by Liav Albani 6 days, 15 hours ago 
Hi!

This is my first patch that I contribute myself to the Linux kernel.
My apologize if there are issues, I tried to solve this issue in one
evening and understand the call graph and what could go wrong.

I work recently on a project of mine called nfiop, and specifically on a
kernel module called ufedm - https://github.com/nfiop/ufedm .

The project has a driver that allows a userspace implementation to
intervene in the write and read callbacks of an MTD device by spawning
an upper MTD device that sends the data back and forth to userspace, so
userspace can implement its custom ECC engine and send back the results.

So far so good. I started working with kernel 6.18.8 on an Olimex A20
board and then moved to my x86 linux machine with a nandsim simulation.
When I started my project, I implemented a memory backing based on a
shmem file for the shared memory buffer, that is expected to be mmap'ed
from my custom character device.

When I updated my host machine, the new kernel version was 7.x so I
decided to update my module as well.
Many changes were comsetic, but the major change was to implement a
mmap_prepare hook.

I wrote something simple, taking notes on the mmap_prepare rules and
the guide from kernel docs.

Then I started to test that my character device is at least working
fine, so I have my unit tests for ioctls and mmap, and the mmap
unit-test failed after the second-run.

A kernel stack looked like this:
```
imbalanced put on file reference count
WARNING: fs/file.c:36 at __file_ref_put_badval.isra.0+0x3f/0x60, CPU#1: test_proxy_mmap/116
Modules linked in: ufedm(OE) virtio_net net_failover failover nandsim nand nandcore bch mtd
CPU: 1 UID: 0 PID: 116 Comm: test_proxy_mmap Tainted: G           OE       7.2.0-rc3-gaf5e34a41cd6 #1 PREEMPT(full)  bdadc55b51524b311c430c1536a97e7071150fd3
Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Arch Linux 1.17.0-2-2 04/01/2014
RIP: 0010:__file_ref_put_badval.isra.0+0x3f/0x60
Code: 85 f6 78 05 c3 cc cc cc cc 48 b8 00 00 00 00 00 00 00 a0 48 89 07 c3 cc cc cc cc 48 83 ec 08 48 89 3c 24 48 8d 3d 61 bc 30 02 <67> 48 0f b9 3a 48 ba 00 00 00 00 00 00 00 e0 48 8b 04 24 48 89 10
RSP: 0018:ffffc900006bbae8 EFLAGS: 00010296
RAX: bfffffffffffffff RBX: ffff88810409a180 RCX: 0000000038748000
RDX: ffff888100a44480 RSI: dfffffffffffffff RDI: ffffffff83b12730
RBP: ffffc900006bbbf8 R08: 0000000000000000 R09: 00007f8b49f8f000
R10: ffffc900006bbba0 R11: ffff88810409a208 R12: ffffc900006bbba0
R13: 00007f8b4a179000 R14: ffff888102222440 R15: 0000000000015040
FS:  0000000038726400(0000) GS:ffff8881b77d1000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000038728618 CR3: 0000000104a1c006 CR4: 0000000000370ef0
Call Trace:
 <TASK>
 __file_ref_put+0x30/0x40
 fput+0x4f/0x80
 remove_vma+0x42/0x60
 vms_complete_munmap_vmas+0x179/0x230
 do_vmi_align_munmap+0x225/0x2a0
 do_vmi_munmap+0xe6/0x1c0
 __vm_munmap+0x113/0x200
 __x64_sys_munmap+0x1b/0x30
 do_syscall_64+0xaa/0x660
 ? ksys_mmap_pgoff+0x168/0x200
 ? do_syscall_64+0xaa/0x660
 ? vfs_write+0x281/0x560
 ? ksys_write+0x7b/0x110
 ? do_syscall_64+0xaa/0x660
 ? do_syscall_64+0x5f/0x660
 ? clear_bhb_loop+0x30/0x80
 ? clear_bhb_loop+0x30/0x80
 entry_SYSCALL_64_after_hwframe+0x76/0x7e
RIP: 0033:0x440bcb
```

I started investigating this, and it became apparent that there's an
issue with ref-counting of the vm_file - I changed it in my mmap_prepare
hook, so it must be the cause, right?

I also tried to artificially increment the ref-count of my shmem file
when I create it (taking another ref), so the second attempt didn't fail
but the third did...

I looked at the stack trace, and the culprit is the fact that we do fput
on a vm_file that we didn't pin - to fix this, I added a flag if we
actually did get_file() on the vm_file or not, on the vma struct.

When testing this patch, I've observed that the problem was completely
gone.

The mmap_prepare hook is still quite new, so presumably fixing corner
cases like this one is a good candidate for upstream, for future drivers
that might do the same as I do in my driver.

Liav Albani (1):
  mm/vma: fix imbalanced file reference count, properly abort
    mmap_prepare

 include/linux/mm_types.h |  1 +
 mm/vma.c                 | 24 +++++++++++++-----------
 2 files changed, 14 insertions(+), 11 deletions(-)

-- 
2.55.0
Re: [PATCH 0/1] Fix an imbalanced file ref count in the mmap syscall
Posted by Pedro Falcato 6 days, 14 hours ago 
On Sat, Jul 18, 2026 at 03:34:16PM +0300, Liav Albani wrote:
> Hi!
> 
> This is my first patch that I contribute myself to the Linux kernel.
> My apologize if there are issues, I tried to solve this issue in one
> evening and understand the call graph and what could go wrong.
> 
> I work recently on a project of mine called nfiop, and specifically on a
> kernel module called ufedm - https://github.com/nfiop/ufedm .
> 
> The project has a driver that allows a userspace implementation to
> intervene in the write and read callbacks of an MTD device by spawning
> an upper MTD device that sends the data back and forth to userspace, so
> userspace can implement its custom ECC engine and send back the results.
> 
> So far so good. I started working with kernel 6.18.8 on an Olimex A20
> board and then moved to my x86 linux machine with a nandsim simulation.
> When I started my project, I implemented a memory backing based on a
> shmem file for the shared memory buffer, that is expected to be mmap'ed
> from my custom character device.
> 
> When I updated my host machine, the new kernel version was 7.x so I
> decided to update my module as well.
> Many changes were comsetic, but the major change was to implement a
> mmap_prepare hook.
> 
> I wrote something simple, taking notes on the mmap_prepare rules and
> the guide from kernel docs.
> 
> Then I started to test that my character device is at least working
> fine, so I have my unit tests for ioctls and mmap, and the mmap
> unit-test failed after the second-run.
> 
> A kernel stack looked like this:
> ```
> imbalanced put on file reference count
> WARNING: fs/file.c:36 at __file_ref_put_badval.isra.0+0x3f/0x60, CPU#1: test_proxy_mmap/116
> Modules linked in: ufedm(OE) virtio_net net_failover failover nandsim nand nandcore bch mtd
> CPU: 1 UID: 0 PID: 116 Comm: test_proxy_mmap Tainted: G           OE       7.2.0-rc3-gaf5e34a41cd6 #1 PREEMPT(full)  bdadc55b51524b311c430c1536a97e7071150fd3
> Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Arch Linux 1.17.0-2-2 04/01/2014
> RIP: 0010:__file_ref_put_badval.isra.0+0x3f/0x60
> Code: 85 f6 78 05 c3 cc cc cc cc 48 b8 00 00 00 00 00 00 00 a0 48 89 07 c3 cc cc cc cc 48 83 ec 08 48 89 3c 24 48 8d 3d 61 bc 30 02 <67> 48 0f b9 3a 48 ba 00 00 00 00 00 00 00 e0 48 8b 04 24 48 89 10
> RSP: 0018:ffffc900006bbae8 EFLAGS: 00010296
> RAX: bfffffffffffffff RBX: ffff88810409a180 RCX: 0000000038748000
> RDX: ffff888100a44480 RSI: dfffffffffffffff RDI: ffffffff83b12730
> RBP: ffffc900006bbbf8 R08: 0000000000000000 R09: 00007f8b49f8f000
> R10: ffffc900006bbba0 R11: ffff88810409a208 R12: ffffc900006bbba0
> R13: 00007f8b4a179000 R14: ffff888102222440 R15: 0000000000015040
> FS:  0000000038726400(0000) GS:ffff8881b77d1000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000000038728618 CR3: 0000000104a1c006 CR4: 0000000000370ef0
> Call Trace:
>  <TASK>
>  __file_ref_put+0x30/0x40
>  fput+0x4f/0x80
>  remove_vma+0x42/0x60
>  vms_complete_munmap_vmas+0x179/0x230
>  do_vmi_align_munmap+0x225/0x2a0
>  do_vmi_munmap+0xe6/0x1c0
>  __vm_munmap+0x113/0x200
>  __x64_sys_munmap+0x1b/0x30
>  do_syscall_64+0xaa/0x660
>  ? ksys_mmap_pgoff+0x168/0x200
>  ? do_syscall_64+0xaa/0x660
>  ? vfs_write+0x281/0x560
>  ? ksys_write+0x7b/0x110
>  ? do_syscall_64+0xaa/0x660
>  ? do_syscall_64+0x5f/0x660
>  ? clear_bhb_loop+0x30/0x80
>  ? clear_bhb_loop+0x30/0x80
>  entry_SYSCALL_64_after_hwframe+0x76/0x7e
> RIP: 0033:0x440bcb
> ```
> 
> I started investigating this, and it became apparent that there's an
> issue with ref-counting of the vm_file - I changed it in my mmap_prepare
> hook, so it must be the cause, right?
> 
> I also tried to artificially increment the ref-count of my shmem file

You need to take a ref when doing the mmap_prepare file setting. mmap (and VMAs)
consumes one reference, thus need to pin the file. Thus why the ref count
manipulation around this whole code...

So, in your case, something like:

	get_file(dev->shm_mapping.filp);

in your proxy_chrdev_mmap_prepare looks correct. Otherwise, something else
could sweep the file under the VMA. Note that this is true for every mmapped
file, even something like /dev/zero (which does what you're doing, creates
a shmem file and changes the VMA desc) implicitly donates the reference to
mmap, which will later be unref'd (and the shmem file torn down) on munmap.

Let me know if I hugely misunderstood your problem :)

-- 
Pedro
Re: [PATCH 0/1] Fix an imbalanced file ref count in the mmap syscall
Posted by Lorenzo Stoakes (ARM) 6 days, 13 hours ago 
On Sat, Jul 18, 2026 at 02:25:15PM +0100, Pedro Falcato wrote:
> On Sat, Jul 18, 2026 at 03:34:16PM +0300, Liav Albani wrote:
> > Hi!
> >
> > This is my first patch that I contribute myself to the Linux kernel.
> > My apologize if there are issues, I tried to solve this issue in one
> > evening and understand the call graph and what could go wrong.
> >
> > I work recently on a project of mine called nfiop, and specifically on a
> > kernel module called ufedm - https://github.com/nfiop/ufedm .
> >
> > The project has a driver that allows a userspace implementation to
> > intervene in the write and read callbacks of an MTD device by spawning
> > an upper MTD device that sends the data back and forth to userspace, so
> > userspace can implement its custom ECC engine and send back the results.
> >
> > So far so good. I started working with kernel 6.18.8 on an Olimex A20
> > board and then moved to my x86 linux machine with a nandsim simulation.
> > When I started my project, I implemented a memory backing based on a
> > shmem file for the shared memory buffer, that is expected to be mmap'ed
> > from my custom character device.
> >
> > When I updated my host machine, the new kernel version was 7.x so I
> > decided to update my module as well.
> > Many changes were comsetic, but the major change was to implement a
> > mmap_prepare hook.
> >
> > I wrote something simple, taking notes on the mmap_prepare rules and
> > the guide from kernel docs.
> >
> > Then I started to test that my character device is at least working
> > fine, so I have my unit tests for ioctls and mmap, and the mmap
> > unit-test failed after the second-run.
> >
> > A kernel stack looked like this:
> > ```
> > imbalanced put on file reference count
> > WARNING: fs/file.c:36 at __file_ref_put_badval.isra.0+0x3f/0x60, CPU#1: test_proxy_mmap/116
> > Modules linked in: ufedm(OE) virtio_net net_failover failover nandsim nand nandcore bch mtd
> > CPU: 1 UID: 0 PID: 116 Comm: test_proxy_mmap Tainted: G           OE       7.2.0-rc3-gaf5e34a41cd6 #1 PREEMPT(full)  bdadc55b51524b311c430c1536a97e7071150fd3
> > Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
> > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Arch Linux 1.17.0-2-2 04/01/2014
> > RIP: 0010:__file_ref_put_badval.isra.0+0x3f/0x60
> > Code: 85 f6 78 05 c3 cc cc cc cc 48 b8 00 00 00 00 00 00 00 a0 48 89 07 c3 cc cc cc cc 48 83 ec 08 48 89 3c 24 48 8d 3d 61 bc 30 02 <67> 48 0f b9 3a 48 ba 00 00 00 00 00 00 00 e0 48 8b 04 24 48 89 10
> > RSP: 0018:ffffc900006bbae8 EFLAGS: 00010296
> > RAX: bfffffffffffffff RBX: ffff88810409a180 RCX: 0000000038748000
> > RDX: ffff888100a44480 RSI: dfffffffffffffff RDI: ffffffff83b12730
> > RBP: ffffc900006bbbf8 R08: 0000000000000000 R09: 00007f8b49f8f000
> > R10: ffffc900006bbba0 R11: ffff88810409a208 R12: ffffc900006bbba0
> > R13: 00007f8b4a179000 R14: ffff888102222440 R15: 0000000000015040
> > FS:  0000000038726400(0000) GS:ffff8881b77d1000(0000) knlGS:0000000000000000
> > CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > CR2: 0000000038728618 CR3: 0000000104a1c006 CR4: 0000000000370ef0
> > Call Trace:
> >  <TASK>
> >  __file_ref_put+0x30/0x40
> >  fput+0x4f/0x80
> >  remove_vma+0x42/0x60
> >  vms_complete_munmap_vmas+0x179/0x230
> >  do_vmi_align_munmap+0x225/0x2a0
> >  do_vmi_munmap+0xe6/0x1c0
> >  __vm_munmap+0x113/0x200
> >  __x64_sys_munmap+0x1b/0x30
> >  do_syscall_64+0xaa/0x660
> >  ? ksys_mmap_pgoff+0x168/0x200
> >  ? do_syscall_64+0xaa/0x660
> >  ? vfs_write+0x281/0x560
> >  ? ksys_write+0x7b/0x110
> >  ? do_syscall_64+0xaa/0x660
> >  ? do_syscall_64+0x5f/0x660
> >  ? clear_bhb_loop+0x30/0x80
> >  ? clear_bhb_loop+0x30/0x80
> >  entry_SYSCALL_64_after_hwframe+0x76/0x7e
> > RIP: 0033:0x440bcb
> > ```
> >
> > I started investigating this, and it became apparent that there's an
> > issue with ref-counting of the vm_file - I changed it in my mmap_prepare
> > hook, so it must be the cause, right?
> >
> > I also tried to artificially increment the ref-count of my shmem file
>
> You need to take a ref when doing the mmap_prepare file setting. mmap (and VMAs)
> consumes one reference, thus need to pin the file. Thus why the ref count
> manipulation around this whole code...
>
> So, in your case, something like:
>
> 	get_file(dev->shm_mapping.filp);
>
> in your proxy_chrdev_mmap_prepare looks correct. Otherwise, something else
> could sweep the file under the VMA. Note that this is true for every mmapped
> file, even something like /dev/zero (which does what you're doing, creates
> a shmem file and changes the VMA desc) implicitly donates the reference to
> mmap, which will later be unref'd (and the shmem file torn down) on munmap.
>
> Let me know if I hugely misunderstood your problem :)
>
> --
> Pedro

Yeah I agree. Thanks for the contribution and your interest in this part of
the kernel :) but any file replacing an existing one must be pinned as it
is then used by the mmap and the pin signifies that.

Perhaps a patch to the documentation could be useful here? In
Documentation/filesystems/mmap_prepare.rst

If you found it confusing that that implies it's not clear, so something
there saying 'if you replace the file, you MUST pin it via get_file()
before assigning desc->vm_file'.

Thanks, Lorenzo
[syzbot ci] Re: Fix an imbalanced file ref count in the mmap syscall
Posted by syzbot ci 6 days, 8 hours ago 
syzbot ci has tested the following series

[v1] Fix an imbalanced file ref count in the mmap syscall
https://lore.kernel.org/all/20260718123417.16339-1-liavalb@gmail.com
* [PATCH] mm/vma: fix imbalanced file reference count, properly abort mmap_prepare

and found the following issue:
WARNING in tear_down_vmas

Full report is available here:
https://ci.syzbot.org/series/63d42f9f-f249-4678-8916-5464a15accd6

***

WARNING in tear_down_vmas

tree:      mm-new
URL:       https://kernel.googlesource.com/pub/scm/linux/kernel/git/akpm/mm.git
base:      3827fe994db98623a5a619a72f15f57bf8274deb
arch:      amd64
compiler:  Debian clang version 22.1.6 (++20260514074242+fc4aad7b5db3-1~exp1~20260514074407.73), Debian LLD 22.1.6
config:    https://ci.syzbot.org/builds/f57e989a-6273-4413-b35d-20dab5a41e92/config
syz repro: https://ci.syzbot.org/findings/8f36751f-0976-460a-b5ab-de97c9074c0a/syz_repro

------------[ cut here ]------------
!vma_is_attached(vma)
WARNING: ./include/linux/mmap_lock.h:435 at vma_assert_attached include/linux/mmap_lock.h:435 [inline], CPU#0: syz.2.19/5813
WARNING: ./include/linux/mmap_lock.h:435 at vma_mark_detached include/linux/mmap_lock.h:455 [inline], CPU#0: syz.2.19/5813
WARNING: ./include/linux/mmap_lock.h:435 at tear_down_vmas+0x381/0x520 mm/mmap.c:1261, CPU#0: syz.2.19/5813
Modules linked in:
CPU: 0 UID: 0 PID: 5813 Comm: syz.2.19 Not tainted syzkaller #0 PREEMPT(full) 
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
RIP: 0010:vma_assert_attached include/linux/mmap_lock.h:435 [inline]
RIP: 0010:vma_mark_detached include/linux/mmap_lock.h:455 [inline]
RIP: 0010:tear_down_vmas+0x381/0x520 mm/mmap.c:1261
Code: 48 89 df e8 01 46 16 00 48 8b 44 24 18 48 39 03 0f 87 e0 00 00 00 e8 ce a8 a8 ff ff 44 24 04 e9 9a fd ff ff e8 c0 a8 a8 ff 90 <0f> 0b 90 e9 2c ff ff ff e8 b2 a8 a8 ff 90 0f 0b 90 e9 94 fe ff ff
RSP: 0018:ffffc90003cdf7e0 EFLAGS: 00010293
RAX: ffffffff821dbb90 RBX: ffff888115e14b40 RCX: ffff888171f9bb80
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: ffffc90003cdf920 R08: ffff888115e14b43 R09: 1ffff11022bc2968
R10: dffffc0000000000 R11: ffffed1022bc2969 R12: ffff888115e14a80
R13: 0000000000000001 R14: 0000000000000000 R15: dffffc0000000000
FS:  00007f15b88e36c0(0000) GS:ffff88818dc1c000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9033eab095 CR3: 0000000173ace000 CR4: 00000000000006f0
Call Trace:
 <TASK>
 exit_mmap+0x4ca/0x9f0 mm/mmap.c:1320
 __mmput+0x118/0x420 kernel/fork.c:1187
 exit_mm+0x221/0x2d0 kernel/exit.c:610
 do_exit+0x6cd/0x2360 kernel/exit.c:992
 do_group_exit+0x22d/0x2f0 kernel/exit.c:1147
 get_signal+0x121b/0x12c0 kernel/signal.c:3038
 arch_do_signal_or_restart+0xbb/0x860 arch/x86/kernel/signal.c:337
 __exit_to_user_mode_loop kernel/entry/common.c:66 [inline]
 exit_to_user_mode_loop+0x104/0x730 kernel/entry/common.c:101
 __exit_to_user_mode_prepare include/linux/irq-entry-common.h:207 [inline]
 syscall_exit_to_user_mode_prepare include/linux/irq-entry-common.h:230 [inline]
 syscall_exit_to_user_mode include/linux/entry-common.h:318 [inline]
 do_syscall_64+0x353/0x580 arch/x86/entry/syscall_64.c:100
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f15b799ce59
Code: Unable to access opcode bytes at 0x7f15b799ce2f.
RSP: 002b:00007f15b88e30e8 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
RAX: fffffffffffffe00 RBX: 00007f15b7c15fa8 RCX: 00007f15b799ce59
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 00007f15b7c15fa8
RBP: 00007f15b7c15fa0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00007f15b7c16038 R14: 00007ffe1b61d9f0 R15: 00007ffe1b61dad8
 </TASK>


***

If these findings have caused you to resend the series or submit a
separate fix, please add the following tag to your commit message:
  Tested-by: syzbot@syzkaller.appspotmail.com

---
This report is generated by a bot. It may contain errors.
syzbot ci engineers can be reached at syzkaller@googlegroups.com.

To test a patch for this bug, please reply with `#syz test`
(should be on a separate line).

The patch should be attached to the email.
Note: arguments like custom git repos and branches are not supported.

Patchew 2.1-devel-ea86937

© 2016 - 2026 Red Hat, Inc.