arch/arm/Kconfig | 1 arch/arm64/Kconfig | 1 arch/loongarch/Kconfig | 1 arch/powerpc/platforms/powernv/Kconfig | 1 arch/powerpc/platforms/pseries/Kconfig | 1 arch/riscv/Kconfig | 1 arch/s390/Kconfig | 1 arch/x86/Kconfig | 2 - arch/x86/kernel/shstk.c | 47 +++++++++++------------------- drivers/android/binder_alloc.c | 39 ++++++------------------- fs/proc/internal.h | 2 - fs/proc/task_mmu.c | 51 --------------------------------- include/linux/mm.h | 12 ------- include/linux/mm_types.h | 7 ---- include/linux/mmap_lock.h | 50 +------------------------------- kernel/fork.c | 2 - mm/Kconfig | 13 -------- mm/mmap_lock.c | 45 +++++++++++++++++++++++++++-- net/ipv4/tcp.c | 31 +++++--------------- 19 files changed, 82 insertions(+), 226 deletions(-)
tl;dr: I hope I'm not missing something big here. The basic observastion here is that forcing code to account for per-VMA lock failure adds a lot of complexity. This series theorizes that with a some Kconfig changes and a new helper, many callers can avoid writing code that falls back to mmap_lock. -- When working on some x86 shadow stack code, it was a real pain to avoid causing recursive locking problems with mmap_lock. One way to avoid those was to avoid mmap_lock and use per-VMA locks instead. They are great, but they are not available in all configs which makes them unusable in generic code, or if you want to completely avoid mmap_lock. Make per-VMA locks available in all configs. Right now, they are only available on select architectures when SMP and MMU are enabled. But all of the primitives that per-VMA locks are built on (RCU, maple trees, refcounts) work just fine without SMP or MMU. Their only real downside is that they make VMAs a wee bit bigger on !MMU and !SMP builds. The upside is much cleaner code, lower complexity and less #ifdeffery. Building on top of universally-available per-VMA locks, introduce a new helper. Since the new API does not require callers to have a fallback to mmap_lock, it's much easier to use. Callers could potentially replace this very common kernel idiom: mmap_read_lock(mm); vma = vma_lookup() // fiddle with vma mmap_read_unlock(mm); with: vma = lock_vma_under_rcu_wait(mm, address); // fiddle with vma vma_end_read(vma); Which avoids mmap_lock entirely in the fast path. Things I think needs more discussion: * The new helper has a horrible name. Suggestions are very welcome. * I'm not very confident that this approach completely avoids the deadlock issues that arise from touching userspace while holding mm-related locks. * Can the helper avoid the goto, maybe by taking the VMA refcount while holding mmap_lock? * mm_struct and vm_area_struct "bloat" I've included a couple patches where I think the new helper really makes the code nicer. I'm keeping the cc list on the short side for now because I'm not actually proposing that we go ahead and do the ipv4 changes, for example. Cc: Suren Baghdasaryan <surenb@google.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: "Liam R. Howlett" <Liam.Howlett@oracle.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: linux-mm@kvack.org arch/arm/Kconfig | 1 arch/arm64/Kconfig | 1 arch/loongarch/Kconfig | 1 arch/powerpc/platforms/powernv/Kconfig | 1 arch/powerpc/platforms/pseries/Kconfig | 1 arch/riscv/Kconfig | 1 arch/s390/Kconfig | 1 arch/x86/Kconfig | 2 - arch/x86/kernel/shstk.c | 47 +++++++++++------------------- drivers/android/binder_alloc.c | 39 ++++++------------------- fs/proc/internal.h | 2 - fs/proc/task_mmu.c | 51 --------------------------------- include/linux/mm.h | 12 ------- include/linux/mm_types.h | 7 ---- include/linux/mmap_lock.h | 50 +------------------------------- kernel/fork.c | 2 - mm/Kconfig | 13 -------- mm/mmap_lock.c | 45 +++++++++++++++++++++++++++-- net/ipv4/tcp.c | 31 +++++--------------- 19 files changed, 82 insertions(+), 226 deletions(-)
BTW, this is *ENTIRELY* an [RFC]. It's just not tagged properly.
On Wed, Apr 29, 2026 at 11:22:28AM -0700, Dave Hansen wrote: > BTW, this is *ENTIRELY* an [RFC]. It's just not tagged properly. Was going to say :) Not going to be able to get to this until after LSF... :) Likely the same for Suren also. Cheers, Lorenzo
On Thu, Apr 30, 2026 at 1:11 AM Lorenzo Stoakes <ljs@kernel.org> wrote: > > On Wed, Apr 29, 2026 at 11:22:28AM -0700, Dave Hansen wrote: > > BTW, this is *ENTIRELY* an [RFC]. It's just not tagged properly. > > Was going to say :) > > Not going to be able to get to this until after LSF... :) Likely the same for > Suren also. Yeah, sorry. Trying to wrap up all the urgent stuff before the trip. I might be able to review the patches later this week before the conference starts, but can't promise. Thanks, Suren. > > Cheers, Lorenzo
On 4/30/26 10:17, Suren Baghdasaryan wrote: > On Thu, Apr 30, 2026 at 1:11 AM Lorenzo Stoakes <ljs@kernel.org> wrote: >> On Wed, Apr 29, 2026 at 11:22:28AM -0700, Dave Hansen wrote: >>> BTW, this is *ENTIRELY* an [RFC]. It's just not tagged properly. >> Was going to say 🙂 >> >> Not going to be able to get to this until after LSF... 🙂 Likely the same for >> Suren also. > Yeah, sorry. Trying to wrap up all the urgent stuff before the trip. I > might be able to review the patches later this week before the > conference starts, but can't promise. Seriously, don't worry about rushing this. After the conference is perfectly fine with me. There are a few things that Sashiko complained about that need to get fixed up anyway.
I'm guessing this is kinda an RFC? :P On Wed, Apr 29, 2026 at 11:19:54AM -0700, Dave Hansen wrote: > tl;dr: I hope I'm not missing something big here. The basic > observastion here is that forcing code to account for per-VMA lock > failure adds a lot of complexity. This series theorizes that with a > some Kconfig changes and a new helper, many callers can avoid writing > code that falls back to mmap_lock. In general very much in support of this! It'd be great to just know that this is available and frankly I think it's a critical part of the kernel. Obviously Suren needs to have a look through, most important of all :) > > -- > > When working on some x86 shadow stack code, it was a real pain to > avoid causing recursive locking problems with mmap_lock. One way > to avoid those was to avoid mmap_lock and use per-VMA locks instead. > They are great, but they are not available in all configs which > makes them unusable in generic code, or if you want to completely > avoid mmap_lock. Yeah, lock ordering is a pain. > > Make per-VMA locks available in all configs. Right now, they are > only available on select architectures when SMP and MMU are enabled. > But all of the primitives that per-VMA locks are built on (RCU, maple > trees, refcounts) work just fine without SMP or MMU. > > Their only real downside is that they make VMAs a wee bit bigger > on !MMU and !SMP builds. > > The upside is much cleaner code, lower complexity and less #ifdeffery. > > Building on top of universally-available per-VMA locks, introduce a > new helper. Since the new API does not require callers to have a > fallback to mmap_lock, it's much easier to use. Callers could > potentially replace this very common kernel idiom: > > mmap_read_lock(mm); > vma = vma_lookup() > // fiddle with vma > mmap_read_unlock(mm); > > with: > > vma = lock_vma_under_rcu_wait(mm, address); I will look at what you're proposing but this seems a bit like something I proposed at LSF (but was probably not the right solution for what was under discussion). Doing this 'right' would require quite a bit of engineering effort. The VMA locks are pretty bloody complicated :) so we have to be careful not to spread the complexity around too much. But I guess you could 'wait' by doing it in the slow path and then using vma_start_read_locked()... Of course that'd not help you with any lock inversions though! Anyway need to read the code :) > // fiddle with vma > vma_end_read(vma); > > Which avoids mmap_lock entirely in the fast path. Yeah it's nice! > > Things I think needs more discussion: > * The new helper has a horrible name. Suggestions are very welcome. > * I'm not very confident that this approach completely avoids the > deadlock issues that arise from touching userspace while holding > mm-related locks. Yeah we have to be careful... > * Can the helper avoid the goto, maybe by taking the VMA refcount > while holding mmap_lock? Surely that'd defeat the purpose of VMA locks though? you'd hold the mmap lock for less time but you're still contending vs. _any_ VMA write locks whilst trying to get a VMA read lock? Unless it's on a slow path... hmm :) > * mm_struct and vm_area_struct "bloat" Probably not a problem really. For any modern system you're using the fields. > > I've included a couple patches where I think the new helper really > makes the code nicer. > > I'm keeping the cc list on the short side for now because I'm not > actually proposing that we go ahead and do the ipv4 changes, for > example. Ack! > > Cc: Suren Baghdasaryan <surenb@google.com> > Cc: Andrew Morton <akpm@linux-foundation.org> > Cc: "Liam R. Howlett" <Liam.Howlett@oracle.com> > Cc: Lorenzo Stoakes <ljs@kernel.org> > Cc: Vlastimil Babka <vbabka@kernel.org> > Cc: Shakeel Butt <shakeel.butt@linux.dev> > Cc: linux-mm@kvack.org > > arch/arm/Kconfig | 1 > arch/arm64/Kconfig | 1 > arch/loongarch/Kconfig | 1 > arch/powerpc/platforms/powernv/Kconfig | 1 > arch/powerpc/platforms/pseries/Kconfig | 1 > arch/riscv/Kconfig | 1 > arch/s390/Kconfig | 1 > arch/x86/Kconfig | 2 - > arch/x86/kernel/shstk.c | 47 +++++++++++------------------- > drivers/android/binder_alloc.c | 39 ++++++------------------- > fs/proc/internal.h | 2 - > fs/proc/task_mmu.c | 51 --------------------------------- > include/linux/mm.h | 12 ------- > include/linux/mm_types.h | 7 ---- > include/linux/mmap_lock.h | 50 +------------------------------- > kernel/fork.c | 2 - > mm/Kconfig | 13 -------- > mm/mmap_lock.c | 45 +++++++++++++++++++++++++++-- > net/ipv4/tcp.c | 31 +++++--------------- > 19 files changed, 82 insertions(+), 226 deletions(-)
On Fri, May 8, 2026 at 9:53 AM Lorenzo Stoakes <ljs@kernel.org> wrote: > > I'm guessing this is kinda an RFC? :P > > On Wed, Apr 29, 2026 at 11:19:54AM -0700, Dave Hansen wrote: > > tl;dr: I hope I'm not missing something big here. The basic > > observastion here is that forcing code to account for per-VMA lock > > failure adds a lot of complexity. This series theorizes that with a > > some Kconfig changes and a new helper, many callers can avoid writing > > code that falls back to mmap_lock. > > In general very much in support of this! > > It'd be great to just know that this is available and frankly I think it's a > critical part of the kernel. > > Obviously Suren needs to have a look through, most important of all :) Yeah, I just got my head above the pile of emails and bugs that loudly met me upon arrival from the conference... I'll start reviewing the patchset first thing tomorrow morning. I apologize for the delay. Overall this seems like a good idea. When per-VMA locks were first introduced, the main and the only user was the pagefault path, so my first reaction to this proposal was, "Wait, why enable these locks on architectures that do not use them in their pagefault handling path?". But nowadays we use them more and more in other parts of the kernel, so making them universally available makes much more sense now. > > > > > -- > > > > When working on some x86 shadow stack code, it was a real pain to > > avoid causing recursive locking problems with mmap_lock. One way > > to avoid those was to avoid mmap_lock and use per-VMA locks instead. > > They are great, but they are not available in all configs which > > makes them unusable in generic code, or if you want to completely > > avoid mmap_lock. > > Yeah, lock ordering is a pain. > > > > > Make per-VMA locks available in all configs. Right now, they are > > only available on select architectures when SMP and MMU are enabled. > > But all of the primitives that per-VMA locks are built on (RCU, maple > > trees, refcounts) work just fine without SMP or MMU. > > > > Their only real downside is that they make VMAs a wee bit bigger > > on !MMU and !SMP builds. > > > > The upside is much cleaner code, lower complexity and less #ifdeffery. > > > > Building on top of universally-available per-VMA locks, introduce a > > new helper. Since the new API does not require callers to have a > > fallback to mmap_lock, it's much easier to use. Callers could > > potentially replace this very common kernel idiom: > > > > mmap_read_lock(mm); > > vma = vma_lookup() > > // fiddle with vma > > mmap_read_unlock(mm); > > > > with: > > > > vma = lock_vma_under_rcu_wait(mm, address); > > I will look at what you're proposing but this seems a bit like something I > proposed at LSF (but was probably not the right solution for what was under > discussion). > > Doing this 'right' would require quite a bit of engineering effort. The VMA > locks are pretty bloody complicated :) so we have to be careful not to spread > the complexity around too much. Yeah, I should look into cleaning things up and introducing some helpers to make the code more understandable. This might be the right time... > > But I guess you could 'wait' by doing it in the slow path and then using > vma_start_read_locked()... > > Of course that'd not help you with any lock inversions though! > > Anyway need to read the code :) > > > // fiddle with vma > > vma_end_read(vma); > > > > Which avoids mmap_lock entirely in the fast path. > > Yeah it's nice! > > > > > Things I think needs more discussion: > > * The new helper has a horrible name. Suggestions are very welcome. > > * I'm not very confident that this approach completely avoids the > > deadlock issues that arise from touching userspace while holding > > mm-related locks. > > Yeah we have to be careful... > > > * Can the helper avoid the goto, maybe by taking the VMA refcount > > while holding mmap_lock? > > Surely that'd defeat the purpose of VMA locks though? you'd hold the mmap lock > for less time but you're still contending vs. _any_ VMA write locks whilst > trying to get a VMA read lock? > > Unless it's on a slow path... hmm :) > > > * mm_struct and vm_area_struct "bloat" > > Probably not a problem really. For any modern system you're using the fields. Yeah, all major architectures already enable these locks. > > > > > I've included a couple patches where I think the new helper really > > makes the code nicer. > > > > I'm keeping the cc list on the short side for now because I'm not > > actually proposing that we go ahead and do the ipv4 changes, for > > example. > > Ack! > > > > > Cc: Suren Baghdasaryan <surenb@google.com> > > Cc: Andrew Morton <akpm@linux-foundation.org> > > Cc: "Liam R. Howlett" <Liam.Howlett@oracle.com> > > Cc: Lorenzo Stoakes <ljs@kernel.org> > > Cc: Vlastimil Babka <vbabka@kernel.org> > > Cc: Shakeel Butt <shakeel.butt@linux.dev> > > Cc: linux-mm@kvack.org > > > > arch/arm/Kconfig | 1 > > arch/arm64/Kconfig | 1 > > arch/loongarch/Kconfig | 1 > > arch/powerpc/platforms/powernv/Kconfig | 1 > > arch/powerpc/platforms/pseries/Kconfig | 1 > > arch/riscv/Kconfig | 1 > > arch/s390/Kconfig | 1 > > arch/x86/Kconfig | 2 - > > arch/x86/kernel/shstk.c | 47 +++++++++++------------------- > > drivers/android/binder_alloc.c | 39 ++++++------------------- > > fs/proc/internal.h | 2 - > > fs/proc/task_mmu.c | 51 --------------------------------- > > include/linux/mm.h | 12 ------- > > include/linux/mm_types.h | 7 ---- > > include/linux/mmap_lock.h | 50 +------------------------------- > > kernel/fork.c | 2 - > > mm/Kconfig | 13 -------- > > mm/mmap_lock.c | 45 +++++++++++++++++++++++++++-- > > net/ipv4/tcp.c | 31 +++++--------------- > > 19 files changed, 82 insertions(+), 226 deletions(-)
On 5/8/26 09:52, Lorenzo Stoakes wrote: ... >> * Can the helper avoid the goto, maybe by taking the VMA refcount >> while holding mmap_lock? > > Surely that'd defeat the purpose of VMA locks though? you'd hold the mmap lock > for less time but you're still contending vs. _any_ VMA write locks whilst > trying to get a VMA read lock? > > Unless it's on a slow path... hmm :) Yup. It's in a slow path. the example helper I had here does: retry: vma = lock_vma_under_rcu(mm, address); if (vma) return vma; mmap_read_lock(mm); vma = vma_lookup(mm, address); mmap_read_unlock(mm); goto retry; It avoids mmap_lock in the common, fast case. I was hoping to replace it with something like: vma = lock_vma_under_rcu(mm, address); if (vma) return vma; mmap_read_lock(mm); vma = vma_lookup(mm, address); vma_start_read(vma->vm_mm, vma); // Is this safe? mmap_read_unlock(mm); return vma; Which still uses mmap_lock, but avoids the goto. I'm pretty sure the first one doesn't have any locking problems. The second one, I need to think about a _lot_ more.
On Fri, May 8, 2026 at 10:01 AM Dave Hansen <dave.hansen@intel.com> wrote: > > On 5/8/26 09:52, Lorenzo Stoakes wrote: > ... > >> * Can the helper avoid the goto, maybe by taking the VMA refcount > >> while holding mmap_lock? > > > > Surely that'd defeat the purpose of VMA locks though? you'd hold the mmap lock > > for less time but you're still contending vs. _any_ VMA write locks whilst > > trying to get a VMA read lock? > > > > Unless it's on a slow path... hmm :) > > Yup. It's in a slow path. the example helper I had here does: > > retry: > vma = lock_vma_under_rcu(mm, address); > if (vma) > return vma; > mmap_read_lock(mm); > vma = vma_lookup(mm, address); > mmap_read_unlock(mm); > goto retry; > > It avoids mmap_lock in the common, fast case. I was hoping to replace it > with something like: > > vma = lock_vma_under_rcu(mm, address); > if (vma) > return vma; > mmap_read_lock(mm); > vma = vma_lookup(mm, address); > vma_start_read(vma->vm_mm, vma); // Is this safe? It is safe if you use vma_start_read_locked() instead. That fallback part is very similar to lock_next_vma_under_mmap_lock() but instead of locking the next VMA you are locking the VMA at a given address. uffd_lock_vma() is very close to the whole operation you described here. > mmap_read_unlock(mm); > return vma; > > Which still uses mmap_lock, but avoids the goto. I'm pretty sure the > first one doesn't have any locking problems. The second one, I need to > think about a _lot_ more. > >
syzbot ci has tested the following series [v1] mm: Make per-VMA locks available in all builds https://lore.kernel.org/all/20260429181954.F50224AE@davehans-spike.ostc.intel.com * [PATCH 1/6] mm: Make per-VMA locks available universally * [PATCH 2/6] binder: Make shrinker rely solely on per-VMA lock * [PATCH 3/6] mm: Add RCU-based VMA lookup that waits for writers * [PATCH 4/6] binder: Remove mmap_lock fallback * [PATCH 5/6] tcp: Remove mmap_lock fallback path * [PATCH 6/6] x86/mm: Avoid mmap lock for shadow stack pop fast path and found the following issue: WARNING in mbind_range Full report is available here: https://ci.syzbot.org/series/374f338e-4b3b-4645-871c-78964f944bbd *** WARNING in mbind_range tree: torvalds URL: https://kernel.googlesource.com/pub/scm/linux/kernel/git/torvalds/linux base: 57b8e2d666a31fa201432d58f5fe3469a0dd83ba arch: amd64 compiler: Debian clang version 21.1.8 (++20251221033036+2078da43e25a-1~exp1~20251221153213.50), Debian LLD 21.1.8 config: https://ci.syzbot.org/builds/786ac31f-0f5e-4ceb-88c7-45f4bee79d60/config syz repro: https://ci.syzbot.org/findings/4b270176-3e48-4ac7-8ddb-f326d6883d93/syz_repro pgoff 200000000 file 0000000000000000 private_data 0000000000000000 flags: 0x8100077(read|write|exec|mayread|maywrite|mayexec|account|softdirty) ------------[ cut here ]------------ 1 WARNING: ./include/linux/mmap_lock.h:332 at vma_assert_write_locked include/linux/mmap_lock.h:332 [inline], CPU#0: syz.2.19/5876 WARNING: ./include/linux/mmap_lock.h:332 at vma_replace_policy mm/mempolicy.c:1016 [inline], CPU#0: syz.2.19/5876 WARNING: ./include/linux/mmap_lock.h:332 at mbind_range+0x57a/0x810 mm/mempolicy.c:1063, CPU#0: syz.2.19/5876 Modules linked in: CPU: 0 UID: 0 PID: 5876 Comm: syz.2.19 Not tainted syzkaller #0 PREEMPT(full) Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 RIP: 0010:vma_assert_write_locked include/linux/mmap_lock.h:332 [inline] RIP: 0010:vma_replace_policy mm/mempolicy.c:1016 [inline] RIP: 0010:mbind_range+0x57a/0x810 mm/mempolicy.c:1063 Code: 97 ff e9 b2 fb ff ff e8 a4 4a 97 ff 90 0f 0b 90 e9 14 fe ff ff e8 96 4a 97 ff 4c 89 ff e8 0e b5 f9 fe c6 05 9c 2e ec 0d 01 90 <0f> 0b 90 4d 85 e4 0f 85 48 fe ff ff e8 75 4a 97 ff 31 db 4d 8d 77 RSP: 0018:ffffc90003bc7c78 EFLAGS: 00010292 RAX: 000000000000011f RBX: 000000000000000b RCX: a410c902b7e34800 RDX: 0000000000000000 RSI: 0000000080000000 RDI: 0000000000000000 RBP: 0000000000000009 R08: ffffc90003bc7967 R09: 1ffff92000778f2c R10: dffffc0000000000 R11: fffff52000778f2d R12: ffff8881012c9e00 R13: dffffc0000000000 R14: ffff888115b99bf8 R15: ffff8881162fa300 FS: 00007fa09187c6c0(0000) GS:ffff88818dc93000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000001b33b63fff CR3: 0000000115bda000 CR4: 00000000000006f0 Call Trace: <TASK> do_mbind mm/mempolicy.c:1560 [inline] kernel_mbind mm/mempolicy.c:1757 [inline] __do_sys_mbind mm/mempolicy.c:1831 [inline] __se_sys_mbind+0xad4/0x10f0 mm/mempolicy.c:1827 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline] do_syscall_64+0x15f/0xf80 arch/x86/entry/syscall_64.c:94 entry_SYSCALL_64_after_hwframe+0x77/0x7f RIP: 0033:0x7fa09099cdd9 Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 e8 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007fa09187c028 EFLAGS: 00000246 ORIG_RAX: 00000000000000ed RAX: ffffffffffffffda RBX: 00007fa090c15fa0 RCX: 00007fa09099cdd9 RDX: 0000000000000001 RSI: 0000000000600000 RDI: 0000200000000000 RBP: 00007fa090a32d69 R08: 0000000000000000 R09: 0000000000000003 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 R13: 00007fa090c16038 R14: 00007fa090c15fa0 R15: 00007ffff596d0f8 </TASK> *** If these findings have caused you to resend the series or submit a separate fix, please add the following tag to your commit message: Tested-by: syzbot@syzkaller.appspotmail.com --- This report is generated by a bot. It may contain errors. syzbot ci engineers can be reached at syzkaller@googlegroups.com. To test a patch for this bug, please reply with `#syz test` (should be on a separate line). The patch should be attached to the email. Note: arguments like custom git repos and branches are not supported.
On 4/30/26 00:55, syzbot ci wrote: > and found the following issue: > WARNING in mbind_range > > Full report is available here: > https://ci.syzbot.org/series/374f338e-4b3b-4645-871c-78964f944bbd > > *** > > WARNING in mbind_range I left a whole bunch of #ifdef CONFIG_PER_VMA_LOCK cruft around in v1. I suspect some of the debugging code ended up in a weird state in a config that I didn't test. I'll try to reproduce this splat, though. It looks straightforward enough.
© 2016 - 2026 Red Hat, Inc.