[PATCH v2 00/10] LUO: PCI subsystem (phase I)

Chris Li posted 10 patches 4 months, 3 weeks ago
MAINTAINERS                    |   4 +
drivers/pci/Kconfig            |  10 +
drivers/pci/Makefile           |   2 +
drivers/pci/liveupdate.c       | 450 +++++++++++++++++++++++++++++++++++++++++
drivers/pci/pci-lu-stub.c      | 140 +++++++++++++
drivers/pci/pci.c              |   7 +-
drivers/pci/pci.h              |   8 +
drivers/pci/pcie/portdrv.c     |  13 ++
drivers/pci/probe.c            |   8 +-
include/linux/dev_liveupdate.h |  69 +++++++
include/linux/device.h         |  15 ++
include/linux/device/driver.h  |   6 +
include/linux/pci.h            |   9 +
13 files changed, 738 insertions(+), 3 deletions(-)
[PATCH v2 00/10] LUO: PCI subsystem (phase I)
Posted by Chris Li 4 months, 3 weeks ago
This is phase I of the LUO PCI series. It does the minimal set of PCI
device liveupdate which is preserving a bus master bit in the PCI command
register.

The LUO PCI subsystem is based on the LUO V2 series.
https://lore.kernel.org/lkml/20250515182322.117840-1-pasha.tatashin@soleen.com/

It registers the PCI as a LUO subsystem and forwards the liveupdate
callback to the device. The struct dev_liveupdate has been add to struct
device to keep track of the liveupdate related context.

A device can be marked as requested for liveupdate during the normal
state.

In the prepare() callback. The PCI core will build a list of the PCI device
for liveupdate based on the PCI device dependency:
1) The requested device is dependent on the PCI bridge it is on to preserve
   the bridge bus master. All the way to the root bridge. If the bus master
   has been disabled on the bridge, the DMA on the children devices will
   get impacted.

The list of liveupdate devices is used for prepare(), cancel(), freeze()
and finalized() callback.

The PCI subsystem will preserve the driver name for each liveupdate PCI
device and only probe that driver after kexec boot up.

Disclaimer:
The data preservation format is not final. It currently uses C struct
directly. It does not deal with version change on the data format yet. I
do have some idea how to address the versioning of data layout. Those
will be outside the scope of this series.

Testing:
Testing was done with Intel diorite NVMe VF device 8086:1457. Bind the
test device with pci-lu-stub driver.

0000:05:00.1 current driver is
0000:05:00.1 bind new driver pci-lu-stub
[  557.006998] pci-lu-stub 0000:05:00.1: Marking device liveupdate busmaster

Now perform luo prepare, the PCI subsystem builds the liveupdate device
list from the PCI root bridge. The test device will have LU_BUSMASTER
and the PCI bridge will have LU_BUSMASTER_BRIDGE.

[  701.573423] pci-lu-stub 0000:05:00.1: PCI liveupdate: collect liveupdate device: flags 1
[  701.582430] pcieport 0000:04:01.0: PCI liveupdate: collect liveupdate device: flags 2
[  701.590297] pci-lu-stub 0000:05:00.1: pci_lu_stub_prepare(): data: 0x1ac6f4000
[  701.598916] PCI liveupdate: prepare data[1f1d28000]
[  701.603832] luo_core: Switched from [normal] to [prepared] state

After kexec reboot. The liveupdate devices are probed and restores the live
update context.
[    3.622083] pci 0000:04:01.0: PCI liveupdate: liveupdate restore flags 2 driver: pcieport data: [0]
[    4.768060] pci 0000:05:00.1: PCI liveupdate: liveupdate restore flags 1 driver: pci-lu-stub data: [1ac6f4000]

Perform luo finish to convert from update state to normal state. The
reserved folio will be freed.

[  310.359830] PCI liveupdate: finish data[1f1d28000]
[  310.364664] pci-lu-stub 0000:05:00.1: pci_lu_stub_finish(): data: 0x1ac6f4000
[  310.371824] luo_core: Switched from [updated] to [normal] state

Signed-off-by: Chris Li <chrisl@kernel.org>
---
Changes in v2:
- reduce the scope of the series to phase I. Only preserve the bus
  master bit.
- Use finer grain flags to specify which liveupdate feature gets
  preserved.
- Modify the pci-lu-stub driver to set the bus master bit before
  requesting preserving the bus master.
- Add WARN_ON() for the PCI device has LU_BUSMASTER but the bus master
  bit is not set.
- Link to v1: https://lore.kernel.org/r/20250728-luo-pci-v1-0-955b078dd653@kernel.org

---
Chris Li (10):
      PCI/LUO: Register with Liveupdate Orchestrator
      PCI/LUO: Create requested liveupdate device list
      PCI/LUO: Forward prepare()/freeze()/cancel() callbacks to driver
      PCI/LUO: Restore state at PCI enumeration
      PCI/LUO: Forward finish callbacks to drivers
      PCI/LUO: Save and restore driver name
      PCI/LUO: Add liveupdate to pcieport driver
      PCI/LUO: Add pci_liveupdate_get_driver_data()
      PCI/LUO: Avoid write to bus master at boot
      PCI: pci-lu-stub: Add a stub driver for Live Update testing

 MAINTAINERS                    |   4 +
 drivers/pci/Kconfig            |  10 +
 drivers/pci/Makefile           |   2 +
 drivers/pci/liveupdate.c       | 450 +++++++++++++++++++++++++++++++++++++++++
 drivers/pci/pci-lu-stub.c      | 140 +++++++++++++
 drivers/pci/pci.c              |   7 +-
 drivers/pci/pci.h              |   8 +
 drivers/pci/pcie/portdrv.c     |  13 ++
 drivers/pci/probe.c            |   8 +-
 include/linux/dev_liveupdate.h |  69 +++++++
 include/linux/device.h         |  15 ++
 include/linux/device/driver.h  |   6 +
 include/linux/pci.h            |   9 +
 13 files changed, 738 insertions(+), 3 deletions(-)
---
base-commit: 9ab803064e3d1be9673d2829785a69fd0578b24e
change-id: 20250724-luo-pci-1291890b710f

Best regards,
-- 
Chris Li <chrisl@kernel.org>
Re: [PATCH v2 00/10] LUO: PCI subsystem (phase I)
Posted by Bjorn Helgaas 4 months, 1 week ago
On Tue, Sep 16, 2025 at 12:45:08AM -0700, Chris Li wrote:
> This is phase I of the LUO PCI series. It does the minimal set of PCI
> device liveupdate which is preserving a bus master bit in the PCI command
> register.
> 
> The LUO PCI subsystem is based on the LUO V2 series.
> https://lore.kernel.org/lkml/20250515182322.117840-1-pasha.tatashin@soleen.com/

Pasha's email points to
https://github.com/googleprodkernel/linux-liveupdate/tree/luo/rfc-v2,
so I cloned https://github.com/googleprodkernel/linux-liveupdate.git
and tried to apply this series on top of the luo/rfc-v2 branch (head
5c8d261fdc15 ("MAINTAINERS: add liveupdate entry")), but it doesn't
apply cleanly.

Also tried the luo/v2 branch (head 75716df00a94 ("libluo: add
tests")), but it doesn't apply there either.

Am I looking the wrong place?  Do you have a public repo with this
series in it?

Bjorn
Re: [PATCH v2 00/10] LUO: PCI subsystem (phase I)
Posted by Pasha Tatashin 4 months, 1 week ago
Hi Bjorn,

My latest submission is the following:
https://lore.kernel.org/all/20250807014442.3829950-1-pasha.tatashin@soleen.com/

And github repo is in cover letter:

https://github.com/googleprodkernel/linux-liveupdate/tree/luo/v3

It applies cleanly against the mainline without the first three
patches, as they were already merged.

Pasha

On Sat, Sep 27, 2025 at 1:13 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> On Tue, Sep 16, 2025 at 12:45:08AM -0700, Chris Li wrote:
> > This is phase I of the LUO PCI series. It does the minimal set of PCI
> > device liveupdate which is preserving a bus master bit in the PCI command
> > register.
> >
> > The LUO PCI subsystem is based on the LUO V2 series.
> > https://lore.kernel.org/lkml/20250515182322.117840-1-pasha.tatashin@soleen.com/
>
> Pasha's email points to
> https://github.com/googleprodkernel/linux-liveupdate/tree/luo/rfc-v2,
> so I cloned https://github.com/googleprodkernel/linux-liveupdate.git
> and tried to apply this series on top of the luo/rfc-v2 branch (head
> 5c8d261fdc15 ("MAINTAINERS: add liveupdate entry")), but it doesn't
> apply cleanly.
>
> Also tried the luo/v2 branch (head 75716df00a94 ("libluo: add
> tests")), but it doesn't apply there either.
>
> Am I looking the wrong place?  Do you have a public repo with this
> series in it?
>
> Bjorn
Re: [PATCH v2 00/10] LUO: PCI subsystem (phase I)
Posted by Bjorn Helgaas 4 months, 1 week ago
On Sat, Sep 27, 2025 at 02:05:38PM -0400, Pasha Tatashin wrote:
> Hi Bjorn,
> 
> My latest submission is the following:
> https://lore.kernel.org/all/20250807014442.3829950-1-pasha.tatashin@soleen.com/
> 
> And github repo is in cover letter:
> 
> https://github.com/googleprodkernel/linux-liveupdate/tree/luo/v3
> 
> It applies cleanly against the mainline without the first three
> patches, as they were already merged.

Not sure what I'm missing.  I've tried various things but none apply
cleanly:

  $ git remote add luo https://github.com/googleprodkernel/linux-liveupdate.git
  $ git fetch luo
  From https://github.com/googleprodkernel/linux-liveupdate
   * [new branch]                hack_pci_pf_stub_demo -> luo/hack_pci_pf_stub_demo
   * [new branch]                iommu/rfc-v1          -> luo/iommu/rfc-v1
   * [new branch]                kho/v5                -> luo/kho/v5
   * [new branch]                kho/v6                -> luo/kho/v6
   * [new branch]                kho/v7                -> luo/kho/v7
   * [new branch]                kho/v8                -> luo/kho/v8
   * [new branch]                lucx/v1               -> luo/lucx/v1
   * [new branch]                luo/kho-v8            -> luo/luo/kho-v8
   * [new branch]                luo/memfd-v0.1        -> luo/luo/memfd-v0.1
   * [new branch]                luo/rfc-v1            -> luo/luo/rfc-v1
   * [new branch]                luo/rfc-v2            -> luo/luo/rfc-v2
   * [new branch]                luo/v1                -> luo/luo/v1
   * [new branch]                luo/v2                -> luo/luo/v2
   * [new branch]                luo/v3                -> luo/luo/v3
   * [new branch]                luo/v4                -> luo/luo/v4
   * [new branch]                master                -> luo/master

  $ b4 am -om/ https://lore.kernel.org/r/20250916-luo-pci-v2-0-c494053c3c08@kernel.org
  Grabbing thread from lore.kernel.org/all/20250916-luo-pci-v2-0-c494053c3c08@kernel.org/t.mbox.gz
  Analyzing 13 messages in the thread
  Looking for additional code-review trailers on lore.kernel.org
  Checking attestation on all messages, may take a moment...
  ---
    ✓ [PATCH v2 1/10] PCI/LUO: Register with Liveupdate Orchestrator
    ✓ [PATCH v2 2/10] PCI/LUO: Create requested liveupdate device list
    ✓ [PATCH v2 3/10] PCI/LUO: Forward prepare()/freeze()/cancel() callbacks to driver
    ✓ [PATCH v2 4/10] PCI/LUO: Restore state at PCI enumeration
    ✓ [PATCH v2 5/10] PCI/LUO: Forward finish callbacks to drivers
    ✓ [PATCH v2 6/10] PCI/LUO: Save and restore driver name
    ✓ [PATCH v2 7/10] PCI/LUO: Add liveupdate to pcieport driver
    ✓ [PATCH v2 8/10] PCI/LUO: Add pci_liveupdate_get_driver_data()
    ✓ [PATCH v2 9/10] PCI/LUO: Avoid write to bus master at boot
    ✓ [PATCH v2 10/10] PCI: pci-lu-stub: Add a stub driver for Live Update testing
    ---
    ✓ Signed: DKIM/kernel.org
  ---
  Total patches: 10
  ---
  Cover: m/v2_20250916_chrisl_luo_pci_subsystem_phase_i.cover
   Link: https://lore.kernel.org/r/20250916-luo-pci-v2-0-c494053c3c08@kernel.org
   Base: base-commit 9ab803064e3d1be9673d2829785a69fd0578b24e not known, ignoring
   Base: not specified
	 git am m/v2_20250916_chrisl_luo_pci_subsystem_phase_i.mbx

  $ git checkout -b wip/2509-chris-luo-pci-v2 luo/luo/rfc-v2; git am m/v2_20250916_chrisl_luo_pci_subsystem_phase_i.mbx
  Updating files: 100% (21294/21294), done.
  branch 'wip/2509-chris-luo-pci-v2' set up to track 'luo/luo/rfc-v2'.
  Switched to a new branch 'wip/2509-chris-luo-pci-v2'
  Applying: PCI/LUO: Register with Liveupdate Orchestrator
  Applying: PCI/LUO: Create requested liveupdate device list
  Applying: PCI/LUO: Forward prepare()/freeze()/cancel() callbacks to driver
  Applying: PCI/LUO: Restore state at PCI enumeration
  Applying: PCI/LUO: Forward finish callbacks to drivers
  Applying: PCI/LUO: Save and restore driver name
  error: patch failed: drivers/pci/probe.c:2714
  error: drivers/pci/probe.c: patch does not apply
  Patch failed at 0006 PCI/LUO: Save and restore driver name
  hint: Use 'git am --show-current-patch=diff' to see the failed patch
  When you have resolved this problem, run "git am --continue".
  If you prefer to skip this patch, run "git am --skip" instead.
  To restore the original branch and stop patching, run "git am --abort".

  $ git checkout -b wip/2509-chris-luo-pci-v2 luo/luo/v2; git am m/v2_20250916_chrisl_luo_pci_subsystem_phase_i.mbx
  Updating files: 100% (12217/12217), done.
  branch 'wip/2509-chris-luo-pci-v2' set up to track 'luo/luo/v2'.
  Switched to a new branch 'wip/2509-chris-luo-pci-v2'
  Applying: PCI/LUO: Register with Liveupdate Orchestrator
  error: patch failed: MAINTAINERS:14014
  error: MAINTAINERS: patch does not apply
  Patch failed at 0001 PCI/LUO: Register with Liveupdate Orchestrator
  hint: Use 'git am --show-current-patch=diff' to see the failed patch
  When you have resolved this problem, run "git am --continue".
  If you prefer to skip this patch, run "git am --skip" instead.
  To restore the original branch and stop patching, run "git am --abort".

  $ git checkout -b wip/2509-chris-luo-pci-v2 luo/luo/v3; git am m/v2_20250916_chrisl_luo_pci_subsystem_phase_i.mbx
  branch 'wip/2509-chris-luo-pci-v2' set up to track 'luo/luo/v3'.
  Switched to a new branch 'wip/2509-chris-luo-pci-v2'
  Applying: PCI/LUO: Register with Liveupdate Orchestrator
  error: patch failed: MAINTAINERS:14014
  error: MAINTAINERS: patch does not apply
  Patch failed at 0001 PCI/LUO: Register with Liveupdate Orchestrator
  hint: Use 'git am --show-current-patch=diff' to see the failed patch
  When you have resolved this problem, run "git am --continue".
  If you prefer to skip this patch, run "git am --skip" instead.
  To restore the original branch and stop patching, run "git am --abort".

Re: [PATCH v2 00/10] LUO: PCI subsystem (phase I)
Posted by Chris Li 4 months, 1 week ago
On Mon, Sep 29, 2025 at 8:04 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> On Sat, Sep 27, 2025 at 02:05:38PM -0400, Pasha Tatashin wrote:
> > Hi Bjorn,
> >
> > My latest submission is the following:
> > https://lore.kernel.org/all/20250807014442.3829950-1-pasha.tatashin@soleen.com/
> >
> > And github repo is in cover letter:
> >
> > https://github.com/googleprodkernel/linux-liveupdate/tree/luo/v3
> >
> > It applies cleanly against the mainline without the first three
> > patches, as they were already merged.
>
> Not sure what I'm missing.  I've tried various things but none apply
> cleanly:

Sorry about that. Let me do a refresh of the LUOPCI V3 patch and send
out the git repo link as well. The issue is that there are other
patches not in the mainline kernel which luopci is dependent on. Using
a git repo would be easier to get a working tree.

Working on it now, please stay tuned.

Chris
Re: [PATCH v2 00/10] LUO: PCI subsystem (phase I)
Posted by Chris Li 4 months ago
Thanks to one that provides good feedback on the PCI series.

I just want to give an update on the state of the LUO PCI series,
based on the feedback I received. The LUO PCI series should be called
from the memfd side and remove global subsystem state if possible.
Which means the PCI series will depend on the VIFO or iommu series.
I have some internal alignment with Vipin (for VFIO) and Samiullah
(for iommu). Here is the new plan for upstream patch submission:

1)  KHO series go first, which is already happening with additional improvement.

2) Next is Pasha's LUO series with memfd support, also happening right now.

3) Next series will be Vipin's VFIO series with preserving one
busmaster bit in the config space of the end point vfio device, there
is no PCI layer involved yet. The VFIO will use some driver trick to
prevent the native driver from binding to the liveupdate device used
by VFIO after kexec. After kexec, the VFIO driver validates that the
busmaster in the PCI config register is already set.

4) After the VFIO series, the PCI can start to preserve the livedupate
device by BDF. Avoid the driver auto probe on the livedupate devices.
At this point the VFIO driver in stage 3 will not need the other
driver trick to avoid the auto bind of native driver. The PCI layer
takes the core of that. This series PCI will have very limited
support, most of the driver callback is not needed, no bridge device
dependent as well.

5) VFIO device will continue DMA across the kexec. This series will
require the IOMMU series for DMA mapping support. The PCI will hook up
with the VFIO and build the list of the liveupdate device, which
includes the PCI bridge with bus master big preserved as well.

So I will pause the LUO PCI series a bit to wait for the integration
with VFIO series.
Meanwhile, I will continue to fix up the LUO PCI series internally for
the other feedback I have received:
- Clean up device info printing, remove raw address value (Greg KH, Jason).
- Remove the device format string (Greg KH).
- Remove the liveupdate struct from struct device, move it to the PCI (Greg KH).
- Remove LUO call back forwarding and hook it up with the VFIO (Jason, David)
- Drive the PCI from memfd context on VFIO or iommu, no subsystem
registration. (Jason)
- up_read(&pci_bus_sem); instead of up_write (Greg KH)
- Avoid preserving the driver name, just avoid auto-probing the
liveupdate devices. Let user space do the driver loading in initrd
(Jason).

That will keep me busy for a while waiting for the VFIO series.

Thanks

Chris


On Mon, Sep 29, 2025 at 11:13 AM Chris Li <chrisl@kernel.org> wrote:
>
> On Mon, Sep 29, 2025 at 8:04 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> >
> > On Sat, Sep 27, 2025 at 02:05:38PM -0400, Pasha Tatashin wrote:
> > > Hi Bjorn,
> > >
> > > My latest submission is the following:
> > > https://lore.kernel.org/all/20250807014442.3829950-1-pasha.tatashin@soleen.com/
> > >
> > > And github repo is in cover letter:
> > >
> > > https://github.com/googleprodkernel/linux-liveupdate/tree/luo/v3
> > >
> > > It applies cleanly against the mainline without the first three
> > > patches, as they were already merged.
> >
> > Not sure what I'm missing.  I've tried various things but none apply
> > cleanly:
>
> Sorry about that. Let me do a refresh of the LUOPCI V3 patch and send
> out the git repo link as well. The issue is that there are other
> patches not in the mainline kernel which luopci is dependent on. Using
> a git repo would be easier to get a working tree.
>
> Working on it now, please stay tuned.
>
> Chris
Re: [PATCH v2 00/10] LUO: PCI subsystem (phase I)
Posted by Pratyush Yadav 4 months ago
On Tue, Oct 07 2025, Chris Li wrote:

[...]
> That will keep me busy for a while waiting for the VFIO series.

I recall we talked in one of the biweekly meetings about some sanity
checking of folios right before reboot (make sure they are right order,
etc.) under a KEXEC_HANDOVER_DEBUG option. If you have some spare time
on your hands, would be cool to see some patches for that as well :-)

-- 
Regards,
Pratyush Yadav
Re: [PATCH v2 00/10] LUO: PCI subsystem (phase I)
Posted by Chris Li 4 months ago
On Thu, Oct 9, 2025 at 4:21 PM Pratyush Yadav <pratyush@kernel.org> wrote:
>
> On Tue, Oct 07 2025, Chris Li wrote:
>
> [...]
> > That will keep me busy for a while waiting for the VFIO series.
>
> I recall we talked in one of the biweekly meetings about some sanity
> checking of folios right before reboot (make sure they are right order,
> etc.) under a KEXEC_HANDOVER_DEBUG option. If you have some spare time
> on your hands, would be cool to see some patches for that as well :-)

Sure, I will add that to my "nice to have" list. No promised I got
time to get to it with the PCI. It belong to the KHO series not PCI
though.

Chris
Re: [PATCH v2 00/10] LUO: PCI subsystem (phase I)
Posted by Jason Miu 3 months, 4 weeks ago
On Thu, Oct 9, 2025 at 9:19 PM Chris Li <chrisl@kernel.org> wrote:
>
> On Thu, Oct 9, 2025 at 4:21 PM Pratyush Yadav <pratyush@kernel.org> wrote:
> >
> > On Tue, Oct 07 2025, Chris Li wrote:
> >
> > [...]
> > > That will keep me busy for a while waiting for the VFIO series.
> >
> > I recall we talked in one of the biweekly meetings about some sanity
> > checking of folios right before reboot (make sure they are right order,
> > etc.) under a KEXEC_HANDOVER_DEBUG option. If you have some spare time
> > on your hands, would be cool to see some patches for that as well :-)
>
> Sure, I will add that to my "nice to have" list. No promised I got
> time to get to it with the PCI. It belong to the KHO series not PCI
> though.
>
> Chris

Hi Pratyush, Chris,

For the folio sanity check with KEXEC_HANDOVER_DEBUG, I can follow
that up. Would you tell me what we like to check before reboot, I may
have missed some context. Thanks!

--
Jason Miu
Re: [PATCH v2 00/10] LUO: PCI subsystem (phase I)
Posted by Pratyush Yadav 3 months, 3 weeks ago
On Fri, Oct 10 2025, Jason Miu wrote:

> On Thu, Oct 9, 2025 at 9:19 PM Chris Li <chrisl@kernel.org> wrote:
>>
>> On Thu, Oct 9, 2025 at 4:21 PM Pratyush Yadav <pratyush@kernel.org> wrote:
>> >
>> > On Tue, Oct 07 2025, Chris Li wrote:
>> >
>> > [...]
>> > > That will keep me busy for a while waiting for the VFIO series.
>> >
>> > I recall we talked in one of the biweekly meetings about some sanity
>> > checking of folios right before reboot (make sure they are right order,
>> > etc.) under a KEXEC_HANDOVER_DEBUG option. If you have some spare time
>> > on your hands, would be cool to see some patches for that as well :-)
>>
>> Sure, I will add that to my "nice to have" list. No promised I got
>> time to get to it with the PCI. It belong to the KHO series not PCI
>> though.
>>

Right. It is only a "nice to have", and not a requirement. And certainly
not for the PCI series.

>
> Hi Pratyush, Chris,
>
> For the folio sanity check with KEXEC_HANDOVER_DEBUG, I can follow
> that up. Would you tell me what we like to check before reboot, I may
> have missed some context. Thanks!

The idea is to sanity-check the preserved folios in the kexec-reboot
flow somewhere. The main check discussed was to make sure the folios are
of the same order as they were preserved with. This will help catch bugs
where folios might split after being preserved.

Maybe we can add some more checks too? Like making sure the folios
aren't freed after they were preserved. But that condition is a bit
trickier to catch. But at least the former should be simple enough to
do as a start.

-- 
Regards,
Pratyush Yadav
Re: [PATCH v2 00/10] LUO: PCI subsystem (phase I)
Posted by David Matlack 4 months ago
On Tue, Oct 7, 2025 at 4:32 PM Chris Li <chrisl@kernel.org> wrote:
>
> Thanks to one that provides good feedback on the PCI series.
>
> I just want to give an update on the state of the LUO PCI series,
> based on the feedback I received. The LUO PCI series should be called
> from the memfd side and remove global subsystem state if possible.

By "memfd side" I believe you are referring to LUO fd preservation
(likely the VFIO cdev fd).

> Which means the PCI series will depend on the VIFO or iommu series.
> I have some internal alignment with Vipin (for VFIO) and Samiullah
> (for iommu). Here is the new plan for upstream patch submission:
>
> 1)  KHO series go first, which is already happening with additional improvement.
>
> 2) Next is Pasha's LUO series with memfd support, also happening right now.
>
> 3) Next series will be Vipin's VFIO series with preserving one
> busmaster bit in the config space of the end point vfio device, there
> is no PCI layer involved yet. The VFIO will use some driver trick to
> prevent the native driver from binding to the liveupdate device used
> by VFIO after kexec. After kexec, the VFIO driver validates that the
> busmaster in the PCI config register is already set.

Yes. Last we discussed Vipin is planning to just compile out the
native driver of the device he is using to test. So we don't expect to
need any kernel code changes to unblock basic testing and posting the
RFC.

>
> 4) After the VFIO series, the PCI can start to preserve the livedupate
> device by BDF. Avoid the driver auto probe on the livedupate devices.
> At this point the VFIO driver in stage 3 will not need the other
> driver trick to avoid the auto bind of native driver. The PCI layer
> takes the core of that. This series PCI will have very limited
> support, most of the driver callback is not needed, no bridge device
> dependent as well.

I suspect we'll need the new file-lifecycle-bound global state thing
that Pasha is working on [1] to accomplish this. So please track
LUOv5+ as a dependency for this.

[1] https://lore.kernel.org/lkml/CA+CK2bB+RdapsozPHe84MP4NVSPLo6vje5hji5MKSg8L6ViAbw@mail.gmail.com/

>
> 5) VFIO device will continue DMA across the kexec. This series will
> require the IOMMU series for DMA mapping support. The PCI will hook up
> with the VFIO and build the list of the liveupdate device, which
> includes the PCI bridge with bus master big preserved as well.
>
> So I will pause the LUO PCI series a bit to wait for the integration
> with VFIO series.
> Meanwhile, I will continue to fix up the LUO PCI series internally for
> the other feedback I have received:
> - Clean up device info printing, remove raw address value (Greg KH, Jason).
> - Remove the device format string (Greg KH).
> - Remove the liveupdate struct from struct device, move it to the PCI (Greg KH).
> - Remove LUO call back forwarding and hook it up with the VFIO (Jason, David)
> - Drive the PCI from memfd context on VFIO or iommu, no subsystem
> registration. (Jason)
> - up_read(&pci_bus_sem); instead of up_write (Greg KH)
> - Avoid preserving the driver name, just avoid auto-probing the
> liveupdate devices. Let user space do the driver loading in initrd
> (Jason).
>
> That will keep me busy for a while waiting for the VFIO series.

Sounds good. Thanks for the update Chris!
Re: [PATCH v2 00/10] LUO: PCI subsystem (phase I)
Posted by Chris Li 4 months ago
On Wed, Oct 8, 2025 at 4:01 PM David Matlack <dmatlack@google.com> wrote:
>
> On Tue, Oct 7, 2025 at 4:32 PM Chris Li <chrisl@kernel.org> wrote:
> >
> > Thanks to one that provides good feedback on the PCI series.
> >
> > I just want to give an update on the state of the LUO PCI series,
> > based on the feedback I received. The LUO PCI series should be called
> > from the memfd side and remove global subsystem state if possible.
>
> By "memfd side" I believe you are referring to LUO fd preservation
> (likely the VFIO cdev fd).

Yes. I haven't taken a closer look at the recent LUO fd preservation
series. It is on my to do list, now I am depending on it.

> > Which means the PCI series will depend on the VIFO or iommu series.
> > I have some internal alignment with Vipin (for VFIO) and Samiullah
> > (for iommu). Here is the new plan for upstream patch submission:
> >
> > 1)  KHO series go first, which is already happening with additional improvement.
> >
> > 2) Next is Pasha's LUO series with memfd support, also happening right now.
> >
> > 3) Next series will be Vipin's VFIO series with preserving one
> > busmaster bit in the config space of the end point vfio device, there
> > is no PCI layer involved yet. The VFIO will use some driver trick to
> > prevent the native driver from binding to the liveupdate device used
> > by VFIO after kexec. After kexec, the VFIO driver validates that the
> > busmaster in the PCI config register is already set.
>
> Yes. Last we discussed Vipin is planning to just compile out the
> native driver of the device he is using to test. So we don't expect to
> need any kernel code changes to unblock basic testing and posting the
> RFC.

Ack.

>
> >
> > 4) After the VFIO series, the PCI can start to preserve the livedupate
> > device by BDF. Avoid the driver auto probe on the livedupate devices.
> > At this point the VFIO driver in stage 3 will not need the other
> > driver trick to avoid the auto bind of native driver. The PCI layer
> > takes the core of that. This series PCI will have very limited
> > support, most of the driver callback is not needed, no bridge device
> > dependent as well.
>
> I suspect we'll need the new file-lifecycle-bound global state thing
> that Pasha is working on [1] to accomplish this. So please track
> LUOv5+ as a dependency for this.
>
> [1] https://lore.kernel.org/lkml/CA+CK2bB+RdapsozPHe84MP4NVSPLo6vje5hji5MKSg8L6ViAbw@mail.gmail.com/

Agree, I need to figure out the boiler plate change to hook up PCI to
the file descriptors.

Thanks for the clarification.

Chris