Qemu does not need pin NVDIMM memory for VFIO device during VFIO hotplug, what's more, if there is no NVDIMM hw in the test machine, the VFIO hotplug operation will need at least 10 minutes to pin RAM as the NVDIMM, this time is not accepted. So we add "nopin=on" option in the memory-backed-file, which can avoid to pin RAM memory for NVDIMM. The new command like below: -object memory-backend-file,id=mem0,share,nopin=on,mem-path=kvm.img,size=9161408512 The default of "nopin" still "off" value, which is same with previous value. Yang Zhong (2): hostmem-file: Add "nopin" option for memory-backend-file nvdimm: Add "nopin" for related documents backends/hostmem-file.c | 23 +++++++++++++++++++++++ docs/nvdimm.txt | 10 ++++++++-- hw/vfio/common.c | 12 +++++++++++- qemu-options.hx | 6 +++++- 4 files changed, 47 insertions(+), 4 deletions(-) -- 1.9.1
On 10/16/2017 04:56 PM, Yang Zhong wrote: > Qemu does not need pin NVDIMM memory for VFIO device during VFIO > hotplug, what's more, if there is no NVDIMM hw in the test machine, > the VFIO hotplug operation will need at least 10 minutes to pin RAM > as the NVDIMM, this time is not accepted. So we add "nopin=on" option > in the memory-backed-file, which can avoid to pin RAM memory for NVDIMM. No. memory-backed-file does not dedicate for nvdimm only, it can be mapped as normal memory as well. Rather more, this is no way to stop guest to use it as DMA.
On 16/10/2017 12:20, Xiao Guangrong wrote: > >> Qemu does not need pin NVDIMM memory for VFIO device during VFIO >> hotplug, what's more, if there is no NVDIMM hw in the test machine, >> the VFIO hotplug operation will need at least 10 minutes to pin RAM >> as the NVDIMM, this time is not accepted. So we add "nopin=on" option >> in the memory-backed-file, which can avoid to pin RAM memory for NVDIMM. > > No. > > memory-backed-file does not dedicate for nvdimm only, it can be mapped > as normal memory as well. Rather more, this is no way to stop guest to > use it as DMA. Right, so a better name for the object property could be "dma" rather than "nopin". I'll let others comment on whether MemoryBackend (not just memory-backend-file) is the right place for the option. I am also not sure whether VFIO is not the right place for the "other side" of the hook. If you add the memory region to the CPU address space and not the PCI address space, you can hide it from all PCI devices. Paolo
On Mon, Oct 16, 2017 at 01:58:59PM +0200, Paolo Bonzini wrote:
> On 16/10/2017 12:20, Xiao Guangrong wrote:
> >
> >> Qemu does not need pin NVDIMM memory for VFIO device during VFIO
> >> hotplug, what's more, if there is no NVDIMM hw in the test machine,
> >> the VFIO hotplug operation will need at least 10 minutes to pin RAM
> >> as the NVDIMM, this time is not accepted. So we add "nopin=on" option
> >> in the memory-backed-file, which can avoid to pin RAM memory for NVDIMM.
> >
> > No.
> >
> > memory-backed-file does not dedicate for nvdimm only, it can be mapped
> > as normal memory as well. Rather more, this is no way to stop guest to
> > use it as DMA.
>
> Right, so a better name for the object property could be "dma" rather
> than "nopin". I'll let others comment on whether MemoryBackend (not
> just memory-backend-file) is the right place for the option.
Paolo, thanks for your comments!
Yes, i can change this property from "nopin" to "dma" in memory-backend
(not in ,memmory-backed-file). thanks!
Yang
> I am also not sure whether VFIO is not the right place for the "other
> side" of the hook. If you add the memory region to the CPU address
> space and not the PCI address space, you can hide it from all PCI devices.
>
> Paolo
thanks for your suggestion!
Your suggestion can avoid DMA target to nvdimm if the nvdimm memory region
was skipped during VFIO hotplug. It is valuable to try this solution. by the
way, please share me some clue for PCI address space related with memory region,
below address_space_mem is right? Many thanks!
static void pci_bus_init(PCIBus *bus, DeviceState *parent,
MemoryRegion *address_space_mem,
MemoryRegion *address_space_io,
uint8_t devfn_min)
Regards,
Yang
On 18/10/2017 07:57, Zhong Yang wrote: > Your suggestion can avoid DMA target to nvdimm if the nvdimm memory region > was skipped during VFIO hotplug. It is valuable to try this solution. by the > way, please share me some clue for PCI address space related with memory region, > below address_space_mem is right? Many thanks! > > static void pci_bus_init(PCIBus *bus, DeviceState *parent, > MemoryRegion *address_space_mem, > MemoryRegion *address_space_io, > uint8_t devfn_min) Yes. You would have to add the region directly to the CPU address space, and not to address_space_memory (which is included in both the CPU and PCI address spaces). Paolo
On Wed, Oct 18, 2017 at 11:48:11AM +0200, Paolo Bonzini wrote: > On 18/10/2017 07:57, Zhong Yang wrote: > > Your suggestion can avoid DMA target to nvdimm if the nvdimm memory region > > was skipped during VFIO hotplug. It is valuable to try this solution. by the > > way, please share me some clue for PCI address space related with memory region, > > below address_space_mem is right? Many thanks! > > > > static void pci_bus_init(PCIBus *bus, DeviceState *parent, > > MemoryRegion *address_space_mem, > > MemoryRegion *address_space_io, > > uint8_t devfn_min) > > Yes. You would have to add the region directly to the CPU address > space, and not to address_space_memory (which is included in both the > CPU and PCI address spaces). > > Paolo Thanks Paolo for your great help! I will try this solution in next week, thanks again. Regards, Yang
On Mon, Oct 16, 2017 at 06:20:32PM +0800, Xiao Guangrong wrote: > > > On 10/16/2017 04:56 PM, Yang Zhong wrote: > >Qemu does not need pin NVDIMM memory for VFIO device during VFIO > >hotplug, what's more, if there is no NVDIMM hw in the test machine, > >the VFIO hotplug operation will need at least 10 minutes to pin RAM > >as the NVDIMM, this time is not accepted. So we add "nopin=on" option > >in the memory-backed-file, which can avoid to pin RAM memory for NVDIMM. > > No. > > memory-backed-file does not dedicate for nvdimm only, it can be mapped > as normal memory as well. Rather more, this is no way to stop guest to > use it as DMA. Guangrong, thanks for your comments. This issue was found in ClearContainer test and the general Qemu should also face same issue. If test machine does not have nvdimm hw and RAM is limitted, the VFIO hotplug or SRIOV will cost some minutes to preallocate memory for nvdimm, even the hotplug faild because of RAM is not enough. We want to find a way to avoid this kind of issue, so the configurable "nopin" is added, but the default operation is not changed, the nvdimm still can be DMA target. thanks! Regards, Yang
On Mon, 16 Oct 2017 16:56:21 +0800 Yang Zhong <yang.zhong@intel.com> wrote: > Qemu does not need pin NVDIMM memory for VFIO device during VFIO > hotplug, what's more, if there is no NVDIMM hw in the test machine, > the VFIO hotplug operation will need at least 10 minutes to pin RAM > as the NVDIMM, this time is not accepted. So we add "nopin=on" option > in the memory-backed-file, which can avoid to pin RAM memory for NVDIMM. > > The new command like below: > -object memory-backend-file,id=mem0,share,nopin=on,mem-path=kvm.img,size=9161408512 > > The default of "nopin" still "off" value, which is same with previous value. If an NVDIMM is not a possible DMA target for a VFIO assigned device then it should be in a different AddressSpace from the device. If an NVDIMM can be a DMA target then it's the correct thing to do to pin it through the IOMMU for a VFIO device even if it might take considerable time to do so. Thanks, Alex
© 2016 - 2024 Red Hat, Inc.