hw/vfio/common.c | 207 ++++++++++++++++++++++++++++++++-- include/hw/vfio/vfio-common.h | 10 ++ linux-headers/linux/vfio.h | 55 ++++++++- 3 files changed, 264 insertions(+), 8 deletions(-)
Hi all, In the past, we clear dirty log immediately after sync dirty log to userspace. This may cause redundant dirty handling if userspace handles dirty log iteratively: After vfio clears dirty log, new dirty log starts to generate. These new dirty log will be reported to userspace even if they are generated before userspace handles the same dirty page. Since a new dirty log tracking method for vfio based on iommu hwdbm[1] has been introduced in the kernel and added a new capability named VFIO_DIRTY_LOG_MANUAL_CLEAR, we can eliminate some redundant dirty handling by supporting it. This series include patches as below: Patch 1: - updated the linux-headers/linux/vfio.h from kernel side Patch 2: - introduced 'struct VFIODMARange' to describe a range of the given DMA mapping and with respect to a VFIO_IOMMU_MAP_DMA operation Patch 3: - implemented the operation to manual clear vfio dirty log, which can eliminate some redundant dirty handling Thanks, Kunkun Jiang [1] https://lore.kernel.org/linux-iommu/20210310090614.26668-1-zhukeqian1@huawei.com/T/#mb168c9738ecd3d8794e2da14f970545d5820f863 Zenghui Yu (3): linux-headers: update against 5.12-rc2 and "vfio log clear" series vfio: Maintain DMA mapping range for the container vfio/migration: Support VFIO_IOMMU_DIRTY_PAGES_FLAG_CLEAR_BITMAP hw/vfio/common.c | 207 ++++++++++++++++++++++++++++++++-- include/hw/vfio/vfio-common.h | 10 ++ linux-headers/linux/vfio.h | 55 ++++++++- 3 files changed, 264 insertions(+), 8 deletions(-) -- 2.23.0
kindly ping, Any comments and reviews are welcome.😁 Thanks, Kunkun Jiang On 2021/3/10 17:41, Kunkun Jiang wrote: > Hi all, > > In the past, we clear dirty log immediately after sync dirty log to > userspace. This may cause redundant dirty handling if userspace > handles dirty log iteratively: > > After vfio clears dirty log, new dirty log starts to generate. These > new dirty log will be reported to userspace even if they are generated > before userspace handles the same dirty page. > > Since a new dirty log tracking method for vfio based on iommu hwdbm[1] > has been introduced in the kernel and added a new capability named > VFIO_DIRTY_LOG_MANUAL_CLEAR, we can eliminate some redundant dirty > handling by supporting it. > > This series include patches as below: > Patch 1: > - updated the linux-headers/linux/vfio.h from kernel side > > Patch 2: > - introduced 'struct VFIODMARange' to describe a range of the given DMA > mapping and with respect to a VFIO_IOMMU_MAP_DMA operation > > Patch 3: > - implemented the operation to manual clear vfio dirty log, which can > eliminate some redundant dirty handling > > Thanks, > Kunkun Jiang > > [1] https://lore.kernel.org/linux-iommu/20210310090614.26668-1-zhukeqian1@huawei.com/T/#mb168c9738ecd3d8794e2da14f970545d5820f863 > > Zenghui Yu (3): > linux-headers: update against 5.12-rc2 and "vfio log clear" series > vfio: Maintain DMA mapping range for the container > vfio/migration: Support VFIO_IOMMU_DIRTY_PAGES_FLAG_CLEAR_BITMAP > > hw/vfio/common.c | 207 ++++++++++++++++++++++++++++++++-- > include/hw/vfio/vfio-common.h | 10 ++ > linux-headers/linux/vfio.h | 55 ++++++++- > 3 files changed, 264 insertions(+), 8 deletions(-) >
> From: Kunkun Jiang > Sent: Wednesday, March 10, 2021 5:41 PM > > Hi all, > > In the past, we clear dirty log immediately after sync dirty log to > userspace. This may cause redundant dirty handling if userspace > handles dirty log iteratively: > > After vfio clears dirty log, new dirty log starts to generate. These > new dirty log will be reported to userspace even if they are generated > before userspace handles the same dirty page. > > Since a new dirty log tracking method for vfio based on iommu hwdbm[1] > has been introduced in the kernel and added a new capability named > VFIO_DIRTY_LOG_MANUAL_CLEAR, we can eliminate some redundant dirty > handling by supporting it. Is there any performance data showing the benefit of this new method? > > This series include patches as below: > Patch 1: > - updated the linux-headers/linux/vfio.h from kernel side > > Patch 2: > - introduced 'struct VFIODMARange' to describe a range of the given DMA > mapping and with respect to a VFIO_IOMMU_MAP_DMA operation > > Patch 3: > - implemented the operation to manual clear vfio dirty log, which can > eliminate some redundant dirty handling > > Thanks, > Kunkun Jiang > > [1] https://lore.kernel.org/linux-iommu/20210310090614.26668-1- > zhukeqian1@huawei.com/T/#mb168c9738ecd3d8794e2da14f970545d5820f > 863 > > Zenghui Yu (3): > linux-headers: update against 5.12-rc2 and "vfio log clear" series > vfio: Maintain DMA mapping range for the container > vfio/migration: Support VFIO_IOMMU_DIRTY_PAGES_FLAG_CLEAR_BITMAP > > hw/vfio/common.c | 207 ++++++++++++++++++++++++++++++++-- > include/hw/vfio/vfio-common.h | 10 ++ > linux-headers/linux/vfio.h | 55 ++++++++- > 3 files changed, 264 insertions(+), 8 deletions(-) > > -- > 2.23.0 >
Hi Kevin, On 2021/3/18 14:28, Tian, Kevin wrote: >> From: Kunkun Jiang >> Sent: Wednesday, March 10, 2021 5:41 PM >> >> Hi all, >> >> In the past, we clear dirty log immediately after sync dirty log to >> userspace. This may cause redundant dirty handling if userspace >> handles dirty log iteratively: >> >> After vfio clears dirty log, new dirty log starts to generate. These >> new dirty log will be reported to userspace even if they are generated >> before userspace handles the same dirty page. >> >> Since a new dirty log tracking method for vfio based on iommu hwdbm[1] >> has been introduced in the kernel and added a new capability named >> VFIO_DIRTY_LOG_MANUAL_CLEAR, we can eliminate some redundant dirty >> handling by supporting it. > Is there any performance data showing the benefit of this new method? > Current dirty log tracking method for VFIO: [1] All pages marked dirty if not all iommu_groups have pinned_scope [2] pinned pages by various vendor drivers if all iommu_groups have pinned scope Both methods are coarse-grained and can not determine which pages are really dirty. Each round may mark the pages that are not really dirty as dirty and send them to the destination. ( It might be better if the range of the pinned_scope was smaller. ) This will result in a waste of resources. HWDBM is short for Hardware Dirty Bit Management. (e.g. smmuv3 HTTU, Hardware Translation Table Update) About SMMU HTTU: HTTU is a feature of ARM SMMUv3, it can update access flag or/and dirty state of the TTD (Translation Table Descriptor) by hardware. With HTTU, stage1 TTD is classified into 3 types: DBM bit AP[2](readonly bit) 1. writable_clean 1 1 2. writable_dirty 1 0 3. readonly 0 1 If HTTU_HD (manage dirty state) is enabled, smmu can change TTD from writable_clean to writable_dirty. Then software can scan TTD to sync dirty state into dirty bitmap. With this feature, we can track the dirty log of DMA continuously and precisely. The capability of VFIO_DIRTY_LOG_MANUAL_CLEAR is similar to that on the KVM side. We add this new log_clear() interface only to split the old log_sync() into two separated procedures: - use log_sync() to collect the collection only, and, - use log_clear() to clear the dirty bitmap. If you're interested in this new method, you can take a look at our set of patches. [1] https://lore.kernel.org/linux-iommu/20210310090614.26668-1-zhukeqian1@huawei.com/ Best regards, Kunkun Jiang >> This series include patches as below: >> Patch 1: >> - updated the linux-headers/linux/vfio.h from kernel side >> >> Patch 2: >> - introduced 'struct VFIODMARange' to describe a range of the given DMA >> mapping and with respect to a VFIO_IOMMU_MAP_DMA operation >> >> Patch 3: >> - implemented the operation to manual clear vfio dirty log, which can >> eliminate some redundant dirty handling >> >> Thanks, >> Kunkun Jiang >> >> [1] https://lore.kernel.org/linux-iommu/20210310090614.26668-1- >> zhukeqian1@huawei.com/T/#mb168c9738ecd3d8794e2da14f970545d5820f >> 863 >> >> Zenghui Yu (3): >> linux-headers: update against 5.12-rc2 and "vfio log clear" series >> vfio: Maintain DMA mapping range for the container >> vfio/migration: Support VFIO_IOMMU_DIRTY_PAGES_FLAG_CLEAR_BITMAP >> >> hw/vfio/common.c | 207 ++++++++++++++++++++++++++++++++-- >> include/hw/vfio/vfio-common.h | 10 ++ >> linux-headers/linux/vfio.h | 55 ++++++++- >> 3 files changed, 264 insertions(+), 8 deletions(-) >> >> -- >> 2.23.0 >> > > .
> From: Kunkun Jiang <jiangkunkun@huawei.com> > Sent: Thursday, March 18, 2021 3:59 PM > > Hi Kevin, > > On 2021/3/18 14:28, Tian, Kevin wrote: > >> From: Kunkun Jiang > >> Sent: Wednesday, March 10, 2021 5:41 PM > >> > >> Hi all, > >> > >> In the past, we clear dirty log immediately after sync dirty log to > >> userspace. This may cause redundant dirty handling if userspace > >> handles dirty log iteratively: > >> > >> After vfio clears dirty log, new dirty log starts to generate. These > >> new dirty log will be reported to userspace even if they are generated > >> before userspace handles the same dirty page. > >> > >> Since a new dirty log tracking method for vfio based on iommu hwdbm[1] > >> has been introduced in the kernel and added a new capability named > >> VFIO_DIRTY_LOG_MANUAL_CLEAR, we can eliminate some redundant > dirty > >> handling by supporting it. > > Is there any performance data showing the benefit of this new method? > > > Current dirty log tracking method for VFIO: > [1] All pages marked dirty if not all iommu_groups have pinned_scope > [2] pinned pages by various vendor drivers if all iommu_groups have > pinned scope > > Both methods are coarse-grained and can not determine which pages are > really dirty. Each round may mark the pages that are not really dirty as > dirty > and send them to the destination. ( It might be better if the range of the > pinned_scope was smaller. ) This will result in a waste of resources. > > HWDBM is short for Hardware Dirty Bit Management. > (e.g. smmuv3 HTTU, Hardware Translation Table Update) > > About SMMU HTTU: > HTTU is a feature of ARM SMMUv3, it can update access flag or/and dirty > state of the TTD (Translation Table Descriptor) by hardware. > > With HTTU, stage1 TTD is classified into 3 types: > DBM bit AP[2](readonly bit) > 1. writable_clean 1 1 > 2. writable_dirty 1 0 > 3. readonly 0 1 > > If HTTU_HD (manage dirty state) is enabled, smmu can change TTD from > writable_clean to writable_dirty. Then software can scan TTD to sync dirty > state into dirty bitmap. With this feature, we can track the dirty log of > DMA continuously and precisely. > > The capability of VFIO_DIRTY_LOG_MANUAL_CLEAR is similar to that on > the KVM side. We add this new log_clear() interface only to split the old > log_sync() into two separated procedures: > > - use log_sync() to collect the collection only, and, > - use log_clear() to clear the dirty bitmap. > > If you're interested in this new method, you can take a look at our set of > patches. > [1] > https://lore.kernel.org/linux-iommu/20210310090614.26668-1- > zhukeqian1@huawei.com/ > I know what you are doing. Intel is also working on VT-d dirty bit support based on above link. What I'm curious is the actual performance gain with this optimization. KVM doing that is one good reference, but IOMMU has different characteristics (e.g. longer invalidation latency) compared to CPU MMU. It's always good to understand what a so-called optimization can actually optimize in a context different from where it's originally proved.😊 Thanks Kevin
Hi Kevin, On 2021/3/18 17:04, Tian, Kevin wrote: >> From: Kunkun Jiang <jiangkunkun@huawei.com> >> Sent: Thursday, March 18, 2021 3:59 PM >> >> Hi Kevin, >> >> On 2021/3/18 14:28, Tian, Kevin wrote: >>>> From: Kunkun Jiang >>>> Sent: Wednesday, March 10, 2021 5:41 PM >>>> >>>> Hi all, >>>> >>>> In the past, we clear dirty log immediately after sync dirty log to >>>> userspace. This may cause redundant dirty handling if userspace >>>> handles dirty log iteratively: >>>> >>>> After vfio clears dirty log, new dirty log starts to generate. These >>>> new dirty log will be reported to userspace even if they are generated >>>> before userspace handles the same dirty page. >>>> >>>> Since a new dirty log tracking method for vfio based on iommu hwdbm[1] >>>> has been introduced in the kernel and added a new capability named >>>> VFIO_DIRTY_LOG_MANUAL_CLEAR, we can eliminate some redundant >> dirty >>>> handling by supporting it. >>> Is there any performance data showing the benefit of this new method? >>> >> Current dirty log tracking method for VFIO: >> [1] All pages marked dirty if not all iommu_groups have pinned_scope >> [2] pinned pages by various vendor drivers if all iommu_groups have >> pinned scope >> >> Both methods are coarse-grained and can not determine which pages are >> really dirty. Each round may mark the pages that are not really dirty as >> dirty >> and send them to the destination. ( It might be better if the range of the >> pinned_scope was smaller. ) This will result in a waste of resources. >> >> HWDBM is short for Hardware Dirty Bit Management. >> (e.g. smmuv3 HTTU, Hardware Translation Table Update) >> >> About SMMU HTTU: >> HTTU is a feature of ARM SMMUv3, it can update access flag or/and dirty >> state of the TTD (Translation Table Descriptor) by hardware. >> >> With HTTU, stage1 TTD is classified into 3 types: >> DBM bit AP[2](readonly bit) >> 1. writable_clean 1 1 >> 2. writable_dirty 1 0 >> 3. readonly 0 1 >> >> If HTTU_HD (manage dirty state) is enabled, smmu can change TTD from >> writable_clean to writable_dirty. Then software can scan TTD to sync dirty >> state into dirty bitmap. With this feature, we can track the dirty log of >> DMA continuously and precisely. >> >> The capability of VFIO_DIRTY_LOG_MANUAL_CLEAR is similar to that on >> the KVM side. We add this new log_clear() interface only to split the old >> log_sync() into two separated procedures: >> >> - use log_sync() to collect the collection only, and, >> - use log_clear() to clear the dirty bitmap. >> >> If you're interested in this new method, you can take a look at our set of >> patches. >> [1] >> https://lore.kernel.org/linux-iommu/20210310090614.26668-1- >> zhukeqian1@huawei.com/ >> > I know what you are doing. Intel is also working on VT-d dirty bit support > based on above link. What I'm curious is the actual performance gain > with this optimization. KVM doing that is one good reference, but IOMMU > has different characteristics (e.g. longer invalidation latency) compared to > CPU MMU. It's always good to understand what a so-called optimization > can actually optimize in a context different from where it's originally proved.😊 > > Thanks > Kevin My understanding is that this is a new method, which is quite different from the previous two. So can you explain in more detail what performance data you want?😁 Thanks, Kunkun Jiang
> From: Kunkun Jiang <jiangkunkun@huawei.com> > Sent: Thursday, March 18, 2021 8:29 PM > > Hi Kevin, > > On 2021/3/18 17:04, Tian, Kevin wrote: > >> From: Kunkun Jiang <jiangkunkun@huawei.com> > >> Sent: Thursday, March 18, 2021 3:59 PM > >> > >> Hi Kevin, > >> > >> On 2021/3/18 14:28, Tian, Kevin wrote: > >>>> From: Kunkun Jiang > >>>> Sent: Wednesday, March 10, 2021 5:41 PM > >>>> > >>>> Hi all, > >>>> > >>>> In the past, we clear dirty log immediately after sync dirty log to > >>>> userspace. This may cause redundant dirty handling if userspace > >>>> handles dirty log iteratively: > >>>> > >>>> After vfio clears dirty log, new dirty log starts to generate. These > >>>> new dirty log will be reported to userspace even if they are generated > >>>> before userspace handles the same dirty page. > >>>> > >>>> Since a new dirty log tracking method for vfio based on iommu > hwdbm[1] > >>>> has been introduced in the kernel and added a new capability named > >>>> VFIO_DIRTY_LOG_MANUAL_CLEAR, we can eliminate some redundant > >> dirty > >>>> handling by supporting it. > >>> Is there any performance data showing the benefit of this new method? > >>> > >> Current dirty log tracking method for VFIO: > >> [1] All pages marked dirty if not all iommu_groups have pinned_scope > >> [2] pinned pages by various vendor drivers if all iommu_groups have > >> pinned scope > >> > >> Both methods are coarse-grained and can not determine which pages are > >> really dirty. Each round may mark the pages that are not really dirty as > >> dirty > >> and send them to the destination. ( It might be better if the range of the > >> pinned_scope was smaller. ) This will result in a waste of resources. > >> > >> HWDBM is short for Hardware Dirty Bit Management. > >> (e.g. smmuv3 HTTU, Hardware Translation Table Update) > >> > >> About SMMU HTTU: > >> HTTU is a feature of ARM SMMUv3, it can update access flag or/and dirty > >> state of the TTD (Translation Table Descriptor) by hardware. > >> > >> With HTTU, stage1 TTD is classified into 3 types: > >> DBM bit AP[2](readonly bit) > >> 1. writable_clean 1 1 > >> 2. writable_dirty 1 0 > >> 3. readonly 0 1 > >> > >> If HTTU_HD (manage dirty state) is enabled, smmu can change TTD from > >> writable_clean to writable_dirty. Then software can scan TTD to sync dirty > >> state into dirty bitmap. With this feature, we can track the dirty log of > >> DMA continuously and precisely. > >> > >> The capability of VFIO_DIRTY_LOG_MANUAL_CLEAR is similar to that on > >> the KVM side. We add this new log_clear() interface only to split the old > >> log_sync() into two separated procedures: > >> > >> - use log_sync() to collect the collection only, and, > >> - use log_clear() to clear the dirty bitmap. > >> > >> If you're interested in this new method, you can take a look at our set of > >> patches. > >> [1] > >> https://lore.kernel.org/linux-iommu/20210310090614.26668-1- > >> zhukeqian1@huawei.com/ > >> > > I know what you are doing. Intel is also working on VT-d dirty bit support > > based on above link. What I'm curious is the actual performance gain > > with this optimization. KVM doing that is one good reference, but IOMMU > > has different characteristics (e.g. longer invalidation latency) compared to > > CPU MMU. It's always good to understand what a so-called optimization > > can actually optimize in a context different from where it's originally > proved.😊 > > > > Thanks > > Kevin > > My understanding is that this is a new method, which is quite different > from the > previous two. So can you explain in more detail what performance data > you want?😁 > > Thanks, > Kunkun Jiang When you have HTTU enabled, compare the migration efficiency with and without this manual clear interface. Thanks Kevin
On 2021/3/18 20:36, Tian, Kevin wrote: >> From: Kunkun Jiang <jiangkunkun@huawei.com> >> Sent: Thursday, March 18, 2021 8:29 PM >> >> Hi Kevin, >> >> On 2021/3/18 17:04, Tian, Kevin wrote: >>>> From: Kunkun Jiang <jiangkunkun@huawei.com> >>>> Sent: Thursday, March 18, 2021 3:59 PM >>>> >>>> Hi Kevin, >>>> >>>> On 2021/3/18 14:28, Tian, Kevin wrote: >>>>>> From: Kunkun Jiang >>>>>> Sent: Wednesday, March 10, 2021 5:41 PM >>>>>> >>>>>> Hi all, >>>>>> >>>>>> In the past, we clear dirty log immediately after sync dirty log to >>>>>> userspace. This may cause redundant dirty handling if userspace >>>>>> handles dirty log iteratively: >>>>>> >>>>>> After vfio clears dirty log, new dirty log starts to generate. These >>>>>> new dirty log will be reported to userspace even if they are generated >>>>>> before userspace handles the same dirty page. >>>>>> >>>>>> Since a new dirty log tracking method for vfio based on iommu >> hwdbm[1] >>>>>> has been introduced in the kernel and added a new capability named >>>>>> VFIO_DIRTY_LOG_MANUAL_CLEAR, we can eliminate some redundant >>>> dirty >>>>>> handling by supporting it. >>>>> Is there any performance data showing the benefit of this new method? >>>>> >>>> Current dirty log tracking method for VFIO: >>>> [1] All pages marked dirty if not all iommu_groups have pinned_scope >>>> [2] pinned pages by various vendor drivers if all iommu_groups have >>>> pinned scope >>>> >>>> Both methods are coarse-grained and can not determine which pages are >>>> really dirty. Each round may mark the pages that are not really dirty as >>>> dirty >>>> and send them to the destination. ( It might be better if the range of the >>>> pinned_scope was smaller. ) This will result in a waste of resources. >>>> >>>> HWDBM is short for Hardware Dirty Bit Management. >>>> (e.g. smmuv3 HTTU, Hardware Translation Table Update) >>>> >>>> About SMMU HTTU: >>>> HTTU is a feature of ARM SMMUv3, it can update access flag or/and dirty >>>> state of the TTD (Translation Table Descriptor) by hardware. >>>> >>>> With HTTU, stage1 TTD is classified into 3 types: >>>> DBM bit AP[2](readonly bit) >>>> 1. writable_clean 1 1 >>>> 2. writable_dirty 1 0 >>>> 3. readonly 0 1 >>>> >>>> If HTTU_HD (manage dirty state) is enabled, smmu can change TTD from >>>> writable_clean to writable_dirty. Then software can scan TTD to sync dirty >>>> state into dirty bitmap. With this feature, we can track the dirty log of >>>> DMA continuously and precisely. >>>> >>>> The capability of VFIO_DIRTY_LOG_MANUAL_CLEAR is similar to that on >>>> the KVM side. We add this new log_clear() interface only to split the old >>>> log_sync() into two separated procedures: >>>> >>>> - use log_sync() to collect the collection only, and, >>>> - use log_clear() to clear the dirty bitmap. >>>> >>>> If you're interested in this new method, you can take a look at our set of >>>> patches. >>>> [1] >>>> https://lore.kernel.org/linux-iommu/20210310090614.26668-1- >>>> zhukeqian1@huawei.com/ >>>> >>> I know what you are doing. Intel is also working on VT-d dirty bit support >>> based on above link. What I'm curious is the actual performance gain >>> with this optimization. KVM doing that is one good reference, but IOMMU >>> has different characteristics (e.g. longer invalidation latency) compared to >>> CPU MMU. It's always good to understand what a so-called optimization >>> can actually optimize in a context different from where it's originally >> proved.😊 >>> Thanks >>> Kevin >> My understanding is that this is a new method, which is quite different >> from the >> previous two. So can you explain in more detail what performance data >> you want?😁 >> >> Thanks, >> Kunkun Jiang > When you have HTTU enabled, compare the migration efficiency with and > without this manual clear interface. > > Thanks > Kevin Oh, I see. From a theoretical analysis, this manual clear interface will be effective. I will perform some tests on it and send out the results. Thanks, Kunkun Jiang
On 2021/3/18 20:36, Tian, Kevin wrote: >> From: Kunkun Jiang <jiangkunkun@huawei.com> >> Sent: Thursday, March 18, 2021 8:29 PM >> >> Hi Kevin, >> >> On 2021/3/18 17:04, Tian, Kevin wrote: >>>> From: Kunkun Jiang <jiangkunkun@huawei.com> >>>> Sent: Thursday, March 18, 2021 3:59 PM >>>> >>>> Hi Kevin, >>>> >>>> On 2021/3/18 14:28, Tian, Kevin wrote: >>>>>> From: Kunkun Jiang >>>>>> Sent: Wednesday, March 10, 2021 5:41 PM >>>>>> >>>>>> Hi all, >>>>>> >>>>>> In the past, we clear dirty log immediately after sync dirty log to >>>>>> userspace. This may cause redundant dirty handling if userspace >>>>>> handles dirty log iteratively: >>>>>> >>>>>> After vfio clears dirty log, new dirty log starts to generate. These >>>>>> new dirty log will be reported to userspace even if they are generated >>>>>> before userspace handles the same dirty page. >>>>>> >>>>>> Since a new dirty log tracking method for vfio based on iommu >> hwdbm[1] >>>>>> has been introduced in the kernel and added a new capability named >>>>>> VFIO_DIRTY_LOG_MANUAL_CLEAR, we can eliminate some redundant >>>> dirty >>>>>> handling by supporting it. >>>>> Is there any performance data showing the benefit of this new method? >>>>> >>>> Current dirty log tracking method for VFIO: >>>> [1] All pages marked dirty if not all iommu_groups have pinned_scope >>>> [2] pinned pages by various vendor drivers if all iommu_groups have >>>> pinned scope >>>> >>>> Both methods are coarse-grained and can not determine which pages are >>>> really dirty. Each round may mark the pages that are not really dirty as >>>> dirty >>>> and send them to the destination. ( It might be better if the range of the >>>> pinned_scope was smaller. ) This will result in a waste of resources. >>>> >>>> HWDBM is short for Hardware Dirty Bit Management. >>>> (e.g. smmuv3 HTTU, Hardware Translation Table Update) >>>> >>>> About SMMU HTTU: >>>> HTTU is a feature of ARM SMMUv3, it can update access flag or/and dirty >>>> state of the TTD (Translation Table Descriptor) by hardware. >>>> >>>> With HTTU, stage1 TTD is classified into 3 types: >>>> DBM bit AP[2](readonly bit) >>>> 1. writable_clean 1 1 >>>> 2. writable_dirty 1 0 >>>> 3. readonly 0 1 >>>> >>>> If HTTU_HD (manage dirty state) is enabled, smmu can change TTD from >>>> writable_clean to writable_dirty. Then software can scan TTD to sync dirty >>>> state into dirty bitmap. With this feature, we can track the dirty log of >>>> DMA continuously and precisely. >>>> >>>> The capability of VFIO_DIRTY_LOG_MANUAL_CLEAR is similar to that on >>>> the KVM side. We add this new log_clear() interface only to split the old >>>> log_sync() into two separated procedures: >>>> >>>> - use log_sync() to collect the collection only, and, >>>> - use log_clear() to clear the dirty bitmap. >>>> >>>> If you're interested in this new method, you can take a look at our set of >>>> patches. >>>> [1] >>>> https://lore.kernel.org/linux-iommu/20210310090614.26668-1- >>>> zhukeqian1@huawei.com/ >>>> >>> I know what you are doing. Intel is also working on VT-d dirty bit support >>> based on above link. What I'm curious is the actual performance gain >>> with this optimization. KVM doing that is one good reference, but IOMMU >>> has different characteristics (e.g. longer invalidation latency) compared to >>> CPU MMU. It's always good to understand what a so-called optimization >>> can actually optimize in a context different from where it's originally >> proved.😊 >>> Thanks >>> Kevin >> My understanding is that this is a new method, which is quite different >> from the >> previous two. So can you explain in more detail what performance data >> you want?😁 >> >> Thanks, >> Kunkun Jiang > When you have HTTU enabled, compare the migration efficiency with and > without this manual clear interface. > > Thanks > Kevin Hi Kevin, Sorry for late reply. I tested it on our FPGA in two scenarios. [1] perform limited times of DMA on a fixed ram area [2] perform infinite DMA on a fixed ram area In scenario [1], we can clearly see that lesser data is being transmitted with this manual clear interface. For example, a total of 10 DMA are performed, the amount of transferred data is the sum of 6 times. This depends on whether the device performs a DMA on the dirty ram area between log_sync() and log_clear(). In scenario [2], with or without this manual clear interface, it doesn't make a big difference. Every time we call log_sync(), the fixed ram area is dirty. So, in general scenarios, it can reduce the amount of transmitted data. In addition, regarding the difference between MMU and IOMMU (e.g. longer invalidation latency) you mentioned last time, I think it has no effect on this manual clear interface. Currently, the way we invalidate IOMMU TLB is iommu_flush_iotlb_all(). It takes lesser time than multiple, range-based invalidation. Thanks, Kunkun Jiang
© 2016 - 2025 Red Hat, Inc.