include/linux/mm.h | 18 +- include/linux/page-isolation.h | 5 +- include/linux/vmstat.h | 8 - mm/debug_page_alloc.c | 12 +- mm/internal.h | 9 - mm/page_alloc.c | 650 +++++++++++++++++++++------------------ mm/page_isolation.c | 122 +++----- 7 files changed, 415 insertions(+), 409 deletions(-)
V4:
- fixed !pcp_order_allowed() case in free_unref_folios()
- reworded the patch 0 changelog a bit for the git log
- rebased to mm-everything-2024-03-19-23-01
- runtime-tested again with various CONFIG_DEBUG_FOOs enabled
---
The page allocator's mobility grouping is intended to keep unmovable
pages separate from reclaimable/compactable ones to allow on-demand
defragmentation for higher-order allocations and huge pages.
Currently, there are several places where accidental type mixing
occurs: an allocation asks for a page of a certain migratetype and
receives another. This ruins pageblocks for compaction, which in turn
makes allocating huge pages more expensive and less reliable.
The series addresses those causes. The last patch adds type checks on
all freelist movements to prevent new violations being introduced.
The benefits can be seen in a mixed workload that stresses the machine
with a memcache-type workload and a kernel build job while
periodically attempting to allocate batches of THP. The following data
is aggregated over 50 consecutive defconfig builds:
VANILLA PATCHED
Hugealloc Time mean 165843.93 ( +0.00%) 113025.88 ( -31.85%)
Hugealloc Time stddev 158957.35 ( +0.00%) 114716.07 ( -27.83%)
Kbuild Real time 310.24 ( +0.00%) 300.73 ( -3.06%)
Kbuild User time 1271.13 ( +0.00%) 1259.42 ( -0.92%)
Kbuild System time 582.02 ( +0.00%) 559.79 ( -3.81%)
THP fault alloc 30585.14 ( +0.00%) 40853.62 ( +33.57%)
THP fault fallback 36626.46 ( +0.00%) 26357.62 ( -28.04%)
THP fault fail rate % 54.49 ( +0.00%) 39.22 ( -27.53%)
Pagealloc fallback 1328.00 ( +0.00%) 1.00 ( -99.85%)
Pagealloc type mismatch 181009.50 ( +0.00%) 0.00 ( -100.00%)
Direct compact stall 434.56 ( +0.00%) 257.66 ( -40.61%)
Direct compact fail 421.70 ( +0.00%) 249.94 ( -40.63%)
Direct compact success 12.86 ( +0.00%) 7.72 ( -37.09%)
Direct compact success rate % 2.86 ( +0.00%) 2.82 ( -0.96%)
Compact daemon scanned migrate 3370059.62 ( +0.00%) 3612054.76 ( +7.18%)
Compact daemon scanned free 7718439.20 ( +0.00%) 5386385.02 ( -30.21%)
Compact direct scanned migrate 309248.62 ( +0.00%) 176721.04 ( -42.85%)
Compact direct scanned free 433582.84 ( +0.00%) 315727.66 ( -27.18%)
Compact migrate scanned daemon % 91.20 ( +0.00%) 94.48 ( +3.56%)
Compact free scanned daemon % 94.58 ( +0.00%) 94.42 ( -0.16%)
Compact total migrate scanned 3679308.24 ( +0.00%) 3788775.80 ( +2.98%)
Compact total free scanned 8152022.04 ( +0.00%) 5702112.68 ( -30.05%)
Alloc stall 872.04 ( +0.00%) 5156.12 ( +490.71%)
Pages kswapd scanned 510645.86 ( +0.00%) 3394.94 ( -99.33%)
Pages kswapd reclaimed 134811.62 ( +0.00%) 2701.26 ( -98.00%)
Pages direct scanned 99546.06 ( +0.00%) 376407.52 ( +278.12%)
Pages direct reclaimed 62123.40 ( +0.00%) 289535.70 ( +366.06%)
Pages total scanned 610191.92 ( +0.00%) 379802.46 ( -37.76%)
Pages scanned kswapd % 76.36 ( +0.00%) 0.10 ( -98.58%)
Swap out 12057.54 ( +0.00%) 15022.98 ( +24.59%)
Swap in 209.16 ( +0.00%) 256.48 ( +22.52%)
File refaults 17701.64 ( +0.00%) 11765.40 ( -33.53%)
Huge page success rate is higher, allocation latencies are shorter and
more predictable.
Stealing (fallback) rate is drastically reduced. Notably, while the
vanilla kernel keeps doing fallbacks on an ongoing basis, the patched
kernel enters a steady state once the distribution of block types is
adequate for the workload. Steals over 50 runs:
VANILLA PATCHED
1504.0 227.0
1557.0 6.0
1391.0 13.0
1080.0 26.0
1057.0 40.0
1156.0 6.0
805.0 46.0
736.0 20.0
1747.0 2.0
1699.0 34.0
1269.0 13.0
1858.0 12.0
907.0 4.0
727.0 2.0
563.0 2.0
3094.0 2.0
10211.0 3.0
2621.0 1.0
5508.0 2.0
1060.0 2.0
538.0 3.0
5773.0 2.0
2199.0 0.0
3781.0 2.0
1387.0 1.0
4977.0 0.0
2865.0 1.0
1814.0 1.0
3739.0 1.0
6857.0 0.0
382.0 0.0
407.0 1.0
3784.0 0.0
297.0 0.0
298.0 0.0
6636.0 0.0
4188.0 0.0
242.0 0.0
9960.0 0.0
5816.0 0.0
354.0 0.0
287.0 0.0
261.0 0.0
140.0 1.0
2065.0 0.0
312.0 0.0
331.0 0.0
164.0 0.0
465.0 1.0
219.0 0.0
Type mismatches are down too. Those count every time an allocation
request asks for one migratetype and gets another. This can still
occur minimally in the patched kernel due to non-stealing fallbacks,
but it's quite rare and follows the pattern of overall fallbacks -
once the block type distribution settles, mismatches cease as well:
VANILLA: PATCHED:
182602.0 268.0
135794.0 20.0
88619.0 19.0
95973.0 0.0
129590.0 0.0
129298.0 0.0
147134.0 0.0
230854.0 0.0
239709.0 0.0
137670.0 0.0
132430.0 0.0
65712.0 0.0
57901.0 0.0
67506.0 0.0
63565.0 4.0
34806.0 0.0
42962.0 0.0
32406.0 0.0
38668.0 0.0
61356.0 0.0
57800.0 0.0
41435.0 0.0
83456.0 0.0
65048.0 0.0
28955.0 0.0
47597.0 0.0
75117.0 0.0
55564.0 0.0
38280.0 0.0
52404.0 0.0
26264.0 0.0
37538.0 0.0
19671.0 0.0
30936.0 0.0
26933.0 0.0
16962.0 0.0
44554.0 0.0
46352.0 0.0
24995.0 0.0
35152.0 0.0
12823.0 0.0
21583.0 0.0
18129.0 0.0
31693.0 0.0
28745.0 0.0
33308.0 0.0
31114.0 0.0
35034.0 0.0
12111.0 0.0
24885.0 0.0
Compaction work is markedly reduced despite much better THP rates.
In the vanilla kernel, reclaim seems to have been driven primarily by
watermark boosting that happens as a result of fallbacks. With those
all but eliminated, watermarks average lower and kswapd does less
work. The uptick in direct reclaim is because THP requests have to
fend for themselves more often - which is intended policy right
now. Aggregate reclaim activity is lowered significantly, though.
---
V3:
- fixed freelist type violations from non-atomic page isolation
updates (Zi Yan)
- fixed incorrect migratetype update ordering during merge (Vlastimil Babka)
- reject moving a zone-straddling block altogether (Vlastimil Babka)
- fixed freelist type violations from lockless migratetype lookups in
cornercase freeing paths (Vlastimil Babka)
- fixed erroneous WARN in the bulk freeing path that was intended to catch
mistakes in the now-removed pcpcache (Mike Kravetz)
- fixed typo in patch 1's changelog (Zi Yan)
- optimized migratetype lookup in free_unref_page_list() (Vlastimil Babka)
- batched vmstat updates in page merging hot path (Vlastimil Babka)
- rebased to mm-everything-2024-03-05-20-43 (v6.8-rc5+)
V2:
- dropped the get_pfnblock_migratetype() optimization
patchlet since somebody else beat me to it (thanks Zi)
- broke out pcp bypass fix since somebody else reported the bug:
https://lore.kernel.org/linux-mm/20230911181108.GA104295@cmpxchg.org/
- fixed the CONFIG_UNACCEPTED_MEMORY build (lkp)
- rebased to v6.6-rc1
include/linux/mm.h | 18 +-
include/linux/page-isolation.h | 5 +-
include/linux/vmstat.h | 8 -
mm/debug_page_alloc.c | 12 +-
mm/internal.h | 9 -
mm/page_alloc.c | 650 +++++++++++++++++++++------------------
mm/page_isolation.c | 122 +++-----
7 files changed, 415 insertions(+), 409 deletions(-)
Based on mm-everything-2024-03-19-23-01.
On 2024/3/21 02:02, Johannes Weiner wrote: > V4: > - fixed !pcp_order_allowed() case in free_unref_folios() > - reworded the patch 0 changelog a bit for the git log > - rebased to mm-everything-2024-03-19-23-01 > - runtime-tested again with various CONFIG_DEBUG_FOOs enabled > > --- > > The page allocator's mobility grouping is intended to keep unmovable > pages separate from reclaimable/compactable ones to allow on-demand > defragmentation for higher-order allocations and huge pages. > > Currently, there are several places where accidental type mixing > occurs: an allocation asks for a page of a certain migratetype and > receives another. This ruins pageblocks for compaction, which in turn > makes allocating huge pages more expensive and less reliable. > > The series addresses those causes. The last patch adds type checks on > all freelist movements to prevent new violations being introduced. > > The benefits can be seen in a mixed workload that stresses the machine > with a memcache-type workload and a kernel build job while > periodically attempting to allocate batches of THP. The following data > is aggregated over 50 consecutive defconfig builds: > > VANILLA PATCHED > Hugealloc Time mean 165843.93 ( +0.00%) 113025.88 ( -31.85%) > Hugealloc Time stddev 158957.35 ( +0.00%) 114716.07 ( -27.83%) > Kbuild Real time 310.24 ( +0.00%) 300.73 ( -3.06%) > Kbuild User time 1271.13 ( +0.00%) 1259.42 ( -0.92%) > Kbuild System time 582.02 ( +0.00%) 559.79 ( -3.81%) > THP fault alloc 30585.14 ( +0.00%) 40853.62 ( +33.57%) > THP fault fallback 36626.46 ( +0.00%) 26357.62 ( -28.04%) > THP fault fail rate % 54.49 ( +0.00%) 39.22 ( -27.53%) > Pagealloc fallback 1328.00 ( +0.00%) 1.00 ( -99.85%) > Pagealloc type mismatch 181009.50 ( +0.00%) 0.00 ( -100.00%) > Direct compact stall 434.56 ( +0.00%) 257.66 ( -40.61%) > Direct compact fail 421.70 ( +0.00%) 249.94 ( -40.63%) > Direct compact success 12.86 ( +0.00%) 7.72 ( -37.09%) > Direct compact success rate % 2.86 ( +0.00%) 2.82 ( -0.96%) > Compact daemon scanned migrate 3370059.62 ( +0.00%) 3612054.76 ( +7.18%) > Compact daemon scanned free 7718439.20 ( +0.00%) 5386385.02 ( -30.21%) > Compact direct scanned migrate 309248.62 ( +0.00%) 176721.04 ( -42.85%) > Compact direct scanned free 433582.84 ( +0.00%) 315727.66 ( -27.18%) > Compact migrate scanned daemon % 91.20 ( +0.00%) 94.48 ( +3.56%) > Compact free scanned daemon % 94.58 ( +0.00%) 94.42 ( -0.16%) > Compact total migrate scanned 3679308.24 ( +0.00%) 3788775.80 ( +2.98%) > Compact total free scanned 8152022.04 ( +0.00%) 5702112.68 ( -30.05%) > Alloc stall 872.04 ( +0.00%) 5156.12 ( +490.71%) > Pages kswapd scanned 510645.86 ( +0.00%) 3394.94 ( -99.33%) > Pages kswapd reclaimed 134811.62 ( +0.00%) 2701.26 ( -98.00%) > Pages direct scanned 99546.06 ( +0.00%) 376407.52 ( +278.12%) > Pages direct reclaimed 62123.40 ( +0.00%) 289535.70 ( +366.06%) > Pages total scanned 610191.92 ( +0.00%) 379802.46 ( -37.76%) > Pages scanned kswapd % 76.36 ( +0.00%) 0.10 ( -98.58%) > Swap out 12057.54 ( +0.00%) 15022.98 ( +24.59%) > Swap in 209.16 ( +0.00%) 256.48 ( +22.52%) > File refaults 17701.64 ( +0.00%) 11765.40 ( -33.53%) > > Huge page success rate is higher, allocation latencies are shorter and > more predictable. > > Stealing (fallback) rate is drastically reduced. Notably, while the > vanilla kernel keeps doing fallbacks on an ongoing basis, the patched > kernel enters a steady state once the distribution of block types is > adequate for the workload. Steals over 50 runs: > > VANILLA PATCHED > 1504.0 227.0 > 1557.0 6.0 > 1391.0 13.0 > 1080.0 26.0 > 1057.0 40.0 > 1156.0 6.0 > 805.0 46.0 > 736.0 20.0 > 1747.0 2.0 > 1699.0 34.0 > 1269.0 13.0 > 1858.0 12.0 > 907.0 4.0 > 727.0 2.0 > 563.0 2.0 > 3094.0 2.0 > 10211.0 3.0 > 2621.0 1.0 > 5508.0 2.0 > 1060.0 2.0 > 538.0 3.0 > 5773.0 2.0 > 2199.0 0.0 > 3781.0 2.0 > 1387.0 1.0 > 4977.0 0.0 > 2865.0 1.0 > 1814.0 1.0 > 3739.0 1.0 > 6857.0 0.0 > 382.0 0.0 > 407.0 1.0 > 3784.0 0.0 > 297.0 0.0 > 298.0 0.0 > 6636.0 0.0 > 4188.0 0.0 > 242.0 0.0 > 9960.0 0.0 > 5816.0 0.0 > 354.0 0.0 > 287.0 0.0 > 261.0 0.0 > 140.0 1.0 > 2065.0 0.0 > 312.0 0.0 > 331.0 0.0 > 164.0 0.0 > 465.0 1.0 > 219.0 0.0 > > Type mismatches are down too. Those count every time an allocation > request asks for one migratetype and gets another. This can still > occur minimally in the patched kernel due to non-stealing fallbacks, > but it's quite rare and follows the pattern of overall fallbacks - > once the block type distribution settles, mismatches cease as well: > > VANILLA: PATCHED: > 182602.0 268.0 > 135794.0 20.0 > 88619.0 19.0 > 95973.0 0.0 > 129590.0 0.0 > 129298.0 0.0 > 147134.0 0.0 > 230854.0 0.0 > 239709.0 0.0 > 137670.0 0.0 > 132430.0 0.0 > 65712.0 0.0 > 57901.0 0.0 > 67506.0 0.0 > 63565.0 4.0 > 34806.0 0.0 > 42962.0 0.0 > 32406.0 0.0 > 38668.0 0.0 > 61356.0 0.0 > 57800.0 0.0 > 41435.0 0.0 > 83456.0 0.0 > 65048.0 0.0 > 28955.0 0.0 > 47597.0 0.0 > 75117.0 0.0 > 55564.0 0.0 > 38280.0 0.0 > 52404.0 0.0 > 26264.0 0.0 > 37538.0 0.0 > 19671.0 0.0 > 30936.0 0.0 > 26933.0 0.0 > 16962.0 0.0 > 44554.0 0.0 > 46352.0 0.0 > 24995.0 0.0 > 35152.0 0.0 > 12823.0 0.0 > 21583.0 0.0 > 18129.0 0.0 > 31693.0 0.0 > 28745.0 0.0 > 33308.0 0.0 > 31114.0 0.0 > 35034.0 0.0 > 12111.0 0.0 > 24885.0 0.0 > > Compaction work is markedly reduced despite much better THP rates. > > In the vanilla kernel, reclaim seems to have been driven primarily by > watermark boosting that happens as a result of fallbacks. With those > all but eliminated, watermarks average lower and kswapd does less > work. The uptick in direct reclaim is because THP requests have to > fend for themselves more often - which is intended policy right > now. Aggregate reclaim activity is lowered significantly, though. > > --- With my 2 fixes, the whole series works well on my platform, so please feel free to add: Tested-by: Baolin Wang <baolin.wang@linux.alibaba.com>
On Mon, Apr 08, 2024 at 05:30:04PM +0800, Baolin Wang wrote: > > > On 2024/3/21 02:02, Johannes Weiner wrote: > > V4: > > - fixed !pcp_order_allowed() case in free_unref_folios() > > - reworded the patch 0 changelog a bit for the git log > > - rebased to mm-everything-2024-03-19-23-01 > > - runtime-tested again with various CONFIG_DEBUG_FOOs enabled > > > > --- > > > > The page allocator's mobility grouping is intended to keep unmovable > > pages separate from reclaimable/compactable ones to allow on-demand > > defragmentation for higher-order allocations and huge pages. > > > > Currently, there are several places where accidental type mixing > > occurs: an allocation asks for a page of a certain migratetype and > > receives another. This ruins pageblocks for compaction, which in turn > > makes allocating huge pages more expensive and less reliable. > > > > The series addresses those causes. The last patch adds type checks on > > all freelist movements to prevent new violations being introduced. > > > > The benefits can be seen in a mixed workload that stresses the machine > > with a memcache-type workload and a kernel build job while > > periodically attempting to allocate batches of THP. The following data > > is aggregated over 50 consecutive defconfig builds: > > > > VANILLA PATCHED > > Hugealloc Time mean 165843.93 ( +0.00%) 113025.88 ( -31.85%) > > Hugealloc Time stddev 158957.35 ( +0.00%) 114716.07 ( -27.83%) > > Kbuild Real time 310.24 ( +0.00%) 300.73 ( -3.06%) > > Kbuild User time 1271.13 ( +0.00%) 1259.42 ( -0.92%) > > Kbuild System time 582.02 ( +0.00%) 559.79 ( -3.81%) > > THP fault alloc 30585.14 ( +0.00%) 40853.62 ( +33.57%) > > THP fault fallback 36626.46 ( +0.00%) 26357.62 ( -28.04%) > > THP fault fail rate % 54.49 ( +0.00%) 39.22 ( -27.53%) > > Pagealloc fallback 1328.00 ( +0.00%) 1.00 ( -99.85%) > > Pagealloc type mismatch 181009.50 ( +0.00%) 0.00 ( -100.00%) > > Direct compact stall 434.56 ( +0.00%) 257.66 ( -40.61%) > > Direct compact fail 421.70 ( +0.00%) 249.94 ( -40.63%) > > Direct compact success 12.86 ( +0.00%) 7.72 ( -37.09%) > > Direct compact success rate % 2.86 ( +0.00%) 2.82 ( -0.96%) > > Compact daemon scanned migrate 3370059.62 ( +0.00%) 3612054.76 ( +7.18%) > > Compact daemon scanned free 7718439.20 ( +0.00%) 5386385.02 ( -30.21%) > > Compact direct scanned migrate 309248.62 ( +0.00%) 176721.04 ( -42.85%) > > Compact direct scanned free 433582.84 ( +0.00%) 315727.66 ( -27.18%) > > Compact migrate scanned daemon % 91.20 ( +0.00%) 94.48 ( +3.56%) > > Compact free scanned daemon % 94.58 ( +0.00%) 94.42 ( -0.16%) > > Compact total migrate scanned 3679308.24 ( +0.00%) 3788775.80 ( +2.98%) > > Compact total free scanned 8152022.04 ( +0.00%) 5702112.68 ( -30.05%) > > Alloc stall 872.04 ( +0.00%) 5156.12 ( +490.71%) > > Pages kswapd scanned 510645.86 ( +0.00%) 3394.94 ( -99.33%) > > Pages kswapd reclaimed 134811.62 ( +0.00%) 2701.26 ( -98.00%) > > Pages direct scanned 99546.06 ( +0.00%) 376407.52 ( +278.12%) > > Pages direct reclaimed 62123.40 ( +0.00%) 289535.70 ( +366.06%) > > Pages total scanned 610191.92 ( +0.00%) 379802.46 ( -37.76%) > > Pages scanned kswapd % 76.36 ( +0.00%) 0.10 ( -98.58%) > > Swap out 12057.54 ( +0.00%) 15022.98 ( +24.59%) > > Swap in 209.16 ( +0.00%) 256.48 ( +22.52%) > > File refaults 17701.64 ( +0.00%) 11765.40 ( -33.53%) > > > > Huge page success rate is higher, allocation latencies are shorter and > > more predictable. > > > > Stealing (fallback) rate is drastically reduced. Notably, while the > > vanilla kernel keeps doing fallbacks on an ongoing basis, the patched > > kernel enters a steady state once the distribution of block types is > > adequate for the workload. Steals over 50 runs: > > > > VANILLA PATCHED > > 1504.0 227.0 > > 1557.0 6.0 > > 1391.0 13.0 > > 1080.0 26.0 > > 1057.0 40.0 > > 1156.0 6.0 > > 805.0 46.0 > > 736.0 20.0 > > 1747.0 2.0 > > 1699.0 34.0 > > 1269.0 13.0 > > 1858.0 12.0 > > 907.0 4.0 > > 727.0 2.0 > > 563.0 2.0 > > 3094.0 2.0 > > 10211.0 3.0 > > 2621.0 1.0 > > 5508.0 2.0 > > 1060.0 2.0 > > 538.0 3.0 > > 5773.0 2.0 > > 2199.0 0.0 > > 3781.0 2.0 > > 1387.0 1.0 > > 4977.0 0.0 > > 2865.0 1.0 > > 1814.0 1.0 > > 3739.0 1.0 > > 6857.0 0.0 > > 382.0 0.0 > > 407.0 1.0 > > 3784.0 0.0 > > 297.0 0.0 > > 298.0 0.0 > > 6636.0 0.0 > > 4188.0 0.0 > > 242.0 0.0 > > 9960.0 0.0 > > 5816.0 0.0 > > 354.0 0.0 > > 287.0 0.0 > > 261.0 0.0 > > 140.0 1.0 > > 2065.0 0.0 > > 312.0 0.0 > > 331.0 0.0 > > 164.0 0.0 > > 465.0 1.0 > > 219.0 0.0 > > > > Type mismatches are down too. Those count every time an allocation > > request asks for one migratetype and gets another. This can still > > occur minimally in the patched kernel due to non-stealing fallbacks, > > but it's quite rare and follows the pattern of overall fallbacks - > > once the block type distribution settles, mismatches cease as well: > > > > VANILLA: PATCHED: > > 182602.0 268.0 > > 135794.0 20.0 > > 88619.0 19.0 > > 95973.0 0.0 > > 129590.0 0.0 > > 129298.0 0.0 > > 147134.0 0.0 > > 230854.0 0.0 > > 239709.0 0.0 > > 137670.0 0.0 > > 132430.0 0.0 > > 65712.0 0.0 > > 57901.0 0.0 > > 67506.0 0.0 > > 63565.0 4.0 > > 34806.0 0.0 > > 42962.0 0.0 > > 32406.0 0.0 > > 38668.0 0.0 > > 61356.0 0.0 > > 57800.0 0.0 > > 41435.0 0.0 > > 83456.0 0.0 > > 65048.0 0.0 > > 28955.0 0.0 > > 47597.0 0.0 > > 75117.0 0.0 > > 55564.0 0.0 > > 38280.0 0.0 > > 52404.0 0.0 > > 26264.0 0.0 > > 37538.0 0.0 > > 19671.0 0.0 > > 30936.0 0.0 > > 26933.0 0.0 > > 16962.0 0.0 > > 44554.0 0.0 > > 46352.0 0.0 > > 24995.0 0.0 > > 35152.0 0.0 > > 12823.0 0.0 > > 21583.0 0.0 > > 18129.0 0.0 > > 31693.0 0.0 > > 28745.0 0.0 > > 33308.0 0.0 > > 31114.0 0.0 > > 35034.0 0.0 > > 12111.0 0.0 > > 24885.0 0.0 > > > > Compaction work is markedly reduced despite much better THP rates. > > > > In the vanilla kernel, reclaim seems to have been driven primarily by > > watermark boosting that happens as a result of fallbacks. With those > > all but eliminated, watermarks average lower and kswapd does less > > work. The uptick in direct reclaim is because THP requests have to > > fend for themselves more often - which is intended policy right > > now. Aggregate reclaim activity is lowered significantly, though. > > > > --- > > With my 2 fixes, the whole series works well on my platform, so please > feel free to add: > Tested-by: Baolin Wang <baolin.wang@linux.alibaba.com> Very much appreciate your testing and the two fixes. Thank you!
On Wed, Mar 20, 2024 at 12:04 PM Johannes Weiner <hannes@cmpxchg.org> wrote: > > V4: > - fixed !pcp_order_allowed() case in free_unref_folios() > - reworded the patch 0 changelog a bit for the git log > - rebased to mm-everything-2024-03-19-23-01 > - runtime-tested again with various CONFIG_DEBUG_FOOs enabled > > --- > > The page allocator's mobility grouping is intended to keep unmovable > pages separate from reclaimable/compactable ones to allow on-demand > defragmentation for higher-order allocations and huge pages. > > Currently, there are several places where accidental type mixing > occurs: an allocation asks for a page of a certain migratetype and > receives another. This ruins pageblocks for compaction, which in turn > makes allocating huge pages more expensive and less reliable. > > The series addresses those causes. The last patch adds type checks on > all freelist movements to prevent new violations being introduced. > > The benefits can be seen in a mixed workload that stresses the machine > with a memcache-type workload and a kernel build job while > periodically attempting to allocate batches of THP. The following data > is aggregated over 50 consecutive defconfig builds: > > VANILLA PATCHED > Hugealloc Time mean 165843.93 ( +0.00%) 113025.88 ( -31.85%) > Hugealloc Time stddev 158957.35 ( +0.00%) 114716.07 ( -27.83%) > Kbuild Real time 310.24 ( +0.00%) 300.73 ( -3.06%) > Kbuild User time 1271.13 ( +0.00%) 1259.42 ( -0.92%) > Kbuild System time 582.02 ( +0.00%) 559.79 ( -3.81%) > THP fault alloc 30585.14 ( +0.00%) 40853.62 ( +33.57%) > THP fault fallback 36626.46 ( +0.00%) 26357.62 ( -28.04%) > THP fault fail rate % 54.49 ( +0.00%) 39.22 ( -27.53%) > Pagealloc fallback 1328.00 ( +0.00%) 1.00 ( -99.85%) > Pagealloc type mismatch 181009.50 ( +0.00%) 0.00 ( -100.00%) > Direct compact stall 434.56 ( +0.00%) 257.66 ( -40.61%) > Direct compact fail 421.70 ( +0.00%) 249.94 ( -40.63%) > Direct compact success 12.86 ( +0.00%) 7.72 ( -37.09%) > Direct compact success rate % 2.86 ( +0.00%) 2.82 ( -0.96%) > Compact daemon scanned migrate 3370059.62 ( +0.00%) 3612054.76 ( +7.18%) > Compact daemon scanned free 7718439.20 ( +0.00%) 5386385.02 ( -30.21%) > Compact direct scanned migrate 309248.62 ( +0.00%) 176721.04 ( -42.85%) > Compact direct scanned free 433582.84 ( +0.00%) 315727.66 ( -27.18%) > Compact migrate scanned daemon % 91.20 ( +0.00%) 94.48 ( +3.56%) > Compact free scanned daemon % 94.58 ( +0.00%) 94.42 ( -0.16%) > Compact total migrate scanned 3679308.24 ( +0.00%) 3788775.80 ( +2.98%) > Compact total free scanned 8152022.04 ( +0.00%) 5702112.68 ( -30.05%) > Alloc stall 872.04 ( +0.00%) 5156.12 ( +490.71%) > Pages kswapd scanned 510645.86 ( +0.00%) 3394.94 ( -99.33%) > Pages kswapd reclaimed 134811.62 ( +0.00%) 2701.26 ( -98.00%) > Pages direct scanned 99546.06 ( +0.00%) 376407.52 ( +278.12%) > Pages direct reclaimed 62123.40 ( +0.00%) 289535.70 ( +366.06%) > Pages total scanned 610191.92 ( +0.00%) 379802.46 ( -37.76%) > Pages scanned kswapd % 76.36 ( +0.00%) 0.10 ( -98.58%) > Swap out 12057.54 ( +0.00%) 15022.98 ( +24.59%) > Swap in 209.16 ( +0.00%) 256.48 ( +22.52%) > File refaults 17701.64 ( +0.00%) 11765.40 ( -33.53%) > > Huge page success rate is higher, allocation latencies are shorter and > more predictable. > > Stealing (fallback) rate is drastically reduced. Notably, while the > vanilla kernel keeps doing fallbacks on an ongoing basis, the patched > kernel enters a steady state once the distribution of block types is > adequate for the workload. Steals over 50 runs: > > VANILLA PATCHED > 1504.0 227.0 > 1557.0 6.0 > 1391.0 13.0 > 1080.0 26.0 > 1057.0 40.0 > 1156.0 6.0 > 805.0 46.0 > 736.0 20.0 > 1747.0 2.0 > 1699.0 34.0 > 1269.0 13.0 > 1858.0 12.0 > 907.0 4.0 > 727.0 2.0 > 563.0 2.0 > 3094.0 2.0 > 10211.0 3.0 > 2621.0 1.0 > 5508.0 2.0 > 1060.0 2.0 > 538.0 3.0 > 5773.0 2.0 > 2199.0 0.0 > 3781.0 2.0 > 1387.0 1.0 > 4977.0 0.0 > 2865.0 1.0 > 1814.0 1.0 > 3739.0 1.0 > 6857.0 0.0 > 382.0 0.0 > 407.0 1.0 > 3784.0 0.0 > 297.0 0.0 > 298.0 0.0 > 6636.0 0.0 > 4188.0 0.0 > 242.0 0.0 > 9960.0 0.0 > 5816.0 0.0 > 354.0 0.0 > 287.0 0.0 > 261.0 0.0 > 140.0 1.0 > 2065.0 0.0 > 312.0 0.0 > 331.0 0.0 > 164.0 0.0 > 465.0 1.0 > 219.0 0.0 > > Type mismatches are down too. Those count every time an allocation > request asks for one migratetype and gets another. This can still > occur minimally in the patched kernel due to non-stealing fallbacks, > but it's quite rare and follows the pattern of overall fallbacks - > once the block type distribution settles, mismatches cease as well: > > VANILLA: PATCHED: > 182602.0 268.0 > 135794.0 20.0 > 88619.0 19.0 > 95973.0 0.0 > 129590.0 0.0 > 129298.0 0.0 > 147134.0 0.0 > 230854.0 0.0 > 239709.0 0.0 > 137670.0 0.0 > 132430.0 0.0 > 65712.0 0.0 > 57901.0 0.0 > 67506.0 0.0 > 63565.0 4.0 > 34806.0 0.0 > 42962.0 0.0 > 32406.0 0.0 > 38668.0 0.0 > 61356.0 0.0 > 57800.0 0.0 > 41435.0 0.0 > 83456.0 0.0 > 65048.0 0.0 > 28955.0 0.0 > 47597.0 0.0 > 75117.0 0.0 > 55564.0 0.0 > 38280.0 0.0 > 52404.0 0.0 > 26264.0 0.0 > 37538.0 0.0 > 19671.0 0.0 > 30936.0 0.0 > 26933.0 0.0 > 16962.0 0.0 > 44554.0 0.0 > 46352.0 0.0 > 24995.0 0.0 > 35152.0 0.0 > 12823.0 0.0 > 21583.0 0.0 > 18129.0 0.0 > 31693.0 0.0 > 28745.0 0.0 > 33308.0 0.0 > 31114.0 0.0 > 35034.0 0.0 > 12111.0 0.0 > 24885.0 0.0 > > Compaction work is markedly reduced despite much better THP rates. > > In the vanilla kernel, reclaim seems to have been driven primarily by > watermark boosting that happens as a result of fallbacks. With those > all but eliminated, watermarks average lower and kswapd does less > work. The uptick in direct reclaim is because THP requests have to > fend for themselves more often - which is intended policy right > now. Aggregate reclaim activity is lowered significantly, though. This series significantly regresses Android and ChromeOS under memory pressure. THPs are virtually nonexistent on client devices, and IIRC, it was mentioned in the early discussions that potential regressions for such a case are somewhat expected? On Android (ARMv8.2), app launch time regressed by about 7%; On ChromeOS (Intel ADL), tab switch time regressed by about 8%. Also PSI (full and some) on both platforms increased by over 20%. I could post the details of the benchmarks and the metrics they measure, but I doubt they would mean much to you. I did ask our test teams to save extra kernel logs that might be more helpful, and I could forward them to you. Note that the numbers above were from the default LRU, not MGLRU, which I specifically asked our test teams to disable to double check the regressions. Given the merge window will be open soon, I don't plan to stand in its way. If we can't fix the regression after a reasonable amount of time, can we find a way to disable this series runtime/build time?
On Fri, May 10, 2024 at 11:14:43PM -0600, Yu Zhao wrote: > On Wed, Mar 20, 2024 at 12:04 PM Johannes Weiner <hannes@cmpxchg.org> wrote: > > > > V4: > > - fixed !pcp_order_allowed() case in free_unref_folios() > > - reworded the patch 0 changelog a bit for the git log > > - rebased to mm-everything-2024-03-19-23-01 > > - runtime-tested again with various CONFIG_DEBUG_FOOs enabled > > > > --- > > > > The page allocator's mobility grouping is intended to keep unmovable > > pages separate from reclaimable/compactable ones to allow on-demand > > defragmentation for higher-order allocations and huge pages. > > > > Currently, there are several places where accidental type mixing > > occurs: an allocation asks for a page of a certain migratetype and > > receives another. This ruins pageblocks for compaction, which in turn > > makes allocating huge pages more expensive and less reliable. > > > > The series addresses those causes. The last patch adds type checks on > > all freelist movements to prevent new violations being introduced. > > > > The benefits can be seen in a mixed workload that stresses the machine > > with a memcache-type workload and a kernel build job while > > periodically attempting to allocate batches of THP. The following data > > is aggregated over 50 consecutive defconfig builds: > > > > VANILLA PATCHED > > Hugealloc Time mean 165843.93 ( +0.00%) 113025.88 ( -31.85%) > > Hugealloc Time stddev 158957.35 ( +0.00%) 114716.07 ( -27.83%) > > Kbuild Real time 310.24 ( +0.00%) 300.73 ( -3.06%) > > Kbuild User time 1271.13 ( +0.00%) 1259.42 ( -0.92%) > > Kbuild System time 582.02 ( +0.00%) 559.79 ( -3.81%) > > THP fault alloc 30585.14 ( +0.00%) 40853.62 ( +33.57%) > > THP fault fallback 36626.46 ( +0.00%) 26357.62 ( -28.04%) > > THP fault fail rate % 54.49 ( +0.00%) 39.22 ( -27.53%) > > Pagealloc fallback 1328.00 ( +0.00%) 1.00 ( -99.85%) > > Pagealloc type mismatch 181009.50 ( +0.00%) 0.00 ( -100.00%) > > Direct compact stall 434.56 ( +0.00%) 257.66 ( -40.61%) > > Direct compact fail 421.70 ( +0.00%) 249.94 ( -40.63%) > > Direct compact success 12.86 ( +0.00%) 7.72 ( -37.09%) > > Direct compact success rate % 2.86 ( +0.00%) 2.82 ( -0.96%) > > Compact daemon scanned migrate 3370059.62 ( +0.00%) 3612054.76 ( +7.18%) > > Compact daemon scanned free 7718439.20 ( +0.00%) 5386385.02 ( -30.21%) > > Compact direct scanned migrate 309248.62 ( +0.00%) 176721.04 ( -42.85%) > > Compact direct scanned free 433582.84 ( +0.00%) 315727.66 ( -27.18%) > > Compact migrate scanned daemon % 91.20 ( +0.00%) 94.48 ( +3.56%) > > Compact free scanned daemon % 94.58 ( +0.00%) 94.42 ( -0.16%) > > Compact total migrate scanned 3679308.24 ( +0.00%) 3788775.80 ( +2.98%) > > Compact total free scanned 8152022.04 ( +0.00%) 5702112.68 ( -30.05%) > > Alloc stall 872.04 ( +0.00%) 5156.12 ( +490.71%) > > Pages kswapd scanned 510645.86 ( +0.00%) 3394.94 ( -99.33%) > > Pages kswapd reclaimed 134811.62 ( +0.00%) 2701.26 ( -98.00%) > > Pages direct scanned 99546.06 ( +0.00%) 376407.52 ( +278.12%) > > Pages direct reclaimed 62123.40 ( +0.00%) 289535.70 ( +366.06%) > > Pages total scanned 610191.92 ( +0.00%) 379802.46 ( -37.76%) > > Pages scanned kswapd % 76.36 ( +0.00%) 0.10 ( -98.58%) > > Swap out 12057.54 ( +0.00%) 15022.98 ( +24.59%) > > Swap in 209.16 ( +0.00%) 256.48 ( +22.52%) > > File refaults 17701.64 ( +0.00%) 11765.40 ( -33.53%) > > > > Huge page success rate is higher, allocation latencies are shorter and > > more predictable. > > > > Stealing (fallback) rate is drastically reduced. Notably, while the > > vanilla kernel keeps doing fallbacks on an ongoing basis, the patched > > kernel enters a steady state once the distribution of block types is > > adequate for the workload. Steals over 50 runs: > > > > VANILLA PATCHED > > 1504.0 227.0 > > 1557.0 6.0 > > 1391.0 13.0 > > 1080.0 26.0 > > 1057.0 40.0 > > 1156.0 6.0 > > 805.0 46.0 > > 736.0 20.0 > > 1747.0 2.0 > > 1699.0 34.0 > > 1269.0 13.0 > > 1858.0 12.0 > > 907.0 4.0 > > 727.0 2.0 > > 563.0 2.0 > > 3094.0 2.0 > > 10211.0 3.0 > > 2621.0 1.0 > > 5508.0 2.0 > > 1060.0 2.0 > > 538.0 3.0 > > 5773.0 2.0 > > 2199.0 0.0 > > 3781.0 2.0 > > 1387.0 1.0 > > 4977.0 0.0 > > 2865.0 1.0 > > 1814.0 1.0 > > 3739.0 1.0 > > 6857.0 0.0 > > 382.0 0.0 > > 407.0 1.0 > > 3784.0 0.0 > > 297.0 0.0 > > 298.0 0.0 > > 6636.0 0.0 > > 4188.0 0.0 > > 242.0 0.0 > > 9960.0 0.0 > > 5816.0 0.0 > > 354.0 0.0 > > 287.0 0.0 > > 261.0 0.0 > > 140.0 1.0 > > 2065.0 0.0 > > 312.0 0.0 > > 331.0 0.0 > > 164.0 0.0 > > 465.0 1.0 > > 219.0 0.0 > > > > Type mismatches are down too. Those count every time an allocation > > request asks for one migratetype and gets another. This can still > > occur minimally in the patched kernel due to non-stealing fallbacks, > > but it's quite rare and follows the pattern of overall fallbacks - > > once the block type distribution settles, mismatches cease as well: > > > > VANILLA: PATCHED: > > 182602.0 268.0 > > 135794.0 20.0 > > 88619.0 19.0 > > 95973.0 0.0 > > 129590.0 0.0 > > 129298.0 0.0 > > 147134.0 0.0 > > 230854.0 0.0 > > 239709.0 0.0 > > 137670.0 0.0 > > 132430.0 0.0 > > 65712.0 0.0 > > 57901.0 0.0 > > 67506.0 0.0 > > 63565.0 4.0 > > 34806.0 0.0 > > 42962.0 0.0 > > 32406.0 0.0 > > 38668.0 0.0 > > 61356.0 0.0 > > 57800.0 0.0 > > 41435.0 0.0 > > 83456.0 0.0 > > 65048.0 0.0 > > 28955.0 0.0 > > 47597.0 0.0 > > 75117.0 0.0 > > 55564.0 0.0 > > 38280.0 0.0 > > 52404.0 0.0 > > 26264.0 0.0 > > 37538.0 0.0 > > 19671.0 0.0 > > 30936.0 0.0 > > 26933.0 0.0 > > 16962.0 0.0 > > 44554.0 0.0 > > 46352.0 0.0 > > 24995.0 0.0 > > 35152.0 0.0 > > 12823.0 0.0 > > 21583.0 0.0 > > 18129.0 0.0 > > 31693.0 0.0 > > 28745.0 0.0 > > 33308.0 0.0 > > 31114.0 0.0 > > 35034.0 0.0 > > 12111.0 0.0 > > 24885.0 0.0 > > > > Compaction work is markedly reduced despite much better THP rates. > > > > In the vanilla kernel, reclaim seems to have been driven primarily by > > watermark boosting that happens as a result of fallbacks. With those > > all but eliminated, watermarks average lower and kswapd does less > > work. The uptick in direct reclaim is because THP requests have to > > fend for themselves more often - which is intended policy right > > now. Aggregate reclaim activity is lowered significantly, though. > > This series significantly regresses Android and ChromeOS under memory > pressure. THPs are virtually nonexistent on client devices, and IIRC, > it was mentioned in the early discussions that potential regressions > for such a case are somewhat expected? This is not expected for the 10 patches here. You might be referring to the discussion around the huge page allocator series, which had fallback restrictions and many changes to reclaim and compaction. Can you confirm that you were testing the latest patches that are in mm-stable as of today? There was a series of follow-up fixes. Especially, please double check you have the follow-up fixes to compaction capturing and the CMA fallback policy. It sounds like the behavior Baolin described before the CMA fix. Lastly, what's the base you backported this series to? > On Android (ARMv8.2), app launch time regressed by about 7%; On > ChromeOS (Intel ADL), tab switch time regressed by about 8%. Also PSI > (full and some) on both platforms increased by over 20%. I could post > the details of the benchmarks and the metrics they measure, but I > doubt they would mean much to you. I did ask our test teams to save > extra kernel logs that might be more helpful, and I could forward them > to you. If the issue persists with the latest patches in -mm, a kernel config and snapshots of /proc/vmstat, /proc/pagetypeinfo, /proc/zoneinfo before/during/after the problematic behavior would be very helpful.
On Mon, May 13, 2024 at 10:03 AM Johannes Weiner <hannes@cmpxchg.org> wrote:
>
> On Fri, May 10, 2024 at 11:14:43PM -0600, Yu Zhao wrote:
> > On Wed, Mar 20, 2024 at 12:04 PM Johannes Weiner <hannes@cmpxchg.org> wrote:
> > >
> > > V4:
> > > - fixed !pcp_order_allowed() case in free_unref_folios()
> > > - reworded the patch 0 changelog a bit for the git log
> > > - rebased to mm-everything-2024-03-19-23-01
> > > - runtime-tested again with various CONFIG_DEBUG_FOOs enabled
> > >
> > > ---
> > >
> > > The page allocator's mobility grouping is intended to keep unmovable
> > > pages separate from reclaimable/compactable ones to allow on-demand
> > > defragmentation for higher-order allocations and huge pages.
> > >
> > > Currently, there are several places where accidental type mixing
> > > occurs: an allocation asks for a page of a certain migratetype and
> > > receives another. This ruins pageblocks for compaction, which in turn
> > > makes allocating huge pages more expensive and less reliable.
> > >
> > > The series addresses those causes. The last patch adds type checks on
> > > all freelist movements to prevent new violations being introduced.
> > >
> > > The benefits can be seen in a mixed workload that stresses the machine
> > > with a memcache-type workload and a kernel build job while
> > > periodically attempting to allocate batches of THP. The following data
> > > is aggregated over 50 consecutive defconfig builds:
> > >
> > > VANILLA PATCHED
> > > Hugealloc Time mean 165843.93 ( +0.00%) 113025.88 ( -31.85%)
> > > Hugealloc Time stddev 158957.35 ( +0.00%) 114716.07 ( -27.83%)
> > > Kbuild Real time 310.24 ( +0.00%) 300.73 ( -3.06%)
> > > Kbuild User time 1271.13 ( +0.00%) 1259.42 ( -0.92%)
> > > Kbuild System time 582.02 ( +0.00%) 559.79 ( -3.81%)
> > > THP fault alloc 30585.14 ( +0.00%) 40853.62 ( +33.57%)
> > > THP fault fallback 36626.46 ( +0.00%) 26357.62 ( -28.04%)
> > > THP fault fail rate % 54.49 ( +0.00%) 39.22 ( -27.53%)
> > > Pagealloc fallback 1328.00 ( +0.00%) 1.00 ( -99.85%)
> > > Pagealloc type mismatch 181009.50 ( +0.00%) 0.00 ( -100.00%)
> > > Direct compact stall 434.56 ( +0.00%) 257.66 ( -40.61%)
> > > Direct compact fail 421.70 ( +0.00%) 249.94 ( -40.63%)
> > > Direct compact success 12.86 ( +0.00%) 7.72 ( -37.09%)
> > > Direct compact success rate % 2.86 ( +0.00%) 2.82 ( -0.96%)
> > > Compact daemon scanned migrate 3370059.62 ( +0.00%) 3612054.76 ( +7.18%)
> > > Compact daemon scanned free 7718439.20 ( +0.00%) 5386385.02 ( -30.21%)
> > > Compact direct scanned migrate 309248.62 ( +0.00%) 176721.04 ( -42.85%)
> > > Compact direct scanned free 433582.84 ( +0.00%) 315727.66 ( -27.18%)
> > > Compact migrate scanned daemon % 91.20 ( +0.00%) 94.48 ( +3.56%)
> > > Compact free scanned daemon % 94.58 ( +0.00%) 94.42 ( -0.16%)
> > > Compact total migrate scanned 3679308.24 ( +0.00%) 3788775.80 ( +2.98%)
> > > Compact total free scanned 8152022.04 ( +0.00%) 5702112.68 ( -30.05%)
> > > Alloc stall 872.04 ( +0.00%) 5156.12 ( +490.71%)
> > > Pages kswapd scanned 510645.86 ( +0.00%) 3394.94 ( -99.33%)
> > > Pages kswapd reclaimed 134811.62 ( +0.00%) 2701.26 ( -98.00%)
> > > Pages direct scanned 99546.06 ( +0.00%) 376407.52 ( +278.12%)
> > > Pages direct reclaimed 62123.40 ( +0.00%) 289535.70 ( +366.06%)
> > > Pages total scanned 610191.92 ( +0.00%) 379802.46 ( -37.76%)
> > > Pages scanned kswapd % 76.36 ( +0.00%) 0.10 ( -98.58%)
> > > Swap out 12057.54 ( +0.00%) 15022.98 ( +24.59%)
> > > Swap in 209.16 ( +0.00%) 256.48 ( +22.52%)
> > > File refaults 17701.64 ( +0.00%) 11765.40 ( -33.53%)
> > >
> > > Huge page success rate is higher, allocation latencies are shorter and
> > > more predictable.
> > >
> > > Stealing (fallback) rate is drastically reduced. Notably, while the
> > > vanilla kernel keeps doing fallbacks on an ongoing basis, the patched
> > > kernel enters a steady state once the distribution of block types is
> > > adequate for the workload. Steals over 50 runs:
> > >
> > > VANILLA PATCHED
> > > 1504.0 227.0
> > > 1557.0 6.0
> > > 1391.0 13.0
> > > 1080.0 26.0
> > > 1057.0 40.0
> > > 1156.0 6.0
> > > 805.0 46.0
> > > 736.0 20.0
> > > 1747.0 2.0
> > > 1699.0 34.0
> > > 1269.0 13.0
> > > 1858.0 12.0
> > > 907.0 4.0
> > > 727.0 2.0
> > > 563.0 2.0
> > > 3094.0 2.0
> > > 10211.0 3.0
> > > 2621.0 1.0
> > > 5508.0 2.0
> > > 1060.0 2.0
> > > 538.0 3.0
> > > 5773.0 2.0
> > > 2199.0 0.0
> > > 3781.0 2.0
> > > 1387.0 1.0
> > > 4977.0 0.0
> > > 2865.0 1.0
> > > 1814.0 1.0
> > > 3739.0 1.0
> > > 6857.0 0.0
> > > 382.0 0.0
> > > 407.0 1.0
> > > 3784.0 0.0
> > > 297.0 0.0
> > > 298.0 0.0
> > > 6636.0 0.0
> > > 4188.0 0.0
> > > 242.0 0.0
> > > 9960.0 0.0
> > > 5816.0 0.0
> > > 354.0 0.0
> > > 287.0 0.0
> > > 261.0 0.0
> > > 140.0 1.0
> > > 2065.0 0.0
> > > 312.0 0.0
> > > 331.0 0.0
> > > 164.0 0.0
> > > 465.0 1.0
> > > 219.0 0.0
> > >
> > > Type mismatches are down too. Those count every time an allocation
> > > request asks for one migratetype and gets another. This can still
> > > occur minimally in the patched kernel due to non-stealing fallbacks,
> > > but it's quite rare and follows the pattern of overall fallbacks -
> > > once the block type distribution settles, mismatches cease as well:
> > >
> > > VANILLA: PATCHED:
> > > 182602.0 268.0
> > > 135794.0 20.0
> > > 88619.0 19.0
> > > 95973.0 0.0
> > > 129590.0 0.0
> > > 129298.0 0.0
> > > 147134.0 0.0
> > > 230854.0 0.0
> > > 239709.0 0.0
> > > 137670.0 0.0
> > > 132430.0 0.0
> > > 65712.0 0.0
> > > 57901.0 0.0
> > > 67506.0 0.0
> > > 63565.0 4.0
> > > 34806.0 0.0
> > > 42962.0 0.0
> > > 32406.0 0.0
> > > 38668.0 0.0
> > > 61356.0 0.0
> > > 57800.0 0.0
> > > 41435.0 0.0
> > > 83456.0 0.0
> > > 65048.0 0.0
> > > 28955.0 0.0
> > > 47597.0 0.0
> > > 75117.0 0.0
> > > 55564.0 0.0
> > > 38280.0 0.0
> > > 52404.0 0.0
> > > 26264.0 0.0
> > > 37538.0 0.0
> > > 19671.0 0.0
> > > 30936.0 0.0
> > > 26933.0 0.0
> > > 16962.0 0.0
> > > 44554.0 0.0
> > > 46352.0 0.0
> > > 24995.0 0.0
> > > 35152.0 0.0
> > > 12823.0 0.0
> > > 21583.0 0.0
> > > 18129.0 0.0
> > > 31693.0 0.0
> > > 28745.0 0.0
> > > 33308.0 0.0
> > > 31114.0 0.0
> > > 35034.0 0.0
> > > 12111.0 0.0
> > > 24885.0 0.0
> > >
> > > Compaction work is markedly reduced despite much better THP rates.
> > >
> > > In the vanilla kernel, reclaim seems to have been driven primarily by
> > > watermark boosting that happens as a result of fallbacks. With those
> > > all but eliminated, watermarks average lower and kswapd does less
> > > work. The uptick in direct reclaim is because THP requests have to
> > > fend for themselves more often - which is intended policy right
> > > now. Aggregate reclaim activity is lowered significantly, though.
> >
> > This series significantly regresses Android and ChromeOS under memory
> > pressure. THPs are virtually nonexistent on client devices, and IIRC,
> > it was mentioned in the early discussions that potential regressions
> > for such a case are somewhat expected?
>
> This is not expected for the 10 patches here. You might be referring
> to the discussion around the huge page allocator series, which had
> fallback restrictions and many changes to reclaim and compaction.
Right, now I remember.
> Can you confirm that you were testing the latest patches that are in
> mm-stable as of today? There was a series of follow-up fixes.
Here is what I have on top of 6.8.y, which I think includes all the
follow-up fixes. The performance delta was measured between 5 & 22.
1 mm: convert free_unref_page_list() to use folios
2 mm: add free_unref_folios()
3 mm: handle large folios in free_unref_folios()
4 mm/page_alloc: remove unused fpi_flags in free_pages_prepare()
5 mm: add alloc_contig_migrate_range allocation statistics
6 mm: page_alloc: remove pcppage migratetype caching
7 mm: page_alloc: optimize free_unref_folios()
8 mm: page_alloc: fix up block types when merging compatible blocks
9 mm: page_alloc: move free pages when converting block during isolation
10 mm: page_alloc: fix move_freepages_block() range error
11 mm: page_alloc: fix freelist movement during block conversion
12 mm-page_alloc-fix-freelist-movement-during-block-conversion-fix
13 mm: page_alloc: close migratetype race between freeing and stealing
14 mm: page_alloc: set migratetype inside move_freepages()
15 mm: page_isolation: prepare for hygienic freelists
16 mm-page_isolation-prepare-for-hygienic-freelists-fix
17 mm: page_alloc: consolidate free page accounting
18 mm: page_alloc: consolidate free page accounting fix
19 mm: page_alloc: consolidate free page accounting fix 2
20 mm: page_alloc: consolidate free page accounting fix 3
21 mm: page_alloc: change move_freepages() to __move_freepages_block()
22 mm: page_alloc: batch vmstat updates in expand()
> Especially, please double check you have the follow-up fixes to
> compaction capturing and the CMA fallback policy. It sounds like the
> behavior Baolin described before the CMA fix.
Yes, that one was included.
> Lastly, what's the base you backported this series to?
It was 6.8, we can potentially try 6.9 this week and 6.10-rc in a few
weeks when it's in good shape for performance benchmarks.
> > On Android (ARMv8.2), app launch time regressed by about 7%; On
> > ChromeOS (Intel ADL), tab switch time regressed by about 8%. Also PSI
> > (full and some) on both platforms increased by over 20%. I could post
> > the details of the benchmarks and the metrics they measure, but I
> > doubt they would mean much to you. I did ask our test teams to save
> > extra kernel logs that might be more helpful, and I could forward them
> > to you.
>
> If the issue persists with the latest patches in -mm, a kernel config
> and snapshots of /proc/vmstat, /proc/pagetypeinfo, /proc/zoneinfo
> before/during/after the problematic behavior would be very helpful.
Assuming all the fixes were included, do you want the logs from 6.8?
We have them available now.
On Mon, May 13, 2024 at 12:10:04PM -0600, Yu Zhao wrote: > On Mon, May 13, 2024 at 10:03 AM Johannes Weiner <hannes@cmpxchg.org> wrote: > > > > On Fri, May 10, 2024 at 11:14:43PM -0600, Yu Zhao wrote: > > > On Wed, Mar 20, 2024 at 12:04 PM Johannes Weiner <hannes@cmpxchg.org> wrote: > > > > > > > > V4: > > > > - fixed !pcp_order_allowed() case in free_unref_folios() > > > > - reworded the patch 0 changelog a bit for the git log > > > > - rebased to mm-everything-2024-03-19-23-01 > > > > - runtime-tested again with various CONFIG_DEBUG_FOOs enabled > > > > > > > > --- > > > > > > > > The page allocator's mobility grouping is intended to keep unmovable > > > > pages separate from reclaimable/compactable ones to allow on-demand > > > > defragmentation for higher-order allocations and huge pages. > > > > > > > > Currently, there are several places where accidental type mixing > > > > occurs: an allocation asks for a page of a certain migratetype and > > > > receives another. This ruins pageblocks for compaction, which in turn > > > > makes allocating huge pages more expensive and less reliable. > > > > > > > > The series addresses those causes. The last patch adds type checks on > > > > all freelist movements to prevent new violations being introduced. > > > > > > > > The benefits can be seen in a mixed workload that stresses the machine > > > > with a memcache-type workload and a kernel build job while > > > > periodically attempting to allocate batches of THP. The following data > > > > is aggregated over 50 consecutive defconfig builds: > > > > > > > > VANILLA PATCHED > > > > Hugealloc Time mean 165843.93 ( +0.00%) 113025.88 ( -31.85%) > > > > Hugealloc Time stddev 158957.35 ( +0.00%) 114716.07 ( -27.83%) > > > > Kbuild Real time 310.24 ( +0.00%) 300.73 ( -3.06%) > > > > Kbuild User time 1271.13 ( +0.00%) 1259.42 ( -0.92%) > > > > Kbuild System time 582.02 ( +0.00%) 559.79 ( -3.81%) > > > > THP fault alloc 30585.14 ( +0.00%) 40853.62 ( +33.57%) > > > > THP fault fallback 36626.46 ( +0.00%) 26357.62 ( -28.04%) > > > > THP fault fail rate % 54.49 ( +0.00%) 39.22 ( -27.53%) > > > > Pagealloc fallback 1328.00 ( +0.00%) 1.00 ( -99.85%) > > > > Pagealloc type mismatch 181009.50 ( +0.00%) 0.00 ( -100.00%) > > > > Direct compact stall 434.56 ( +0.00%) 257.66 ( -40.61%) > > > > Direct compact fail 421.70 ( +0.00%) 249.94 ( -40.63%) > > > > Direct compact success 12.86 ( +0.00%) 7.72 ( -37.09%) > > > > Direct compact success rate % 2.86 ( +0.00%) 2.82 ( -0.96%) > > > > Compact daemon scanned migrate 3370059.62 ( +0.00%) 3612054.76 ( +7.18%) > > > > Compact daemon scanned free 7718439.20 ( +0.00%) 5386385.02 ( -30.21%) > > > > Compact direct scanned migrate 309248.62 ( +0.00%) 176721.04 ( -42.85%) > > > > Compact direct scanned free 433582.84 ( +0.00%) 315727.66 ( -27.18%) > > > > Compact migrate scanned daemon % 91.20 ( +0.00%) 94.48 ( +3.56%) > > > > Compact free scanned daemon % 94.58 ( +0.00%) 94.42 ( -0.16%) > > > > Compact total migrate scanned 3679308.24 ( +0.00%) 3788775.80 ( +2.98%) > > > > Compact total free scanned 8152022.04 ( +0.00%) 5702112.68 ( -30.05%) > > > > Alloc stall 872.04 ( +0.00%) 5156.12 ( +490.71%) > > > > Pages kswapd scanned 510645.86 ( +0.00%) 3394.94 ( -99.33%) > > > > Pages kswapd reclaimed 134811.62 ( +0.00%) 2701.26 ( -98.00%) > > > > Pages direct scanned 99546.06 ( +0.00%) 376407.52 ( +278.12%) > > > > Pages direct reclaimed 62123.40 ( +0.00%) 289535.70 ( +366.06%) > > > > Pages total scanned 610191.92 ( +0.00%) 379802.46 ( -37.76%) > > > > Pages scanned kswapd % 76.36 ( +0.00%) 0.10 ( -98.58%) > > > > Swap out 12057.54 ( +0.00%) 15022.98 ( +24.59%) > > > > Swap in 209.16 ( +0.00%) 256.48 ( +22.52%) > > > > File refaults 17701.64 ( +0.00%) 11765.40 ( -33.53%) [...] > > > > > > This series significantly regresses Android and ChromeOS under memory > > > pressure. THPs are virtually nonexistent on client devices, and IIRC, > > > it was mentioned in the early discussions that potential regressions > > > for such a case are somewhat expected? > > > > This is not expected for the 10 patches here. You might be referring > > to the discussion around the huge page allocator series, which had > > fallback restrictions and many changes to reclaim and compaction. > > Right, now I remember. > > > Can you confirm that you were testing the latest patches that are in > > mm-stable as of today? There was a series of follow-up fixes. > > Here is what I have on top of 6.8.y, which I think includes all the > follow-up fixes. The performance delta was measured between 5 & 22. > > 1 mm: convert free_unref_page_list() to use folios > 2 mm: add free_unref_folios() > 3 mm: handle large folios in free_unref_folios() > 4 mm/page_alloc: remove unused fpi_flags in free_pages_prepare() > 5 mm: add alloc_contig_migrate_range allocation statistics > 6 mm: page_alloc: remove pcppage migratetype caching > 7 mm: page_alloc: optimize free_unref_folios() > 8 mm: page_alloc: fix up block types when merging compatible blocks > 9 mm: page_alloc: move free pages when converting block during isolation > 10 mm: page_alloc: fix move_freepages_block() range error > 11 mm: page_alloc: fix freelist movement during block conversion > 12 mm-page_alloc-fix-freelist-movement-during-block-conversion-fix > 13 mm: page_alloc: close migratetype race between freeing and stealing > 14 mm: page_alloc: set migratetype inside move_freepages() > 15 mm: page_isolation: prepare for hygienic freelists > 16 mm-page_isolation-prepare-for-hygienic-freelists-fix > 17 mm: page_alloc: consolidate free page accounting > 18 mm: page_alloc: consolidate free page accounting fix > 19 mm: page_alloc: consolidate free page accounting fix 2 > 20 mm: page_alloc: consolidate free page accounting fix 3 > 21 mm: page_alloc: change move_freepages() to __move_freepages_block() > 22 mm: page_alloc: batch vmstat updates in expand() It does look complete to me. Did you encounter any conflicts during the backport? Is there any chance you can fold the fixes into their respective main patches and bisect the sequence? Again, it's not expected behavior given the fairly conservative changes above. It sounds like a bug. > > Especially, please double check you have the follow-up fixes to > > compaction capturing and the CMA fallback policy. It sounds like the > > behavior Baolin described before the CMA fix. > > Yes, that one was included. Ok. > > Lastly, what's the base you backported this series to? > > It was 6.8, we can potentially try 6.9 this week and 6.10-rc in a few > weeks when it's in good shape for performance benchmarks. If you could try 6.9 as well, that would be great. I backported the series to 6.9 the other day (git cherry-picks from mm-stable) and I didn't encounter any conflicts. > > > On Android (ARMv8.2), app launch time regressed by about 7%; On > > > ChromeOS (Intel ADL), tab switch time regressed by about 8%. Also PSI > > > (full and some) on both platforms increased by over 20%. I could post > > > the details of the benchmarks and the metrics they measure, but I > > > doubt they would mean much to you. I did ask our test teams to save > > > extra kernel logs that might be more helpful, and I could forward them > > > to you. > > > > If the issue persists with the latest patches in -mm, a kernel config > > and snapshots of /proc/vmstat, /proc/pagetypeinfo, /proc/zoneinfo > > before/during/after the problematic behavior would be very helpful. > > Assuming all the fixes were included, do you want the logs from 6.8? > We have them available now. Yes, that would be helpful. If you have them, it would also be quite useful to have the vmstat before-after-test delta from a good kernel, for baseline comparison. Thanks
On Mon, May 13, 2024 at 1:04 PM Johannes Weiner <hannes@cmpxchg.org> wrote:
>
> On Mon, May 13, 2024 at 12:10:04PM -0600, Yu Zhao wrote:
> > On Mon, May 13, 2024 at 10:03 AM Johannes Weiner <hannes@cmpxchg.org> wrote:
> > >
> > > On Fri, May 10, 2024 at 11:14:43PM -0600, Yu Zhao wrote:
> > > > On Wed, Mar 20, 2024 at 12:04 PM Johannes Weiner <hannes@cmpxchg.org> wrote:
> > > > >
> > > > > V4:
> > > > > - fixed !pcp_order_allowed() case in free_unref_folios()
> > > > > - reworded the patch 0 changelog a bit for the git log
> > > > > - rebased to mm-everything-2024-03-19-23-01
> > > > > - runtime-tested again with various CONFIG_DEBUG_FOOs enabled
> > > > >
> > > > > ---
> > > > >
> > > > > The page allocator's mobility grouping is intended to keep unmovable
> > > > > pages separate from reclaimable/compactable ones to allow on-demand
> > > > > defragmentation for higher-order allocations and huge pages.
> > > > >
> > > > > Currently, there are several places where accidental type mixing
> > > > > occurs: an allocation asks for a page of a certain migratetype and
> > > > > receives another. This ruins pageblocks for compaction, which in turn
> > > > > makes allocating huge pages more expensive and less reliable.
> > > > >
> > > > > The series addresses those causes. The last patch adds type checks on
> > > > > all freelist movements to prevent new violations being introduced.
> > > > >
> > > > > The benefits can be seen in a mixed workload that stresses the machine
> > > > > with a memcache-type workload and a kernel build job while
> > > > > periodically attempting to allocate batches of THP. The following data
> > > > > is aggregated over 50 consecutive defconfig builds:
> > > > >
> > > > > VANILLA PATCHED
> > > > > Hugealloc Time mean 165843.93 ( +0.00%) 113025.88 ( -31.85%)
> > > > > Hugealloc Time stddev 158957.35 ( +0.00%) 114716.07 ( -27.83%)
> > > > > Kbuild Real time 310.24 ( +0.00%) 300.73 ( -3.06%)
> > > > > Kbuild User time 1271.13 ( +0.00%) 1259.42 ( -0.92%)
> > > > > Kbuild System time 582.02 ( +0.00%) 559.79 ( -3.81%)
> > > > > THP fault alloc 30585.14 ( +0.00%) 40853.62 ( +33.57%)
> > > > > THP fault fallback 36626.46 ( +0.00%) 26357.62 ( -28.04%)
> > > > > THP fault fail rate % 54.49 ( +0.00%) 39.22 ( -27.53%)
> > > > > Pagealloc fallback 1328.00 ( +0.00%) 1.00 ( -99.85%)
> > > > > Pagealloc type mismatch 181009.50 ( +0.00%) 0.00 ( -100.00%)
> > > > > Direct compact stall 434.56 ( +0.00%) 257.66 ( -40.61%)
> > > > > Direct compact fail 421.70 ( +0.00%) 249.94 ( -40.63%)
> > > > > Direct compact success 12.86 ( +0.00%) 7.72 ( -37.09%)
> > > > > Direct compact success rate % 2.86 ( +0.00%) 2.82 ( -0.96%)
> > > > > Compact daemon scanned migrate 3370059.62 ( +0.00%) 3612054.76 ( +7.18%)
> > > > > Compact daemon scanned free 7718439.20 ( +0.00%) 5386385.02 ( -30.21%)
> > > > > Compact direct scanned migrate 309248.62 ( +0.00%) 176721.04 ( -42.85%)
> > > > > Compact direct scanned free 433582.84 ( +0.00%) 315727.66 ( -27.18%)
> > > > > Compact migrate scanned daemon % 91.20 ( +0.00%) 94.48 ( +3.56%)
> > > > > Compact free scanned daemon % 94.58 ( +0.00%) 94.42 ( -0.16%)
> > > > > Compact total migrate scanned 3679308.24 ( +0.00%) 3788775.80 ( +2.98%)
> > > > > Compact total free scanned 8152022.04 ( +0.00%) 5702112.68 ( -30.05%)
> > > > > Alloc stall 872.04 ( +0.00%) 5156.12 ( +490.71%)
> > > > > Pages kswapd scanned 510645.86 ( +0.00%) 3394.94 ( -99.33%)
> > > > > Pages kswapd reclaimed 134811.62 ( +0.00%) 2701.26 ( -98.00%)
> > > > > Pages direct scanned 99546.06 ( +0.00%) 376407.52 ( +278.12%)
> > > > > Pages direct reclaimed 62123.40 ( +0.00%) 289535.70 ( +366.06%)
> > > > > Pages total scanned 610191.92 ( +0.00%) 379802.46 ( -37.76%)
> > > > > Pages scanned kswapd % 76.36 ( +0.00%) 0.10 ( -98.58%)
> > > > > Swap out 12057.54 ( +0.00%) 15022.98 ( +24.59%)
> > > > > Swap in 209.16 ( +0.00%) 256.48 ( +22.52%)
> > > > > File refaults 17701.64 ( +0.00%) 11765.40 ( -33.53%)
>
> [...]
>
> > > >
> > > > This series significantly regresses Android and ChromeOS under memory
> > > > pressure. THPs are virtually nonexistent on client devices, and IIRC,
> > > > it was mentioned in the early discussions that potential regressions
> > > > for such a case are somewhat expected?
> > >
> > > This is not expected for the 10 patches here. You might be referring
> > > to the discussion around the huge page allocator series, which had
> > > fallback restrictions and many changes to reclaim and compaction.
> >
> > Right, now I remember.
> >
> > > Can you confirm that you were testing the latest patches that are in
> > > mm-stable as of today? There was a series of follow-up fixes.
> >
> > Here is what I have on top of 6.8.y, which I think includes all the
> > follow-up fixes. The performance delta was measured between 5 & 22.
> >
> > 1 mm: convert free_unref_page_list() to use folios
> > 2 mm: add free_unref_folios()
> > 3 mm: handle large folios in free_unref_folios()
> > 4 mm/page_alloc: remove unused fpi_flags in free_pages_prepare()
> > 5 mm: add alloc_contig_migrate_range allocation statistics
> > 6 mm: page_alloc: remove pcppage migratetype caching
> > 7 mm: page_alloc: optimize free_unref_folios()
> > 8 mm: page_alloc: fix up block types when merging compatible blocks
> > 9 mm: page_alloc: move free pages when converting block during isolation
> > 10 mm: page_alloc: fix move_freepages_block() range error
> > 11 mm: page_alloc: fix freelist movement during block conversion
> > 12 mm-page_alloc-fix-freelist-movement-during-block-conversion-fix
> > 13 mm: page_alloc: close migratetype race between freeing and stealing
> > 14 mm: page_alloc: set migratetype inside move_freepages()
> > 15 mm: page_isolation: prepare for hygienic freelists
> > 16 mm-page_isolation-prepare-for-hygienic-freelists-fix
> > 17 mm: page_alloc: consolidate free page accounting
> > 18 mm: page_alloc: consolidate free page accounting fix
> > 19 mm: page_alloc: consolidate free page accounting fix 2
> > 20 mm: page_alloc: consolidate free page accounting fix 3
> > 21 mm: page_alloc: change move_freepages() to __move_freepages_block()
> > 22 mm: page_alloc: batch vmstat updates in expand()
>
> It does look complete to me. Did you encounter any conflicts during
> the backport? Is there any chance you can fold the fixes into their
> respective main patches and bisect the sequence?
>
> Again, it's not expected behavior given the fairly conservative
> changes above. It sounds like a bug.
>
> > > Especially, please double check you have the follow-up fixes to
> > > compaction capturing and the CMA fallback policy. It sounds like the
> > > behavior Baolin described before the CMA fix.
> >
> > Yes, that one was included.
>
> Ok.
>
> > > Lastly, what's the base you backported this series to?
> >
> > It was 6.8, we can potentially try 6.9 this week and 6.10-rc in a few
> > weeks when it's in good shape for performance benchmarks.
>
> If you could try 6.9 as well, that would be great. I backported the
> series to 6.9 the other day (git cherry-picks from mm-stable) and I
> didn't encounter any conflicts.
>
> > > > On Android (ARMv8.2), app launch time regressed by about 7%; On
> > > > ChromeOS (Intel ADL), tab switch time regressed by about 8%. Also PSI
> > > > (full and some) on both platforms increased by over 20%. I could post
> > > > the details of the benchmarks and the metrics they measure, but I
> > > > doubt they would mean much to you. I did ask our test teams to save
> > > > extra kernel logs that might be more helpful, and I could forward them
> > > > to you.
> > >
> > > If the issue persists with the latest patches in -mm, a kernel config
> > > and snapshots of /proc/vmstat, /proc/pagetypeinfo, /proc/zoneinfo
> > > before/during/after the problematic behavior would be very helpful.
> >
> > Assuming all the fixes were included, do you want the logs from 6.8?
> > We have them available now.
>
> Yes, that would be helpful.
>
> If you have them, it would also be quite useful to have the vmstat
> before-after-test delta from a good kernel, for baseline comparison.
Sorry for taking this long -- I wanted to see if the regression is
still reproducible on v6.9.
Apparently we got the similar results on v6.9 with the following
patches cherry-picked cleanly from v6.10-rc1:
1 mm: page_alloc: remove pcppage migratetype caching
2 mm: page_alloc: optimize free_unref_folios()
3 mm: page_alloc: fix up block types when merging compatible blocks
4 mm: page_alloc: move free pages when converting block during isolation
5 mm: page_alloc: fix move_freepages_block() range error
6 mm: page_alloc: fix freelist movement during block conversion
7 mm: page_alloc: close migratetype race between freeing and stealing
8 mm: page_alloc: set migratetype inside move_freepages()
9 mm: page_isolation: prepare for hygienic freelists
10 mm: page_alloc: consolidate free page accounting
11 mm: page_alloc: change move_freepages() to __move_freepages_block()
12 mm: page_alloc: batch vmstat updates in expand()
Unfortunately I just realized that that automated benchmark didn't
collect the kernel stats before it starts (since it always starts on a
freshly booted device). While this is being fixed, I'm attaching the
kernel stats collected after the benchmark finished. I grabbed 10 runs
for each (baseline/patched), and if you need more, please let me know.
(And we should have the stats before the benchmark soon.)
�7zXZ �ִF ! t/��7�B�] 6���Ѳ4��/��f���%� s�� �� =�� p�ah/����1�*~���p���>��\ �kXGfd� kU��&����_(�d7���U��Ϸ)\n��)�����@U,�ĸWw��V�d M5��~���� 2���1f� �3�����5��wiApEձ�`��%���G�� -g{���as���m��ػf����\5��4egU� ާ�= ���X6�j� �WJιA�7�� :n!�0��-�؆N�[� �dQQ�D�w�L���ʦ7�d�u��P��4�sεd�{�MVNǨ <0/I����O�4 �ڀ/�iGU%�������� �0��R��z�=��Aέ]_��`� ��B����h�y�t�ךf��I �7��Σ�K��#�q;dj2�[�� �m��x�UHS� 5ڸȡ �xW�Qn��u�S�q�� b �h �j������Ǻ����X��jj��nlH�� �Zv���mm�YS��X��f������|�Yʖ�. ���,'tl}��c'�L^�=�2!�e��f����4����zS��`�����K�(2F_>
{ +%�t��Sз�Zm���#��0#x��?GN
���XN*��ב�'j�q����ۻw����Q<̒4J�N+�$���L����
��ǩ�%j��ش��� ��������Rx���w�W��a��� ���g�� XKyl�&�v�wZ� 2�8W�K��7�W%���{���O�\:( g����i������ �u�׆R�߇X���l�1�>ِ{b# 4'���:]�FT�@^I ]�F`uT�W�_�th� �*v�9����p����
a�,F���Q߅ j�xg8x']��w�DK�s��ڧw���^Ik"�R�+5�����8��pb��?�G��賤&�^3��cg7 _!�)M���A/*:yE�p�zb�ªQt��2�Y�K�e���o0��\4�� `�������WG���<A
J]j ɠ9��>� �����{e ��[ �]�yۗ�I�`1��ԑm����1���$&Q� � $ ,Úף1�X�o%�b�� �
�&@�sy�KJM �^�Qu�Wy^~���P�vse%�t~�� �+ymz�3&*}���R-��t�u��C�-� (�%��S%煴�LZ�}<�d����hb��W� `:� �TD�����F#o�䣒��QVk}GQg�6�Z���MG����$$�C�|����� ���>��Fd�說��d� � f�ζ.}� c���_/��JهD���X�0Z�ȣW�����M�S%}#�0��������Ɵ}y� ��t��Y?�T�C+$d�Z|���p'i*9NN��T�iZ�Vغ G���Z\3֧C�wW��k���A�¦�W�k�kZ��������Xr|�d Cg�yM[~h���M��SY'�ˁH/C�V�_� ��7�D#�kf C��;��Jw�_�m�1� uv��G%,��)B=Ӌ��(��� �F���7�?��>*���Z��#�a�f���f
V ����l�w��x��)��@/ �ҎVC8 �G�М�q8ܞ��Cx�]X 4b���7G���5s�.�����]��? `C^��3"2��w���p��֛��C�����:��贡&L� ���9"y�� �ۋ䗂 �~g}F\�m�)��Mh:�-��:p�@�L�W8�u#�Mm�.%\��G;���� VS��J�f^� �Y֍���Ӎ㢂��|)c�d�H�:���S�z�|��������ˁ1
���4��|��[�zN�O.�U 0P�q�[��Zh9i���0'�1vR6ͧ��ZC:Ϣ��ܲ�.�g���� �R�Qa1�R��9�O�j;�-q3�QR�4 �"��J]
8B�^��! 1�=B����uH+f� ��&��|���r��� Ւ ^5����n��Ӯ�]vc@��Y�>�(��z�l\��� bv$���n�h��\4��8�v��ؔ�#��wȖڋO���v��Oӿ���T#;46x_$J2l�� ze�8���e�N
��'1�~ێNJ{f�g�n�9���Ekm�� �����ˁ��Xs�RҖu4���Śp�l 8 5 ��Y�R�⒣7Qa�^hҟG����%�
�_�����@�,�V�X�Ӿk���l��Q�q�2]�1�\���W An�0� ,nEXP}� s�3&�����~�����[y}��3b���>m���Z�:1�+�,}R>�{��6[� �_��A��{�u���0� ���]�̰nIf�6R- �h�� ΅'�kj�6�UT������]��Qs��%UE<���n�E��a���'
����}��y��*�`�5��Ú>�a�%/��� �5
���m�)�@d#]�9uQKR0�*�&}�(,
Y�
��ǣ���0p��NS��9�� , )�(i� �����Y蔗�����ݻ����C�<��o�&���4:��{߸0�5���&I�/�0� .�r���Ao�MTt��$��8t3lpQ7��+���P�<�<q��Qsr�}0HP�蟀� ,E������AU>b�7�>�m�電�����%}�����žb���AȟP]��ʒ7�Jë�P ��V�"=Y�g��rL�F�HT�L����9��NⱩŮ��-�]F��/���Qld
$� b����m�(jcty��r0�T��|=���-ѭuS��M���=�Z6D�ڞφdGnE}�ޮ�% Y&ͮ���B�W/tש��l�����=�k=�s,��u����EU��k����j�e�W�I�$��uf�����ľ�
a-QfW�ݠ=�>�u��bY�j�g��'~Eq���K� �c6h�L��W`w/��}U� �e*Y`:`�:}�� -@�������q�w�e�z���.ͥ��=b#:f�m����.��
� ��b� 6���N���j���{��6zW�hh�,2 I;����EzF�JQ6 �\��|�
ٽ��Na������� �a��{�͎��*σFe?�[M���� "A��P��*���Vl�l�� e�1�;ֲ�����4^�/ �$b��g(O2�8(JB����r�F�y$5�0��r��t�.Ƽ�:*����\\ l=�Eb�s畺�IX����N
���.� C�N�t��`�(�Ҥ��l>|N�m�ܖ�)O(�ndA����G=w���,����?���#)"梷�� �c��ԓ�z?� ��5r��@�N����
�Ũ������KR8�bY �Y��.Ԅ/aZB���c㾂W�,A�躰o�p|���mp�����E$�9qs�� ����-t��do~?G㽞3�mw�:cC8�.w����G����~��CVaA�㰣yK{H��~���jVX���\/���RM�8����HӽԨ��Iɮ��$k�A�=ڷQ��fB�ihv����p�ǻ�>���)���{����C�,ޯ��z���ޒ0���l7 G�7���ȗ1�� )Ȑ��Fx1�㽕� �#�v�N��rHe�U�����U�Pg�d�4o���焿�� ����xv<�����w���� ߈��M-�h�k-px���&�P�4 ݔ,��� �W�
&,�09T� �V���K�~��+�6
e�"L�!'ĢT�%� ԶM�dz xA����%�4e蓝�(�\i��e#M^� wC�� NiC����` ��q��`�%?2�q�V�y�4�b���ѯ�� \���W%��<~��N'�.��_0���]C���i�V3��\�Ձ�Eg���L9�^F
�k���FD *n�Jq,_h�e��p�1~�I�Ԗ0�l�# �_����t�' ��Js;��F�T���=�� ڡ��ʇ��
=5��C�&���۰� ��#+n�_K�CP��y�v�"�FQ�k��#�wd��Z�-ɋ��ƒNv�%[�AMy�c����%ś�Ou�6G��d��`v�J٭g�&.�9������ ���@�~'i�r�%K��d3 ��)� �H��J0�oA������.���t� �эc s2O�ݡ W�d�/�n�L"+��Lܭ�"S��U������x��܋�P�(�ݖz�zY���˹̞\����cM��V<Y!oOi�Ҍ�M�E��$�6���,�F_W����lag��D��g���1yuQ�,��������D�v��L�fu\�˟���9if�p�{d�)�8R ��~ۏ;�B����f� ����� ���D.��4���6��@M- 2�\�Pl �
����ofSO\��P^W;')Q�Ne��y���4gN�p=� ��:3�?����'�K�=�A8@I���aG�Y^�,,tH<��o����u���!_B�d�̐`����@��n�6��T����c�͠��PU1�}[��5�B�� �=5�ղ�^���O��l4�G�`h�&'�I�Ɲ+��cdC0�E�K.�����K�
����ז)it� b9����EϮ(S���5����mi��MVΌ{b7I��W���f<� �3��Gu�ܪ����b�N�l�YmK+I�l�
���r��ю��3��9Rk��Cq�\p��!�q�ˑ뼻`��oSb��d�N �<kg��x<$S�Uh0 �j��~��F� �+Q���3*[���i �iȚ>�� �|��ܐ�4���6�)����] � ��Ї��J�wPYP2E�O ;�� ���c�?��Ĥ��Y*��LM>��訵��I�����_VҜk�Ώ�Tz�l��%�^�#]���&�!�d��AC11 ��Eb����n�DWX����n�E�nF"� ]���¬~�7[GW�W߰�bC�H��:Ti�;0����?�� 2¾�?��� �,�I�A����E5j���y�&.?��`C2j����7�s��� } �.-MJ���;!�R�͂:�M^$� I �!U�hҜ���M7�:�!��kT��������&��31��Td�v� �`�,D�0�`.)8�ɧ�)�*a���>0 O�.t�K�]dN���bqв���唙<�d���i�혟N���V��k���gB*��aSy�����K��I�3�oկ���q�a��z�ƦRU� �� C�-%�� �!��6�PE0Q�7 N(Q�,�[��={����� .]����] ���Rҝ��ު�/@ �ϦҞ�|T��(q������~݆���b�)���R�q6��S؟ݡ�+��z�^�5�[=��<��{�r0�s5�EM ��_�P����C��#70Iv�������|0
�U� ��Kq^�:[�o�p��O� n��N��<��� �T,���m�����0����x����UHvl���*��yg�K�����!4��X��i-/�C�w9������ug�킲/��8 M$�?�<���&�HD\��1-X�d+��πG/:S��!�U��S �#����"X�A�76�� lLz�TihIJ� ��x
��6bU_4�
dOl�(;٣O3��V&Ԃ?1ă|�k=r���1c��@�Ug5�H9�8�(ĺ�x�`�G�;Z ���Z�J��4#�ʝ.�F_<P6�*b��<R��B��۪Ͻǥ�bک&M�&v������ )Ş9���g�ܛ�۞r��@�^��;���W�S�/�4=�P��#m�R(1TD(� I�� ���e�# � 5+ �sI佰������d NK�t��n�җ�B\����@��h�C���;Uwë~�Kw�qSW6Z�;�1�b!j����%�#r��X�#O���;�h R��/J[�?���h�����Ѧ����l0�r
*6<�"E�s���b�0�(�b�ױ�J�=�9��h�����\�/Z�z���v�^*�m*���[ �H�$��t8�xB4uϤ�4����.�,�-6.ӣ@��-e�
%]�2���u6..ig��[� �e)�M��O���{���:/���8�D�n�L���Y��g�}�3 ��#��e����JoR f����]����Ҁ�N�䮬�
G�X�~WGb5%j uG��gX�\w�V9i^ �� �^\�Am������w��Fb�ћ��{��j�,C��J�a��}�WM{ޞ�z?X�4����U�g�1�T�~�
6V'�(�!�빘
���H��BV���\��� -�` �z���G�y�OxĻ6א��T���hV3��P�A#�������
\uf���Fۦc�6R[z~ҡ j�Biz�e��Ё�I������"���s�Ria}a�ޝ真����;��Z�}�)��6DHr-�5��� aI�t�� � ]��:.��G�����%���9��ghg5yM��P�^�H� ��\�R�h
�1�W�:���_qG] ��7��r�U6pa��,.�18p.�������M�+��/�`��TvUv�俟'���7
U5�={o���pF�����v�� �h�>2{��!. ��&�KV04NyA���]6�@��d&7a% �:�Oq��G���6����R"�v�6S �5��M��sA�q��s! rh9�#�E�O�� XU'E�楱���ď�=��T�d�����zER+ ���1�&���~�h�� 9U�^,ppu�����ޚ�rS���Q��퍤�������d���
������-��abI��>Y��JR�pm�ҋz��@ѓ%ӎ��� �D� �oS�Tfa[֪�[] x�(H���u��-i�� &��;yȌ?#�~�9x�"��=pa�BXIc�$�����0�䖕�<�~��
?��ҶJ�pf������+�N}\�r�����eD�ڼ}zr����G���v[�d��i��GI �9I�巅q�3���5��'�ĕU�")��`��$�)����.�ׄ�J�E��͞��gʗLv`���� s��^,H�`�����Q�~x{ݑ'�^�<��ӡ�+�[�{ �d�u��;1��fLk��-�����p\� ��5����S�m�x�|�-��s����>�[f��&Q��E]E���D��0)b����y��.�#�Y⢊E����5t
�x�*AV�=����G%�_/T.�J,![=�n��SF#d
�d�/��А� PE�B�*����C8kL�� ']u����EKR�ܩ�/��t�뇕��Px5��+��X�W�]ۗ=�KE|���l�s~���=���W��� �l+&��/�H���0k�+U8�F&�����pU��7,Ü��o}�C��}SV���� ��$�.��c��+oI�d�ËD V�xx�����ȴ6�OW�NU,��o�e�\�V��� r�����bm Jd c��<�8�?�IU�vn� ��E��_7jnB�m�;i �<@7�Mk$�U{���у�kB�+eCo������ �X�8P�?':Ӊ6 4/�b��-*Z%Wܭ��g���'@�Q$�o!��$����,|��n.���)_���p����QWM��N�.�Vk�v�&���k�4y�3A��S� U_�"�ɾ�x�fe�@x�L��q�n��S�D��G=g���v�x��vdg�� Lh�6��ӷ3 �<p��]��w5 � N9�-�؍lF*Cp ���J���2�Q+P�j/�Dv3�PeK��?�i��*rF��H^8��2�;~n T���N����MP& ȃR���������ӝ ���n`��TՍ�u�
v2�t�g]�0��#��_M'o)+! �����.�g��*������њ�7h��4s�Pq/8�ے8�qnoI\ �"WL>}�M�pXc@�!�꒮'zZ��=7 y��u9�c F�q\X��6����!g5��f.� �o ��q2�n,��_�Rb~5Zk��Ll��/4�i�lu�!� Q�!��n������g�.[
�) MǴVCk��]���Ň�
��D�R���[���۪,H6�B wh�"'�;0�����C���m^�݀�ĨoT�M�81�,�O��_�{��@ ���x�u����m� ��t[s ��=:~Tl�<<q]X�8�@5��䰯G eI��s�Hs+��S[��;S�f��n h{���=a�`�l�e��-���?�Ӥ�C� _9iI�>5yܺ��i�5��з��S7���z��k�p���>��Xk} �⌞?��ԽP�j�'� [�e��IY�Uh�;�j Ȱ�:EHZ���4�m�_<L|@Ք
Yj��H�_E�v��4�����<�H��j��� q� V��V����.��yKY@�b�9O]�Ԋܼsw �Pq��x�
�()���l�k�����*��GfӺR�c��>l�>��� �������A���I�盝.�c��ܨ]�rЋx���q�+� xV<� d���L�'wp��j?��/]kh���*Ǿ;�m�����͐�6hJ�����F��ņ_+&Y�͞~� =��B���/� _�F�U�S� �@�m�"Y���� �`�F��D�g
X{�c`99s K��3��=�����T ۴.�,� 騩�H�9>a����,EI��7%z���x�)#�P ~q�����nA�BwN'��d�" D�����W6����!����z+�)���[�����$��@��-�������Ptw ћ[ڲ\���2U_�r1䮇I�\���x�:�����g..�� ���*��У��)M���A�Gи�!�� !>����IyT_��6+f-I�� X�/�G��j��Z �\����� U��S�NȂ��M�0�w�/[#q�h�����І�����*j&珸��U�}��p�w� �~)
.ۤH�� ���� l�W��i�tuk�xV��I�����̮Uz:R�
!���~'�����%�}Fb�Q>n��8�-�nfƔ �r���D��y��panp�d��\����L7R��9�Md�f�m[;�#�Kj�$&f�{�*� ���Bu���{(�z�7�� D]�x��fM���˛t�^ (.��#� 4�d�&�I�z�PQ|� ���C�=A�K���� |Ѻ i��� � 4Xn
*er!��#L<{��%�6F�B��M �Y�( �>^kK=��1E�V�b�{�Ḛ4� )-�Jۍ����9��c�-��2����Ng**�@�S���3) W�Vhۿ��l��t�,ib �Ȩ����)=�Xˌ���^C���;��A n�y�wՂ�ء)�|˩��>���eo ��"[�A1]f�4 'Ku� t�S�
��0�)����yH�%������00ǧ�o/�X����Md����hI�c��Y3H��{��JK�Ġi�D%%#Ŵh��9��d�ڋ𝦅u��SA��]K����/�\2�/Q�Gs�+�:Cx��]/�=b�M(�b5�n _��c; �;���r�;�7@�)i��>���@h����dY��V1n$�����ΠF��UC1��*+�0�=�S�^3���#��
?� 7�.fx�ŵ�̇��q���N!rH� 9*�& ��J�n �1���R���;v�)�*�w���vW�S� �6������ ���\��W ��� ���HQ��Ԙ���
;�c�diQ�D �m,�o_E���������� yX+��8y�q:s6M ڽLGsˁ��6&�/,�V���f ��<��Ig��SB�2:���� 8h�� �O�"�"}�h{��F�zˑ�? � G0E�=�}h9��v��].��/Lh�۱�h�ۥ���&���@&��h��ք3ML��p�8�1�)���]���u$a��
`�i�g4�%���y-�5�*����]�Ʋ���[�& i�כ�����k0#�%�N�8sn� ZH�Ƅ�_}A�j �)��TrͶ�W��̒mkxY^YΞh(��aKUc���M��V����)�
E����ZsLJ�Yl�Rr�YDPڒ��k���
ث~�n�w��2��\M9/@��Ȧw�:�����]�� 0��|����k��5A|J���pĀ�����Ai��nd�S��?s�r��I���� '�)�l[���c>d$����t|�����d6�J��D[�z��B^�zwb%
r��/�r��m��Y%Q��5�:�m��,�� �Djk�
��
�ɩ,wc) �r�A�N�#�����ӛ��vܜW��#h��NA��.m��=�x�_ �Y��Hʥ���"�E��.���]!�XW�]��(C�%
|�fM���X��l�u�5�ΐ8A ,��#i\ �E"_x�嬾�jʠǷ������'��y��J�s" ^ށ�CY���Q�����b�:Q�nbJP/Φ$��g�p\�NI �tD�7�(�&L;~���{�3"W��*8^�^r'��wH��i��3 �O�� ��a�!~D�축�"��Ϙ@����qrt�B�?����D��O0�=d�b;�<��er �m�`o����- �5%)W�j��+duD?��c�]�K��\p���2� Ooi7�S!�s�d�t`� � �i����^x�k%��`�m���o=U<,5D���c��-1��B�H:�g}�ꉦ�0�q k��:BP�m��#pHa�G(�t1A��raf�<}\=Lq�')'�U뎧 _>? �+�A����3<�wv�y�^�q�~m��O�m���,y��F`q��.�$��K�XS���ߵ\Je�Z8�8��_���`tXƬ��Z���<��0U�m��"�{@��ۙ�埼���*��O4�{��{��P7���ʓS�����a��� ���[�J>=���^Q�]dG�^L��i)x�k:}��h���;���H��|�#���Q�OA�a���-��Y� �l�g ���"�wk.����t�r. QX�"r�������),V���%y_+����'`��`��G ���QE���!�pV�)NPiaZ�����įŖ� s�����.<��2�* C0�s*?�s4«T} 4m*�n�!A��Zb@��$zV���Q�_y��vƽ6� U!��g�T���7�ׅ�������I]� iX"��QZۣa�Y7��l���¬�i�F1�j�L;��13S`{�9��q(""=[� � � �/ȣ����{�`�+Aͱ�=�1�g�
%���M���T0�}�2�U1=�x�:2|�j���Mru|WJ�g�=���5j ���% �vru� �.��
���ʲRP�����Dڲ'$�<8�<�&nE��W�S�J^E�r`� �㝎���oF���z+�-u�D.T}�w�� ��x��M�35�P��@Y��. �1����ա���ws-=N��xГY���])�x��3�� �dj���_vɠ;�z �� B9M��68�^QR.G���KR.��c�C6�y������fr �,$$�dޛ�9'~���d�:*� �J,� ��M�ǘ0aB�Qr٦�����c��3��{m���� �_�nh،�tX� ��%nn��8ĕ��B�9� �~?��5ʟw�!K�� �Yb���k\=����^ �xDT,K��:>�)k�)��o�5��X�P-y��'��\�$��i%�d:�� y@����# �-�-/ov���f�Z���@���� .�� '� XD�85�� �?��b2|4����F(M�D��������~�x]�}�ؙ�M �c4N�����䈕X2!����ӵ�gSk�wb2<�{��� ����KRB ��栏\:��#��ZE��ϺQ��Y�v�[ډ%�$��*KN�����KtYP@�v�i߆c$��,��P^RH���ރ]p��{K}�+�ܑr�X�m�����$ YZ¹�Y ���{�8���݄�+��
�y' 3��
Y����ј�
:�"��
��_8�HS�l��H� �n���B�E> �Z1Kz�`�%��R>��]�?�mSJ_���K�����ذg���(���,�(v$5����������Ak�ɫf��1! *�yK_���O��P jXކ��&�7浴�K"�K�߃��υ(��&��+p9�9�m-@��]�$$F�{�:��ռ�H ߎ �E
����}��$A�a����d,Y�m�Z-� P-S������g7o�5s����@�7��� ࠕM���M� O��;;�t���@��B�G� c��/����?\і9�� �l��@*P����>�l��(c��? жה&:�{��5K�$X��F�4�Q>��Vud�
��{�)N,F���rY}�N��6���1^l�{U�^���� 'jS9D�=8��� 4֏�D�6��&gP�� TSji��gr�b }d,�[�
�&`���V��J�I6�x�5��t��;�)R.��Bݸ�/qk��@�M��vM��Ζ(e
ƭT̹�9��D2i_�{�:u��Εs<�x������]�q05��
�Fv9Y="��B�� bK����X$V�# ��rw/"���r��Ӻ��#+F����Н�Mg��'�����]�l��
$����ߌ.���و�P��Yp�j�#ޛ)E�@/j3�k��I�a{���}�31
V�iO��?��dݱG�mL������!H�jm�hOF�Y�W�� i/CQ�(���ShF�SH
�|x�Ç �k8b��g &��ks�2�k��3s=�C��q��¬/]b��m��N]��"��@9�V;���=9��ˍ����P��m\�)���L��Y�\'���?���n!����Ọ0�R��_^E:���Q]M�9��Q�7Z��>i�b�xЯ�`�W��ь�����C3��S�h0���i=\b ���P�{M�k>��hc�Ŏk �i(����I3�
?���ϕFޞ��S�P
� �$!0ik�*Z��r �>�#~dP�~��\���wxO!�\}p}� ����x�,xv�K����ܲ��_r�>T֝ � (� G\��
_ʹ��[LB�V�j�� ST�"`��j��k?Mi>U������]NtU��� Kf�j� �� ���Q�n��I=�z�vw�A�\2��T� � a���C`�م o)�f��E��\ �?�
Y/M�� w��ZE�\ݍ�uy֭ �z��{�f@�U�d��1��O�R�s� �=C�J\� i�%î�J�Y�7��~;��z��Cg�&�}�HS����2ml_�7�#�)�T@^�0��f�XO�)>/J�,.$^��
�4wr�c��LB����_̶F.gG���a�! ){��J.̒06(F ~JI
�j�/���?]��p�M��–�/����u-j�v���v��),�j�6�~L�:�YYl���]� ���B 6����z��% *fZ}x��B�{��3�0瀎����:�a4�ɗ�7F�V�p�����)��1]g��&7��kk�SQS�4�/h���\=��h%���� �UH�V�����z�ǠDG���7���$
�]�0������0}\U�:.�+� ��D&���uC]"[�CTX2|_���42�mƤ�^��p=�_Y+��m�X��s7h+��5#�8ky��D=��ӬP �ގ�� �
��N��R��-S�e�k�,�3�3
n�� 5����9�=۞��b`�)E��jg$�-�n���-��m �)�&� h�
o���[4�ɆfkM� hC���)\����5�g@J�$�(E a
3:��B�&L��,�FGO�u��Ew�#�>cJ�OȎ� ��9P?"46;���Nb���%���CI^k�v�8L[��ʣr��-�z�i鏙ũ�b����!rh~s���Qwn�����$�.ʳ�y�z��}m�y�O�ꛦ�L25~�Tqv{I�m'(j[���$����L۬�f�_�0|Ђ0��Z5@%�^�]G����Qܑ $'#T� =�� �6C���03������b}j�Bygb �vw&��2�����=Lr��oH�mc�{���z�OU3q�$� �n����xs �Ѩ�ؤt\��U�.j���y/Ԋm��B�c��͇dVnxP�͚���L9�n��@�7+;،c>�bL��ql"��XE�j�O|�.r`i~� |��q9W7vd�Ĭ
���_§�g4I`ϵ뭰8k���4�,5�9���ᒰZۈV?������%�E�k0���H�7 ���e�/K���v}���{2�>5o'�\�D��0��%�d�fyj�/LgӰ*r_N�.81}�Y���%��ɲs+_�k�l�5DJ yZm�戔UN�^ ���bMIٔ` �Ϥ�!��n��j� [֗��{�����+ +�� C`9+2�-���g4S7*Ї W%���'�� �1� ������B�W�]�D��9�9NϚ� �����{������%
�U�n���~=�y ߃�P~�#�El(S�DKBS}�.�w�9��p&��-��P�Jb�c��_`^uCUy��mUE��*x��j|�@M��� r��
fΎ�p `i�Յ���b|a�8��!�6��i��K�#�8�Zn�H�ѯf�PU�z4d�.�
� _"0fOT�{m���������n%A�%Q�_ \�D�'9�rgEV
H���� �R���#~z�������4��� ۡ�] ��iKf�u�O���1
�8���6%2]v���o��I
�E��"�zԃ��j�l�yEM����Z�מ %)�հd/V
��@5�ǃG����z&�x�z=��'��e �3S�%�Or3+p�|��hL���i��,��N,>���[` 9\6���� F�et���m�I��st�����h��l�6��w
��ҫz�|�G����}3ӻ�+�ghzW��
�V�rxN��T^,����7M�����>
^��P*:m1y\.�棽*C�D�YϪiJ�g� M�u� � �븊sč�њKڧ�W�H'��C�����Dژ
s�H�S]ֵ�:%�7 ��w�����v
�UE 2�
�i�e������1O�� �B'�T������R3� p���u��y�=�E?z�և�t�a� $�~D�
��)�˅:IgA��'�$�3��L*��d��pZ�1��M�(�zY��c*�v\��I�m�j-��kH� �^�(��y��- � Ur�((�B��M��ݏL��(�-@ �(0�
�$������tn�_�dܓEV�c�\�I��=�K��=LV~��ء�&?"\Aȶ��Kr��Gh
�EUc0y_��� =�7c�#����Di��ބ��mޓ��>IFˀb,�Դ��� �����I� �����@m�K��0`z/�w:q��p��Lw��X�R��D5Ȱ��=k����2�D�<�T �:� ��X%�K� ��Dh,jr����mb�q�m�x�hW�N���)\�'�I [�`� ����l���櫹�4�(�q�p>���
)i 0�Q���3�1 �*�� őC���k��=OkpyaA�����Lt�#v<G��t��q{�; �R�J�t��StUg����ѯ j��SZ�� �~"�Ӧn���W� �X��R <������M���h����͝ݫ`��N(��k����$����$��b� �A �\ �۵�4w��'��LEb��F��䡚����%��
�����3�;(��ERd�f�n�/�7��qA��
Đ�5�LI {���Z��y�Q-��S_� �����k3.����[W��̪(��+�y�?�`�~��ۄf)z�E� T[�!���>�:῎� 1JX^
,7���Rw�i���N�z��Fdj�F��*�%�P�
���M�o �D�� ��%� �.�V�z��� �!#�
���/A���2\#%���Q�.S��Z)W��'R
�M�����8��:4T +�� �%n����g�`^��q���ߞ�9�PƗ�'���K�N�O-%��_d\���f�Qz"��� u5��w� ^�nc��-����,���{b~� ��k�m����J!BN���p�+���H�l��06G���[d�u�I�pZ}�)��έ�t!߮
�����?5s.:�*�"$/���YoR����c��
A��h _�du8���IX��
Y��
T���x8%['��tAsgn�u�${G���[ݤl�/� p �P2��*qp��#�]�E{�0� �w%��,0k �(V��A����-f��阳�s�����Y�̉�x�v��%���s���S��{IE7� ���J6l=��v ��, |�;��Ud��?[}�5W��JI�w>� ��ſb�܇�%�z˳�U��P���(K�}��x90YC=�JS
���. k�6������[�a��P ��B�t Cp�:�Q�H��P'�J�Ȑ�>T� m~� �r���� �-i) E�mIBf�j��T����1z��Ù� ��&� =u1=+��UP��K%-t�z�x��ҕ�e���l�- ���~�� rO�/e�
���� �� �F*j��C���*6C�G^�͑8�u�@&:%�%�a�}ĝ�0�/� 2=�5A�._2�z�v��a���[Rg�%z�go,IgN"0_�����.+�4oe�ab��քǭ�aT $ʝ&Qx��W:6 _ut����R� �3��uB>�9w��d���ӆ��sT|��2Z�b�-CY�s���(mZ�0��,|�'��jA�= ���,f�',-�f�����j���ygz�O�`?|�3
��?�l���T�xYE="�M֚'hs��ʦ�!��5���H9#/����
ճ�l��Hp C�:��6א.+?M���hlQ��!�N�Z^��u��{�
�����.�B�� G}a%l�e��055M|[�s/4�(Xv��EJU� !�~:d٤�; �{��-G!l�t`� ����&�[�yʸ6kֳ�F��?3~�D@�I�T_xz'D��AyR F���e;};1�ru�� �q�INLBY���_�UP�J{S�G�։�,T� �<�i>D��� b �k٬?N�� b�_�����;�PWU"��DR
�ǥ� �D T1����6k� <�ս{��� iE"���1�o��Y��Q���Fڷ��"�8�r�K$Y/���E_^���W"���7�٩�|�:[ �%W`�|X�`���' �;�o���C*��?g�
W��쫆������8nb��F���4,2�`����2eD��v
Z)���yI������M��8qQ�����)�� �?��� �����)���g� YZ
On Tue, Jun 04, 2024 at 10:53:55PM -0600, Yu Zhao wrote: > On Mon, May 13, 2024 at 1:04 PM Johannes Weiner <hannes@cmpxchg.org> wrote: > > On Mon, May 13, 2024 at 12:10:04PM -0600, Yu Zhao wrote: > > > On Mon, May 13, 2024 at 10:03 AM Johannes Weiner <hannes@cmpxchg.org> wrote: > > > > On Fri, May 10, 2024 at 11:14:43PM -0600, Yu Zhao wrote: > > > > > This series significantly regresses Android and ChromeOS under memory > > > > > pressure. THPs are virtually nonexistent on client devices, and IIRC, > > > > > it was mentioned in the early discussions that potential regressions > > > > > for such a case are somewhat expected? > > > > > > > > This is not expected for the 10 patches here. You might be referring > > > > to the discussion around the huge page allocator series, which had > > > > fallback restrictions and many changes to reclaim and compaction. > > > > > > Right, now I remember. > > > > > > > > On Android (ARMv8.2), app launch time regressed by about 7%; On > > > > > ChromeOS (Intel ADL), tab switch time regressed by about 8%. Also PSI > > > > > (full and some) on both platforms increased by over 20%. I could post > > > > > the details of the benchmarks and the metrics they measure, but I > > > > > doubt they would mean much to you. I did ask our test teams to save > > > > > extra kernel logs that might be more helpful, and I could forward them > > > > > to you. > > > > > > > > If the issue persists with the latest patches in -mm, a kernel config > > > > and snapshots of /proc/vmstat, /proc/pagetypeinfo, /proc/zoneinfo > > > > before/during/after the problematic behavior would be very helpful. > > > > > > Assuming all the fixes were included, do you want the logs from 6.8? > > > We have them available now. > > > > Yes, that would be helpful. > > > > If you have them, it would also be quite useful to have the vmstat > > before-after-test delta from a good kernel, for baseline comparison. > > Sorry for taking this long -- I wanted to see if the regression is > still reproducible on v6.9. > > Apparently we got the similar results on v6.9 with the following > patches cherry-picked cleanly from v6.10-rc1: > > 1 mm: page_alloc: remove pcppage migratetype caching > 2 mm: page_alloc: optimize free_unref_folios() > 3 mm: page_alloc: fix up block types when merging compatible blocks > 4 mm: page_alloc: move free pages when converting block during isolation > 5 mm: page_alloc: fix move_freepages_block() range error > 6 mm: page_alloc: fix freelist movement during block conversion > 7 mm: page_alloc: close migratetype race between freeing and stealing > 8 mm: page_alloc: set migratetype inside move_freepages() > 9 mm: page_isolation: prepare for hygienic freelists > 10 mm: page_alloc: consolidate free page accounting > 11 mm: page_alloc: change move_freepages() to __move_freepages_block() > 12 mm: page_alloc: batch vmstat updates in expand() > > Unfortunately I just realized that that automated benchmark didn't > collect the kernel stats before it starts (since it always starts on a > freshly booted device). While this is being fixed, I'm attaching the > kernel stats collected after the benchmark finished. I grabbed 10 runs > for each (baseline/patched), and if you need more, please let me know. > (And we should have the stats before the benchmark soon.) Thanks for grabbing these, and sorry about the delay, I was traveling last week. You mentioned "THPs are virtually non-existant". But the workload doesn't seem to allocate anon THPs at all. For file THP, the patched kernel's median for allocation success is 90% of baseline, but the inter-run min/max deviation from the median in baseline is 85%/108% and in patched and 85%/112% in patched, so this is quite noisy. Was that initial comment regarding a different workload? This other data point has me stumped. Comparing medians, there is a 1.5% reduction in anon refaults and a 4.8% increase in file refaults. And indeed, there is more files and less anon being scanned. I think this could explain the PSI delta, since AFAIK you have zram or zswap, and anon decompression loads are cheaper than filesystem IO. The above patches don't do anything that directly influences the anon-file reclaim balance. However, if file THPs fall back to 4k file pages more, that *might* be able to explain a shift in reclaim balance, if some hot subpages in those THPs were protecting colder subpages from being reclaimed and refaulting. In that case, the root cause would still be a simple THP success rate regression. To confirm this theory, could you run the baseline and the patched sets both with THP disabled entirely? Can you elaborate more on what the workload is doing exactly? What are the parameters of the test machine (CPUs, memory size)? It'd be helpful if I could reproduce this locally as well.
On Mon, Jun 10, 2024 at 9:28 AM Johannes Weiner <hannes@cmpxchg.org> wrote: > > On Tue, Jun 04, 2024 at 10:53:55PM -0600, Yu Zhao wrote: > > On Mon, May 13, 2024 at 1:04 PM Johannes Weiner <hannes@cmpxchg.org> wrote: > > > On Mon, May 13, 2024 at 12:10:04PM -0600, Yu Zhao wrote: > > > > On Mon, May 13, 2024 at 10:03 AM Johannes Weiner <hannes@cmpxchg.org> wrote: > > > > > On Fri, May 10, 2024 at 11:14:43PM -0600, Yu Zhao wrote: > > > > > > This series significantly regresses Android and ChromeOS under memory > > > > > > pressure. THPs are virtually nonexistent on client devices, and IIRC, > > > > > > it was mentioned in the early discussions that potential regressions > > > > > > for such a case are somewhat expected? > > > > > > > > > > This is not expected for the 10 patches here. You might be referring > > > > > to the discussion around the huge page allocator series, which had > > > > > fallback restrictions and many changes to reclaim and compaction. > > > > > > > > Right, now I remember. > > > > > > > > > > On Android (ARMv8.2), app launch time regressed by about 7%; On > > > > > > ChromeOS (Intel ADL), tab switch time regressed by about 8%. Also PSI > > > > > > (full and some) on both platforms increased by over 20%. I could post > > > > > > the details of the benchmarks and the metrics they measure, but I > > > > > > doubt they would mean much to you. I did ask our test teams to save > > > > > > extra kernel logs that might be more helpful, and I could forward them > > > > > > to you. > > > > > > > > > > If the issue persists with the latest patches in -mm, a kernel config > > > > > and snapshots of /proc/vmstat, /proc/pagetypeinfo, /proc/zoneinfo > > > > > before/during/after the problematic behavior would be very helpful. > > > > > > > > Assuming all the fixes were included, do you want the logs from 6.8? > > > > We have them available now. > > > > > > Yes, that would be helpful. > > > > > > If you have them, it would also be quite useful to have the vmstat > > > before-after-test delta from a good kernel, for baseline comparison. > > > > Sorry for taking this long -- I wanted to see if the regression is > > still reproducible on v6.9. > > > > Apparently we got the similar results on v6.9 with the following > > patches cherry-picked cleanly from v6.10-rc1: > > > > 1 mm: page_alloc: remove pcppage migratetype caching > > 2 mm: page_alloc: optimize free_unref_folios() > > 3 mm: page_alloc: fix up block types when merging compatible blocks > > 4 mm: page_alloc: move free pages when converting block during isolation > > 5 mm: page_alloc: fix move_freepages_block() range error > > 6 mm: page_alloc: fix freelist movement during block conversion > > 7 mm: page_alloc: close migratetype race between freeing and stealing > > 8 mm: page_alloc: set migratetype inside move_freepages() > > 9 mm: page_isolation: prepare for hygienic freelists > > 10 mm: page_alloc: consolidate free page accounting > > 11 mm: page_alloc: change move_freepages() to __move_freepages_block() > > 12 mm: page_alloc: batch vmstat updates in expand() > > > > Unfortunately I just realized that that automated benchmark didn't > > collect the kernel stats before it starts (since it always starts on a > > freshly booted device). While this is being fixed, I'm attaching the > > kernel stats collected after the benchmark finished. I grabbed 10 runs > > for each (baseline/patched), and if you need more, please let me know. > > (And we should have the stats before the benchmark soon.) > > Thanks for grabbing these, and sorry about the delay, I was traveling > last week. > > You mentioned "THPs are virtually non-existant". But the workload > doesn't seem to allocate anon THPs at all. Sorry for not being clear there: you are correct. I meant that client devices rarely use 2MB THPs or __GFP_COMP. (They simply can't due to both internal and external fragmentations, but we are trying!) > For file THP, the patched > kernel's median for allocation success is 90% of baseline, but the > inter-run min/max deviation from the median in baseline is 85%/108% > and in patched and 85%/112% in patched, so this is quite noisy. Was > that initial comment regarding a different workload? No, in both cases (Android and ChromeOS) we tried, we were hoping the series could help with mTHP (64KB and 32KB). But we hit the regressions with their baseline (4KB). Again, 2MB THPs, if they are used, are reserved (allocated and mlocked to hold text/code sections after a reboot). So they shouldn't matter, and I highly doubt the regressions are because of them. > This other data point has me stumped. Comparing medians, there is a > 1.5% reduction in anon refaults and a 4.8% increase in file > refaults. And indeed, there is more files and less anon being scanned. > I think this could explain the PSI delta, since AFAIK you have zram or > zswap, and anon decompression loads are cheaper than filesystem IO. > > The above patches don't do anything that directly influences the > anon-file reclaim balance. However, if file THPs fall back to 4k file > pages more, that *might* be able to explain a shift in reclaim > balance, if some hot subpages in those THPs were protecting colder > subpages from being reclaimed and refaulting. > > In that case, the root cause would still be a simple THP success rate > regression. To confirm this theory, could you run the baseline and the > patched sets both with THP disabled entirely? Will try this. And is bisecting within this series possible? > Can you elaborate more on what the workload is doing exactly? These are simple benchmarks that measure the system and foreground app/tab performance under memory pressure, e.g., [1]. They open a bunch of apps/tabs (respectively on Android/ChromeOS) and switch between them. At a given time, one of them is foreground and the rest are background, obviously. When an app/tab has been in the background for a while, the userspace may call madvise(PAGEOUT) to reclaim (most of) its LRU pages, leaving unmovable kernel memory there. This strategy allows client systems to cache more apps/tabs in the background and reduce their startup/switch time. But it's also a major source of fragmentation (I'm sure you get why so I won't go into details here. And userspace also tries to make a better decision between reclaim/compact/kill based on fragmentation, but it's not easy.) [1] https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/refs/heads/main/src/go.chromium.org/tast-tests/cros/local/bundles/cros/platform/memory_pressure.go > What are > the parameters of the test machine (CPUs, memory size)? It'd be > helpful if I could reproduce this locally as well. The data I shared previously is from an Intel i7-1255U + 4GB Chromebook. More data attached -- it contains vmstat, zoneinfo and pagetypeinfo files collected before the benchmark (after fresh reboots) and after the benchmark.
On Wed, Jun 12, 2024 at 12:52:20PM -0600, Yu Zhao wrote: > On Mon, Jun 10, 2024 at 9:28 AM Johannes Weiner <hannes@cmpxchg.org> wrote: > > > > On Tue, Jun 04, 2024 at 10:53:55PM -0600, Yu Zhao wrote: > > > On Mon, May 13, 2024 at 1:04 PM Johannes Weiner <hannes@cmpxchg.org> wrote: > > > > On Mon, May 13, 2024 at 12:10:04PM -0600, Yu Zhao wrote: > > > > > On Mon, May 13, 2024 at 10:03 AM Johannes Weiner <hannes@cmpxchg.org> wrote: > > > > > > On Fri, May 10, 2024 at 11:14:43PM -0600, Yu Zhao wrote: > > > > > > > This series significantly regresses Android and ChromeOS under memory > > > > > > > pressure. THPs are virtually nonexistent on client devices, and IIRC, > > > > > > > it was mentioned in the early discussions that potential regressions > > > > > > > for such a case are somewhat expected? > > > > > > > > > > > > This is not expected for the 10 patches here. You might be referring > > > > > > to the discussion around the huge page allocator series, which had > > > > > > fallback restrictions and many changes to reclaim and compaction. > > > > > > > > > > Right, now I remember. > > > > > > > > > > > > On Android (ARMv8.2), app launch time regressed by about 7%; On > > > > > > > ChromeOS (Intel ADL), tab switch time regressed by about 8%. Also PSI > > > > > > > (full and some) on both platforms increased by over 20%. I could post > > > > > > > the details of the benchmarks and the metrics they measure, but I > > > > > > > doubt they would mean much to you. I did ask our test teams to save > > > > > > > extra kernel logs that might be more helpful, and I could forward them > > > > > > > to you. > > > > > > > > > > > > If the issue persists with the latest patches in -mm, a kernel config > > > > > > and snapshots of /proc/vmstat, /proc/pagetypeinfo, /proc/zoneinfo > > > > > > before/during/after the problematic behavior would be very helpful. > > > > > > > > > > Assuming all the fixes were included, do you want the logs from 6.8? > > > > > We have them available now. > > > > > > > > Yes, that would be helpful. > > > > > > > > If you have them, it would also be quite useful to have the vmstat > > > > before-after-test delta from a good kernel, for baseline comparison. > > > > > > Sorry for taking this long -- I wanted to see if the regression is > > > still reproducible on v6.9. > > > > > > Apparently we got the similar results on v6.9 with the following > > > patches cherry-picked cleanly from v6.10-rc1: > > > > > > 1 mm: page_alloc: remove pcppage migratetype caching > > > 2 mm: page_alloc: optimize free_unref_folios() > > > 3 mm: page_alloc: fix up block types when merging compatible blocks > > > 4 mm: page_alloc: move free pages when converting block during isolation > > > 5 mm: page_alloc: fix move_freepages_block() range error > > > 6 mm: page_alloc: fix freelist movement during block conversion > > > 7 mm: page_alloc: close migratetype race between freeing and stealing > > > 8 mm: page_alloc: set migratetype inside move_freepages() > > > 9 mm: page_isolation: prepare for hygienic freelists > > > 10 mm: page_alloc: consolidate free page accounting > > > 11 mm: page_alloc: change move_freepages() to __move_freepages_block() > > > 12 mm: page_alloc: batch vmstat updates in expand() > > > > > > Unfortunately I just realized that that automated benchmark didn't > > > collect the kernel stats before it starts (since it always starts on a > > > freshly booted device). While this is being fixed, I'm attaching the > > > kernel stats collected after the benchmark finished. I grabbed 10 runs > > > for each (baseline/patched), and if you need more, please let me know. > > > (And we should have the stats before the benchmark soon.) > > > > Thanks for grabbing these, and sorry about the delay, I was traveling > > last week. > > > > You mentioned "THPs are virtually non-existant". But the workload > > doesn't seem to allocate anon THPs at all. > > Sorry for not being clear there: you are correct. > > I meant that client devices rarely use 2MB THPs or __GFP_COMP. (They > simply can't due to both internal and external fragmentations, but we > are trying!) Ah, understood. So this is nominally a non-THP workload, and we're suspecting a simple 4k allocation issue in low memory conditions. Thanks for clarifying. However, I don't think 4k alone would explain pressure just yet. PSI is triggered by reclaim and compaction, but with this series type fallbacks are still allowed to the full extent before entering any such remediation. The series merely fixes type safety and eliminates avoidable/accidental mixing. So I'm thinking something else must still be going on. Either THP (however limited the use in this workload); or the userspace feedback mechanism you mention below... > > For file THP, the patched > > kernel's median for allocation success is 90% of baseline, but the > > inter-run min/max deviation from the median in baseline is 85%/108% > > and in patched and 85%/112% in patched, so this is quite noisy. Was > > that initial comment regarding a different workload? > > No, in both cases (Android and ChromeOS) we tried, we were hoping the > series could help with mTHP (64KB and 32KB). But we hit the > regressions with their baseline (4KB). Again, 2MB THPs, if they are > used, are reserved (allocated and mlocked to hold text/code sections > after a reboot). So they shouldn't matter, and I highly doubt the > regressions are because of them. Ok. > > This other data point has me stumped. Comparing medians, there is a > > 1.5% reduction in anon refaults and a 4.8% increase in file > > refaults. And indeed, there is more files and less anon being scanned. > > I think this could explain the PSI delta, since AFAIK you have zram or > > zswap, and anon decompression loads are cheaper than filesystem IO. > > > > The above patches don't do anything that directly influences the > > anon-file reclaim balance. However, if file THPs fall back to 4k file > > pages more, that *might* be able to explain a shift in reclaim > > balance, if some hot subpages in those THPs were protecting colder > > subpages from being reclaimed and refaulting. > > > > In that case, the root cause would still be a simple THP success rate > > regression. To confirm this theory, could you run the baseline and the > > patched sets both with THP disabled entirely? > > Will try this. And is bisecting within this series possible? Yes. I built and put each commit incrementally through my test machinery before sending them out. I can't vouch for all configurations, of course, but I'd expect it to work. > > Can you elaborate more on what the workload is doing exactly? > > These are simple benchmarks that measure the system and foreground > app/tab performance under memory pressure, e.g., [1]. They open a > bunch of apps/tabs (respectively on Android/ChromeOS) and switch > between them. At a given time, one of them is foreground and the rest > are background, obviously. When an app/tab has been in the background > for a while, the userspace may call madvise(PAGEOUT) to reclaim (most > of) its LRU pages, leaving unmovable kernel memory there. This > strategy allows client systems to cache more apps/tabs in the > background and reduce their startup/switch time. But it's also a major > source of fragmentation (I'm sure you get why so I won't go into > details here. And userspace also tries to make a better decision > between reclaim/compact/kill based on fragmentation, but it's not > easy.) Thanks for the detailed explanation. That last bit is interesting: how does it determine "fragmentation"? The series might well affect this metric. > [1] https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/refs/heads/main/src/go.chromium.org/tast-tests/cros/local/bundles/cros/platform/memory_pressure.go > > > What are > > the parameters of the test machine (CPUs, memory size)? It'd be > > helpful if I could reproduce this locally as well. > > The data I shared previously is from an Intel i7-1255U + 4GB Chromebook. > > More data attached -- it contains vmstat, zoneinfo and pagetypeinfo > files collected before the benchmark (after fresh reboots) and after > the benchmark. Thanks, I'll take a look.
On 3/20/24 7:02 PM, Johannes Weiner wrote:
> V4:
> - fixed !pcp_order_allowed() case in free_unref_folios()
> - reworded the patch 0 changelog a bit for the git log
> - rebased to mm-everything-2024-03-19-23-01
> - runtime-tested again with various CONFIG_DEBUG_FOOs enabled
>
> ---
>
> The page allocator's mobility grouping is intended to keep unmovable
> pages separate from reclaimable/compactable ones to allow on-demand
> defragmentation for higher-order allocations and huge pages.
>
> Currently, there are several places where accidental type mixing
> occurs: an allocation asks for a page of a certain migratetype and
> receives another. This ruins pageblocks for compaction, which in turn
> makes allocating huge pages more expensive and less reliable.
>
> The series addresses those causes. The last patch adds type checks on
> all freelist movements to prevent new violations being introduced.
>
> The benefits can be seen in a mixed workload that stresses the machine
> with a memcache-type workload and a kernel build job while
> periodically attempting to allocate batches of THP. The following data
> is aggregated over 50 consecutive defconfig builds:
Great stuff. What would you say to the following on top?
----8<----
From 84f8a6d3a9e34c7ed8b438c3152d56e359a4ffb4 Mon Sep 17 00:00:00 2001
From: Vlastimil Babka <vbabka@suse.cz>
Date: Wed, 27 Mar 2024 10:19:47 +0100
Subject: [PATCH] mm: page_alloc: change move_freepages() to
__move_freepages_block()
The function is now supposed to be called only on a single pageblock and
checks start_pfn and end_pfn accordingly. Rename it to make this more
obvious and drop the end_pfn parameter which can be determined trivially
and none of the callers use it for anything else.
Also make the (now internal) end_pfn exclusive, which is more common.
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
---
mm/page_alloc.c | 43 ++++++++++++++++++++-----------------------
1 file changed, 20 insertions(+), 23 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 34c84ef16b66..75aefbd52ef9 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1566,18 +1566,18 @@ static inline struct page *__rmqueue_cma_fallback(struct zone *zone,
* Change the type of a block and move all its free pages to that
* type's freelist.
*/
-static int move_freepages(struct zone *zone, unsigned long start_pfn,
- unsigned long end_pfn, int old_mt, int new_mt)
+static int __move_freepages_block(struct zone *zone, unsigned long start_pfn,
+ int old_mt, int new_mt)
{
struct page *page;
- unsigned long pfn;
+ unsigned long pfn, end_pfn;
unsigned int order;
int pages_moved = 0;
VM_WARN_ON(start_pfn & (pageblock_nr_pages - 1));
- VM_WARN_ON(start_pfn + pageblock_nr_pages - 1 != end_pfn);
+ end_pfn = pageblock_end_pfn(start_pfn);
- for (pfn = start_pfn; pfn <= end_pfn;) {
+ for (pfn = start_pfn; pfn < end_pfn;) {
page = pfn_to_page(pfn);
if (!PageBuddy(page)) {
pfn++;
@@ -1603,14 +1603,13 @@ static int move_freepages(struct zone *zone, unsigned long start_pfn,
static bool prep_move_freepages_block(struct zone *zone, struct page *page,
unsigned long *start_pfn,
- unsigned long *end_pfn,
int *num_free, int *num_movable)
{
unsigned long pfn, start, end;
pfn = page_to_pfn(page);
start = pageblock_start_pfn(pfn);
- end = pageblock_end_pfn(pfn) - 1;
+ end = pageblock_end_pfn(pfn);
/*
* The caller only has the lock for @zone, don't touch ranges
@@ -1621,16 +1620,15 @@ static bool prep_move_freepages_block(struct zone *zone, struct page *page,
*/
if (!zone_spans_pfn(zone, start))
return false;
- if (!zone_spans_pfn(zone, end))
+ if (!zone_spans_pfn(zone, end - 1))
return false;
*start_pfn = start;
- *end_pfn = end;
if (num_free) {
*num_free = 0;
*num_movable = 0;
- for (pfn = start; pfn <= end;) {
+ for (pfn = start; pfn < end;) {
page = pfn_to_page(pfn);
if (PageBuddy(page)) {
int nr = 1 << buddy_order(page);
@@ -1656,13 +1654,12 @@ static bool prep_move_freepages_block(struct zone *zone, struct page *page,
static int move_freepages_block(struct zone *zone, struct page *page,
int old_mt, int new_mt)
{
- unsigned long start_pfn, end_pfn;
+ unsigned long start_pfn;
- if (!prep_move_freepages_block(zone, page, &start_pfn, &end_pfn,
- NULL, NULL))
+ if (!prep_move_freepages_block(zone, page, &start_pfn, NULL, NULL))
return -1;
- return move_freepages(zone, start_pfn, end_pfn, old_mt, new_mt);
+ return __move_freepages_block(zone, start_pfn, old_mt, new_mt);
}
#ifdef CONFIG_MEMORY_ISOLATION
@@ -1733,10 +1730,9 @@ static void split_large_buddy(struct zone *zone, struct page *page,
bool move_freepages_block_isolate(struct zone *zone, struct page *page,
int migratetype)
{
- unsigned long start_pfn, end_pfn, pfn;
+ unsigned long start_pfn, pfn;
- if (!prep_move_freepages_block(zone, page, &start_pfn, &end_pfn,
- NULL, NULL))
+ if (!prep_move_freepages_block(zone, page, &start_pfn, NULL, NULL))
return false;
/* No splits needed if buddies can't span multiple blocks */
@@ -1767,8 +1763,9 @@ bool move_freepages_block_isolate(struct zone *zone, struct page *page,
return true;
}
move:
- move_freepages(zone, start_pfn, end_pfn,
- get_pfnblock_migratetype(page, start_pfn), migratetype);
+ __move_freepages_block(zone, start_pfn,
+ get_pfnblock_migratetype(page, start_pfn),
+ migratetype);
return true;
}
#endif /* CONFIG_MEMORY_ISOLATION */
@@ -1868,7 +1865,7 @@ steal_suitable_fallback(struct zone *zone, struct page *page,
unsigned int alloc_flags, bool whole_block)
{
int free_pages, movable_pages, alike_pages;
- unsigned long start_pfn, end_pfn;
+ unsigned long start_pfn;
int block_type;
block_type = get_pageblock_migratetype(page);
@@ -1901,8 +1898,8 @@ steal_suitable_fallback(struct zone *zone, struct page *page,
goto single_page;
/* moving whole block can fail due to zone boundary conditions */
- if (!prep_move_freepages_block(zone, page, &start_pfn, &end_pfn,
- &free_pages, &movable_pages))
+ if (!prep_move_freepages_block(zone, page, &start_pfn, &free_pages,
+ &movable_pages))
goto single_page;
/*
@@ -1932,7 +1929,7 @@ steal_suitable_fallback(struct zone *zone, struct page *page,
*/
if (free_pages + alike_pages >= (1 << (pageblock_order-1)) ||
page_group_by_mobility_disabled) {
- move_freepages(zone, start_pfn, end_pfn, block_type, start_type);
+ __move_freepages_block(zone, start_pfn, block_type, start_type);
return __rmqueue_smallest(zone, order, start_type);
}
--
2.44.0
On Wed, Mar 27, 2024 at 10:30:30AM +0100, Vlastimil Babka wrote: > On 3/20/24 7:02 PM, Johannes Weiner wrote: > > V4: > > - fixed !pcp_order_allowed() case in free_unref_folios() > > - reworded the patch 0 changelog a bit for the git log > > - rebased to mm-everything-2024-03-19-23-01 > > - runtime-tested again with various CONFIG_DEBUG_FOOs enabled > > > > --- > > > > The page allocator's mobility grouping is intended to keep unmovable > > pages separate from reclaimable/compactable ones to allow on-demand > > defragmentation for higher-order allocations and huge pages. > > > > Currently, there are several places where accidental type mixing > > occurs: an allocation asks for a page of a certain migratetype and > > receives another. This ruins pageblocks for compaction, which in turn > > makes allocating huge pages more expensive and less reliable. > > > > The series addresses those causes. The last patch adds type checks on > > all freelist movements to prevent new violations being introduced. > > > > The benefits can be seen in a mixed workload that stresses the machine > > with a memcache-type workload and a kernel build job while > > periodically attempting to allocate batches of THP. The following data > > is aggregated over 50 consecutive defconfig builds: > > Great stuff. What would you say to the following on top? > > ----8<---- > From 84f8a6d3a9e34c7ed8b438c3152d56e359a4ffb4 Mon Sep 17 00:00:00 2001 > From: Vlastimil Babka <vbabka@suse.cz> > Date: Wed, 27 Mar 2024 10:19:47 +0100 > Subject: [PATCH] mm: page_alloc: change move_freepages() to > __move_freepages_block() > > The function is now supposed to be called only on a single pageblock and > checks start_pfn and end_pfn accordingly. Rename it to make this more > obvious and drop the end_pfn parameter which can be determined trivially > and none of the callers use it for anything else. > > Also make the (now internal) end_pfn exclusive, which is more common. > > Signed-off-by: Vlastimil Babka <vbabka@suse.cz> Nice, that's better. Acked-by: Johannes Weiner <hannes@cmpxchg.org>
On 27 Mar 2024, at 5:30, Vlastimil Babka wrote: > On 3/20/24 7:02 PM, Johannes Weiner wrote: >> V4: >> - fixed !pcp_order_allowed() case in free_unref_folios() >> - reworded the patch 0 changelog a bit for the git log >> - rebased to mm-everything-2024-03-19-23-01 >> - runtime-tested again with various CONFIG_DEBUG_FOOs enabled >> >> --- >> >> The page allocator's mobility grouping is intended to keep unmovable >> pages separate from reclaimable/compactable ones to allow on-demand >> defragmentation for higher-order allocations and huge pages. >> >> Currently, there are several places where accidental type mixing >> occurs: an allocation asks for a page of a certain migratetype and >> receives another. This ruins pageblocks for compaction, which in turn >> makes allocating huge pages more expensive and less reliable. >> >> The series addresses those causes. The last patch adds type checks on >> all freelist movements to prevent new violations being introduced. >> >> The benefits can be seen in a mixed workload that stresses the machine >> with a memcache-type workload and a kernel build job while >> periodically attempting to allocate batches of THP. The following data >> is aggregated over 50 consecutive defconfig builds: > > Great stuff. What would you say to the following on top? > > ----8<---- > From 84f8a6d3a9e34c7ed8b438c3152d56e359a4ffb4 Mon Sep 17 00:00:00 2001 > From: Vlastimil Babka <vbabka@suse.cz> > Date: Wed, 27 Mar 2024 10:19:47 +0100 > Subject: [PATCH] mm: page_alloc: change move_freepages() to > __move_freepages_block() > > The function is now supposed to be called only on a single pageblock and > checks start_pfn and end_pfn accordingly. Rename it to make this more > obvious and drop the end_pfn parameter which can be determined trivially > and none of the callers use it for anything else. > > Also make the (now internal) end_pfn exclusive, which is more common. > > Signed-off-by: Vlastimil Babka <vbabka@suse.cz> > --- > mm/page_alloc.c | 43 ++++++++++++++++++++----------------------- > 1 file changed, 20 insertions(+), 23 deletions(-) > This looks good to me and makes sense. Reviewed-by: Zi Yan <ziy@nvidia.com> -- Best Regards, Yan, Zi
© 2016 - 2025 Red Hat, Inc.