[PATCH V2 11/13] selftests/resctrl: Use cache size to determine "fill_buf" buffer size

Reinette Chatre posted 13 patches 2 months, 2 weeks ago
There is a newer version of this series
[PATCH V2 11/13] selftests/resctrl: Use cache size to determine "fill_buf" buffer size
Posted by Reinette Chatre 2 months, 2 weeks ago
By default the MBM and MBA tests use the "fill_buf" benchmark to
read from a buffer with the goal to measure the memory bandwidth
generated by this buffer access.

Care should be taken when sizing the buffer used by the "fill_buf"
benchmark. If the buffer is small enough to fit in the cache then
it cannot be expected that the benchmark will generate much memory
bandwidth. For example, on a system with 320MB L3 cache the existing
hardcoded default of 250MB is insufficient.

Use the measured cache size to determine a buffer size that can be
expected to trigger memory access while keeping the existing default
as minimum that has been appropriate for testing so far.

Signed-off-by: Reinette Chatre <reinette.chatre@intel.com>
---
Changes since V1:
- Ensure buffer is at least double L3 cache size. (Ilpo)
- Support user override of default buffer size. (Ilpo)
---
 tools/testing/selftests/resctrl/mba_test.c | 8 +++++++-
 tools/testing/selftests/resctrl/mbm_test.c | 8 +++++++-
 2 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/resctrl/mba_test.c b/tools/testing/selftests/resctrl/mba_test.c
index 7e43056c8737..d8d9637c1951 100644
--- a/tools/testing/selftests/resctrl/mba_test.c
+++ b/tools/testing/selftests/resctrl/mba_test.c
@@ -182,7 +182,13 @@ static int mba_run_test(const struct resctrl_test *test, const struct user_param
 		fill_buf.memflush = uparams->fill_buf->memflush;
 		param.fill_buf = &fill_buf;
 	} else if (!uparams->benchmark_cmd[0]) {
-		fill_buf.buf_size = DEFAULT_SPAN;
+		unsigned long cache_total_size = 0;
+
+		ret = get_cache_size(uparams->cpu, "L3", &cache_total_size);
+		if (ret)
+			return ret;
+		fill_buf.buf_size = cache_total_size * 2 > DEFAULT_SPAN ?
+				    cache_total_size * 2 : DEFAULT_SPAN;
 		fill_buf.memflush = 1;
 		param.fill_buf = &fill_buf;
 	}
diff --git a/tools/testing/selftests/resctrl/mbm_test.c b/tools/testing/selftests/resctrl/mbm_test.c
index b1f03a73333f..7635ee6b9339 100644
--- a/tools/testing/selftests/resctrl/mbm_test.c
+++ b/tools/testing/selftests/resctrl/mbm_test.c
@@ -149,7 +149,13 @@ static int mbm_run_test(const struct resctrl_test *test, const struct user_param
 		fill_buf.memflush = uparams->fill_buf->memflush;
 		param.fill_buf = &fill_buf;
 	} else if (!uparams->benchmark_cmd[0]) {
-		fill_buf.buf_size = DEFAULT_SPAN;
+		unsigned long cache_total_size = 0;
+
+		ret = get_cache_size(uparams->cpu, "L3", &cache_total_size);
+		if (ret)
+			return ret;
+		fill_buf.buf_size = cache_total_size * 2 > DEFAULT_SPAN ?
+				    cache_total_size * 2 : DEFAULT_SPAN;
 		fill_buf.memflush = 1;
 		param.fill_buf = &fill_buf;
 	}
-- 
2.46.0
Re: [PATCH V2 11/13] selftests/resctrl: Use cache size to determine "fill_buf" buffer size
Posted by Ilpo Järvinen 1 month, 3 weeks ago
On Thu, 12 Sep 2024, Reinette Chatre wrote:

> By default the MBM and MBA tests use the "fill_buf" benchmark to
> read from a buffer with the goal to measure the memory bandwidth
> generated by this buffer access.
> 
> Care should be taken when sizing the buffer used by the "fill_buf"
> benchmark. If the buffer is small enough to fit in the cache then
> it cannot be expected that the benchmark will generate much memory
> bandwidth. For example, on a system with 320MB L3 cache the existing
> hardcoded default of 250MB is insufficient.
> 
> Use the measured cache size to determine a buffer size that can be
> expected to trigger memory access while keeping the existing default
> as minimum that has been appropriate for testing so far.
> 
> Signed-off-by: Reinette Chatre <reinette.chatre@intel.com>
> ---
> Changes since V1:
> - Ensure buffer is at least double L3 cache size. (Ilpo)
> - Support user override of default buffer size. (Ilpo)
> ---
>  tools/testing/selftests/resctrl/mba_test.c | 8 +++++++-
>  tools/testing/selftests/resctrl/mbm_test.c | 8 +++++++-
>  2 files changed, 14 insertions(+), 2 deletions(-)
> 
> diff --git a/tools/testing/selftests/resctrl/mba_test.c b/tools/testing/selftests/resctrl/mba_test.c
> index 7e43056c8737..d8d9637c1951 100644
> --- a/tools/testing/selftests/resctrl/mba_test.c
> +++ b/tools/testing/selftests/resctrl/mba_test.c
> @@ -182,7 +182,13 @@ static int mba_run_test(const struct resctrl_test *test, const struct user_param
>  		fill_buf.memflush = uparams->fill_buf->memflush;
>  		param.fill_buf = &fill_buf;
>  	} else if (!uparams->benchmark_cmd[0]) {
> -		fill_buf.buf_size = DEFAULT_SPAN;
> +		unsigned long cache_total_size = 0;
> +
> +		ret = get_cache_size(uparams->cpu, "L3", &cache_total_size);
> +		if (ret)
> +			return ret;
> +		fill_buf.buf_size = cache_total_size * 2 > DEFAULT_SPAN ?
> +				    cache_total_size * 2 : DEFAULT_SPAN;
>  		fill_buf.memflush = 1;
>  		param.fill_buf = &fill_buf;
>  	}
> diff --git a/tools/testing/selftests/resctrl/mbm_test.c b/tools/testing/selftests/resctrl/mbm_test.c
> index b1f03a73333f..7635ee6b9339 100644
> --- a/tools/testing/selftests/resctrl/mbm_test.c
> +++ b/tools/testing/selftests/resctrl/mbm_test.c
> @@ -149,7 +149,13 @@ static int mbm_run_test(const struct resctrl_test *test, const struct user_param
>  		fill_buf.memflush = uparams->fill_buf->memflush;
>  		param.fill_buf = &fill_buf;
>  	} else if (!uparams->benchmark_cmd[0]) {
> -		fill_buf.buf_size = DEFAULT_SPAN;
> +		unsigned long cache_total_size = 0;
> +
> +		ret = get_cache_size(uparams->cpu, "L3", &cache_total_size);
> +		if (ret)
> +			return ret;
> +		fill_buf.buf_size = cache_total_size * 2 > DEFAULT_SPAN ?
> +				    cache_total_size * 2 : DEFAULT_SPAN;
>  		fill_buf.memflush = 1;
>  		param.fill_buf = &fill_buf;
>  	}
> 

It has a bit of code duplication feel in it so I'd consider adding 
something like ssize_t get_default_span() (perhaps there exists a better 
name for it that is not "span" based). Also DEFAULT_SPAN is no longer
truly the default span.

But neither is the end of the world as is...

Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>


-- 
 i.

Re: [PATCH V2 11/13] selftests/resctrl: Use cache size to determine "fill_buf" buffer size
Posted by Reinette Chatre 1 month, 3 weeks ago
Hi Ilpo,

On 10/4/24 7:20 AM, Ilpo Järvinen wrote:
> On Thu, 12 Sep 2024, Reinette Chatre wrote:
> 
>> By default the MBM and MBA tests use the "fill_buf" benchmark to
>> read from a buffer with the goal to measure the memory bandwidth
>> generated by this buffer access.
>>
>> Care should be taken when sizing the buffer used by the "fill_buf"
>> benchmark. If the buffer is small enough to fit in the cache then
>> it cannot be expected that the benchmark will generate much memory
>> bandwidth. For example, on a system with 320MB L3 cache the existing
>> hardcoded default of 250MB is insufficient.
>>
>> Use the measured cache size to determine a buffer size that can be
>> expected to trigger memory access while keeping the existing default
>> as minimum that has been appropriate for testing so far.
>>
>> Signed-off-by: Reinette Chatre <reinette.chatre@intel.com>
>> ---
>> Changes since V1:
>> - Ensure buffer is at least double L3 cache size. (Ilpo)
>> - Support user override of default buffer size. (Ilpo)
>> ---
>>  tools/testing/selftests/resctrl/mba_test.c | 8 +++++++-
>>  tools/testing/selftests/resctrl/mbm_test.c | 8 +++++++-
>>  2 files changed, 14 insertions(+), 2 deletions(-)
>>
>> diff --git a/tools/testing/selftests/resctrl/mba_test.c b/tools/testing/selftests/resctrl/mba_test.c
>> index 7e43056c8737..d8d9637c1951 100644
>> --- a/tools/testing/selftests/resctrl/mba_test.c
>> +++ b/tools/testing/selftests/resctrl/mba_test.c
>> @@ -182,7 +182,13 @@ static int mba_run_test(const struct resctrl_test *test, const struct user_param
>>  		fill_buf.memflush = uparams->fill_buf->memflush;
>>  		param.fill_buf = &fill_buf;
>>  	} else if (!uparams->benchmark_cmd[0]) {
>> -		fill_buf.buf_size = DEFAULT_SPAN;
>> +		unsigned long cache_total_size = 0;
>> +
>> +		ret = get_cache_size(uparams->cpu, "L3", &cache_total_size);
>> +		if (ret)
>> +			return ret;
>> +		fill_buf.buf_size = cache_total_size * 2 > DEFAULT_SPAN ?
>> +				    cache_total_size * 2 : DEFAULT_SPAN;
>>  		fill_buf.memflush = 1;
>>  		param.fill_buf = &fill_buf;
>>  	}
>> diff --git a/tools/testing/selftests/resctrl/mbm_test.c b/tools/testing/selftests/resctrl/mbm_test.c
>> index b1f03a73333f..7635ee6b9339 100644
>> --- a/tools/testing/selftests/resctrl/mbm_test.c
>> +++ b/tools/testing/selftests/resctrl/mbm_test.c
>> @@ -149,7 +149,13 @@ static int mbm_run_test(const struct resctrl_test *test, const struct user_param
>>  		fill_buf.memflush = uparams->fill_buf->memflush;
>>  		param.fill_buf = &fill_buf;
>>  	} else if (!uparams->benchmark_cmd[0]) {
>> -		fill_buf.buf_size = DEFAULT_SPAN;
>> +		unsigned long cache_total_size = 0;
>> +
>> +		ret = get_cache_size(uparams->cpu, "L3", &cache_total_size);
>> +		if (ret)
>> +			return ret;
>> +		fill_buf.buf_size = cache_total_size * 2 > DEFAULT_SPAN ?
>> +				    cache_total_size * 2 : DEFAULT_SPAN;
>>  		fill_buf.memflush = 1;
>>  		param.fill_buf = &fill_buf;
>>  	}
>>
> 
> It has a bit of code duplication feel in it so I'd consider adding 
> something like ssize_t get_default_span() (perhaps there exists a better 
> name for it that is not "span" based). Also DEFAULT_SPAN is no longer
> truly the default span.

Good suggestion.  How about get_fill_buf_size()?

Regarding the "DEFAULT_SPAN" it seems to have some 'magic number smell' to
it. When I tried to eliminate it and only rely on cache size the tests
were failing on some earlier platforms with smaller L3. I then started
treating it as a minimum in order to keep original behavior on these earlier
systems. Perhaps renaming it to "MINIMUM_SPAN" would make this change
clear?

> 
> But neither is the end of the world as is...
> 
> Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
> 
> 

Thank you very much.

Reinette