ipmi: fix SSIF not responding under certain cond.

[PATCH] ipmi: fix SSIF not responding under certain cond.

Posted by Zhang Yuchen 2 years, 10 months ago

The ipmi communication is not restored after a specific version of BMC is
upgraded on our server.
The ipmi driver does not respond after printing the following log:

    ipmi_ssif: Invalid response getting flags: 1c 1

I found that after entering this branch, ssif_info->ssif_state always
holds SSIF_GETTING_FLAGS and never return to IDLE.

As a result, the driver cannot be loaded, because the driver status is
checked during the unload process and must be IDLE in shutdown_ssif():

        while (ssif_info->ssif_state != SSIF_IDLE)
                schedule_timeout(1);

The process trigger this problem is:

1. One msg timeout and next msg start send, and call
ssif_set_need_watch().

2. ssif_set_need_watch()->watch_timeout()->start_flag_fetch() change
ssif_state to SSIF_GETTING_FLAGS.

3. In msg_done_handler() ssif_state == SSIF_GETTING_FLAGS, if an error
message is received, the second branch does not modify the ssif_state.

4. All retry action need IS_SSIF_IDLE() == True. Include retry action in
watch_timeout(), msg_done_handler(). Sending msg does not work either.
SSIF_IDLE is also checked in start_next_msg().

5. The only thing that can be triggered in the SSIF driver is
watch_timeout(), after destory_user(), this timer will stop too.

So, if enter this branch, the ssif_state will remain SSIF_GETTING_FLAGS
and can't send msg, no timer started, can't unload.

We did a comparative test before and after adding this patch, and the
result is effective.

Fixes: 259307074bfc ("ipmi: Add SMBus interface driver (SSIF)")

Signed-off-by: Zhang Yuchen <zhangyuchen.lcr@bytedance.com>
---
 drivers/char/ipmi/ipmi_ssif.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/char/ipmi/ipmi_ssif.c b/drivers/char/ipmi/ipmi_ssif.c
index a5ddebb1edea..48be3694fa64 100644
--- a/drivers/char/ipmi/ipmi_ssif.c
+++ b/drivers/char/ipmi/ipmi_ssif.c
@@ -784,9 +784,9 @@ static void msg_done_handler(struct ssif_info *ssif_info, int result,
 		} else if (data[0] != (IPMI_NETFN_APP_REQUEST | 1) << 2
 			   || data[1] != IPMI_GET_MSG_FLAGS_CMD) {
 			/*
-			 * Don't abort here, maybe it was a queued
-			 * response to a previous command.
+			 * Recv error response, give up.
 			 */
+			ssif_info->ssif_state = SSIF_IDLE;
 			ipmi_ssif_unlock_cond(ssif_info, flags);
 			dev_warn(&ssif_info->client->dev,
 				 "Invalid response getting flags: %x %x\n",
-- 
2.20.1

Re: [PATCH] ipmi: fix SSIF not responding under certain cond.

Posted by Corey Minyard 2 years, 10 months ago

On Wed, Apr 12, 2023 at 03:49:07PM +0800, Zhang Yuchen wrote:
> The ipmi communication is not restored after a specific version of BMC is
> upgraded on our server.
> The ipmi driver does not respond after printing the following log:

Excellent explaination, this is queued in my branch for the next release
and marked for backport.

Thank you,

-corey

> 
>     ipmi_ssif: Invalid response getting flags: 1c 1
> 
> I found that after entering this branch, ssif_info->ssif_state always
> holds SSIF_GETTING_FLAGS and never return to IDLE.
> 
> As a result, the driver cannot be loaded, because the driver status is
> checked during the unload process and must be IDLE in shutdown_ssif():
> 
>         while (ssif_info->ssif_state != SSIF_IDLE)
>                 schedule_timeout(1);
> 
> The process trigger this problem is:
> 
> 1. One msg timeout and next msg start send, and call
> ssif_set_need_watch().
> 
> 2. ssif_set_need_watch()->watch_timeout()->start_flag_fetch() change
> ssif_state to SSIF_GETTING_FLAGS.
> 
> 3. In msg_done_handler() ssif_state == SSIF_GETTING_FLAGS, if an error
> message is received, the second branch does not modify the ssif_state.
> 
> 4. All retry action need IS_SSIF_IDLE() == True. Include retry action in
> watch_timeout(), msg_done_handler(). Sending msg does not work either.
> SSIF_IDLE is also checked in start_next_msg().
> 
> 5. The only thing that can be triggered in the SSIF driver is
> watch_timeout(), after destory_user(), this timer will stop too.
> 
> So, if enter this branch, the ssif_state will remain SSIF_GETTING_FLAGS
> and can't send msg, no timer started, can't unload.
> 
> We did a comparative test before and after adding this patch, and the
> result is effective.
> 
> Fixes: 259307074bfc ("ipmi: Add SMBus interface driver (SSIF)")
> 
> Signed-off-by: Zhang Yuchen <zhangyuchen.lcr@bytedance.com>
> ---
>  drivers/char/ipmi/ipmi_ssif.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/char/ipmi/ipmi_ssif.c b/drivers/char/ipmi/ipmi_ssif.c
> index a5ddebb1edea..48be3694fa64 100644
> --- a/drivers/char/ipmi/ipmi_ssif.c
> +++ b/drivers/char/ipmi/ipmi_ssif.c
> @@ -784,9 +784,9 @@ static void msg_done_handler(struct ssif_info *ssif_info, int result,
>  		} else if (data[0] != (IPMI_NETFN_APP_REQUEST | 1) << 2
>  			   || data[1] != IPMI_GET_MSG_FLAGS_CMD) {
>  			/*
> -			 * Don't abort here, maybe it was a queued
> -			 * response to a previous command.
> +			 * Recv error response, give up.
>  			 */
> +			ssif_info->ssif_state = SSIF_IDLE;
>  			ipmi_ssif_unlock_cond(ssif_info, flags);
>  			dev_warn(&ssif_info->client->dev,
>  				 "Invalid response getting flags: %x %x\n",
> -- 
> 2.20.1
>