[v1] PCI: hv: Warn when wait_for_response() waits indefinitely

[PATCH] PCI: hv: Warn when wait_for_response() waits indefinitely

Posted by Sahil Chandna 1 month ago

A guest can wait indefinitely in wait_for_response() for the host to
send either a rescind message or a packet completion. If the
host does not send either, the guest can remain blocked with no
diagnostic indicating a reason.
This was observed during a guest kernel upgrade in which the
host-side application handling the PCI channel faulted, causing the
guest to never receive the completion request.
Add a periodic warning in wait_for_response() when the wait exceeds
a timeout so that such a hang is visible in the guest's kernel log
and can be correlated with host-side state.

Suggested-by: Hamza Mahfooz <hamzamahfooz@linux.microsoft.com>
Signed-off-by: Sahil Chandna <sahilchandna@linux.microsoft.com>
---
This was sent earlier upstream [1]
[1] https://lore.kernel.org/linux-hyperv/20260612174010.2598695-1-hamzamahfooz@linux.microsoft.com/
---
 drivers/pci/controller/pci-hyperv.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/drivers/pci/controller/pci-hyperv.c b/drivers/pci/controller/pci-hyperv.c
index cfc8fa403dad..c4fba0039164 100644
--- a/drivers/pci/controller/pci-hyperv.c
+++ b/drivers/pci/controller/pci-hyperv.c
@@ -1040,11 +1040,18 @@ static void put_pcichild(struct hv_pci_dev *hpdev)

 /*
  * There is no good way to get notified from vmbus_onoffer_rescind(),
- * so let's use polling here, since this is not a hot path.
+ * so let's use polling here, since this is not a hot path. If
+ * wait_for_response() has been polling for PCI_RESPONSE_HANG_TIMEOUT_SEC
+ * without either a rescind or completion, add a periodic warning.
  */
+#define PCI_RESPONSE_HANG_TIMEOUT_SEC 300
+
 static int wait_for_response(struct hv_device *hdev,
 			     struct completion *comp)
 {
+	unsigned long delay = secs_to_jiffies(PCI_RESPONSE_HANG_TIMEOUT_SEC);
+	u64 timeout = get_jiffies_64() + delay;
+
 	while (true) {
 		if (hdev->channel->rescind) {
 			dev_warn_once(&hdev->device, "The device is gone.\n");
@@ -1053,6 +1060,11 @@ static int wait_for_response(struct hv_device *hdev,

 		if (wait_for_completion_timeout(comp, HZ / 10))
 			break;
+
+		if (time_after64(get_jiffies_64(), timeout)) {
+			dev_warn(&hdev->device, "PCI stuck waiting for response.\n");
+			timeout = get_jiffies_64() + delay;
+		}
 	}

 	return 0;
--
2.53.0

Re: [PATCH] PCI: hv: Warn when wait_for_response() waits indefinitely

Posted by Naman Jain 1 month ago


On 8/25/2026 10:47 AM, Sahil Chandna wrote:
> A guest can wait indefinitely in wait_for_response() for the host to
> send either a rescind message or a packet completion. If the
> host does not send either, the guest can remain blocked with no
> diagnostic indicating a reason.
> This was observed during a guest kernel upgrade in which the
> host-side application handling the PCI channel faulted, causing the
> guest to never receive the completion request.
> Add a periodic warning in wait_for_response() when the wait exceeds
> a timeout so that such a hang is visible in the guest's kernel log
> and can be correlated with host-side state.
> 
> Suggested-by: Hamza Mahfooz <hamzamahfooz@linux.microsoft.com>
> Signed-off-by: Sahil Chandna <sahilchandna@linux.microsoft.com>
> ---
> This was sent earlier upstream [1]
> [1] https://lore.kernel.org/linux-hyperv/20260612174010.2598695-1-hamzamahfooz@linux.microsoft.com/
> ---
>   drivers/pci/controller/pci-hyperv.c | 14 +++++++++++++-
>   1 file changed, 13 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/pci/controller/pci-hyperv.c b/drivers/pci/controller/pci-hyperv.c
> index cfc8fa403dad..c4fba0039164 100644
> --- a/drivers/pci/controller/pci-hyperv.c
> +++ b/drivers/pci/controller/pci-hyperv.c
> @@ -1040,11 +1040,18 @@ static void put_pcichild(struct hv_pci_dev *hpdev)
> 
>   /*
>    * There is no good way to get notified from vmbus_onoffer_rescind(),
> - * so let's use polling here, since this is not a hot path.
> + * so let's use polling here, since this is not a hot path. If
> + * wait_for_response() has been polling for PCI_RESPONSE_HANG_TIMEOUT_SEC
> + * without either a rescind or completion, add a periodic warning.
>    */
> +#define PCI_RESPONSE_HANG_TIMEOUT_SEC 300
> +
>   static int wait_for_response(struct hv_device *hdev,
>   			     struct completion *comp)
>   {
> +	unsigned long delay = secs_to_jiffies(PCI_RESPONSE_HANG_TIMEOUT_SEC);
> +	u64 timeout = get_jiffies_64() + delay;
> +
>   	while (true) {
>   		if (hdev->channel->rescind) {
>   			dev_warn_once(&hdev->device, "The device is gone.\n");
> @@ -1053,6 +1060,11 @@ static int wait_for_response(struct hv_device *hdev,
> 
>   		if (wait_for_completion_timeout(comp, HZ / 10))
>   			break;
> +
> +		if (time_after64(get_jiffies_64(), timeout)) {
> +			dev_warn(&hdev->device, "PCI stuck waiting for response.\n");
> +			timeout = get_jiffies_64() + delay;
> +		}
>   	}

There can be some enhancements in above patch to address these problems:
1. Logging forever every 5 minutes in case of no completion or rescind.
2. If we now print warning once, not knowing if completion ever arrived.
3. On solving pt. 1 and 2 by adding a print for completion, one should 
avoid adding a print by default for regular timely completions.

Basically something like this:

#define PCI_RESPONSE_WARN_TIMEOUT_SEC 300

static int wait_for_response(struct hv_device *hdev,
                  struct completion *comp)
{
     unsigned long warn_at =
         jiffies + secs_to_jiffies(PCI_RESPONSE_WARN_TIMEOUT_SEC);
     bool warned = false;

     while (true) {
         if (hdev->channel->rescind) {
             dev_warn_once(&hdev->device, "The device is gone.\n");
             return -ENODEV;
         }

         if (wait_for_completion_timeout(comp, HZ / 10)) {
             if (warned || time_after_eq(jiffies, warn_at))
                 dev_warn(&hdev->device,
                      "PCI response received after prolonged wait.\n");
             return 0;
         }

         if (!warned && time_after_eq(jiffies, warn_at)) {
             dev_warn(&hdev->device,
                  "PCI still waiting for response.\n");
             warned = true;
         }
     }
}

Regards,
Naman

Re: [PATCH] PCI: hv: Warn when wait_for_response() waits indefinitely

Posted by Sahil Chandna 1 month ago

On 25-08-2026 12:16, Naman Jain wrote:
> 
> 
> On 8/25/2026 10:47 AM, Sahil Chandna wrote:
>> A guest can wait indefinitely in wait_for_response() for the host to
>> send either a rescind message or a packet completion. If the
>> host does not send either, the guest can remain blocked with no
>> diagnostic indicating a reason.
>> This was observed during a guest kernel upgrade in which the
>> host-side application handling the PCI channel faulted, causing the
>> guest to never receive the completion request.
>> Add a periodic warning in wait_for_response() when the wait exceeds
>> a timeout so that such a hang is visible in the guest's kernel log
>> and can be correlated with host-side state.
>>
>> Suggested-by: Hamza Mahfooz <hamzamahfooz@linux.microsoft.com>
>> Signed-off-by: Sahil Chandna <sahilchandna@linux.microsoft.com>
>> ---
>> This was sent earlier upstream [1]
>> [1] https://lore.kernel.org/linux-hyperv/20260612174010.2598695-1-hamzamahfooz@linux.microsoft.com/
>> ---
>>   drivers/pci/controller/pci-hyperv.c | 14 +++++++++++++-
>>   1 file changed, 13 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/pci/controller/pci-hyperv.c b/drivers/pci/controller/pci-hyperv.c
>> index cfc8fa403dad..c4fba0039164 100644
>> --- a/drivers/pci/controller/pci-hyperv.c
>> +++ b/drivers/pci/controller/pci-hyperv.c
>> @@ -1040,11 +1040,18 @@ static void put_pcichild(struct hv_pci_dev *hpdev)
>>
>>   /*
>>    * There is no good way to get notified from vmbus_onoffer_rescind(),
>> - * so let's use polling here, since this is not a hot path.
>> + * so let's use polling here, since this is not a hot path. If
>> + * wait_for_response() has been polling for PCI_RESPONSE_HANG_TIMEOUT_SEC
>> + * without either a rescind or completion, add a periodic warning.
>>    */
>> +#define PCI_RESPONSE_HANG_TIMEOUT_SEC 300
>> +
>>   static int wait_for_response(struct hv_device *hdev,
>>   			     struct completion *comp)
>>   {
>> +	unsigned long delay = secs_to_jiffies(PCI_RESPONSE_HANG_TIMEOUT_SEC);
>> +	u64 timeout = get_jiffies_64() + delay;
>> +
>>   	while (true) {
>>   		if (hdev->channel->rescind) {
>>   			dev_warn_once(&hdev->device, "The device is gone.\n");
>> @@ -1053,6 +1060,11 @@ static int wait_for_response(struct hv_device *hdev,
>>
>>   		if (wait_for_completion_timeout(comp, HZ / 10))
>>   			break;
>> +
>> +		if (time_after64(get_jiffies_64(), timeout)) {
>> +			dev_warn(&hdev->device, "PCI stuck waiting for response.\n");
>> +			timeout = get_jiffies_64() + delay;
>> +		}
>>   	}
> 
> There can be some enhancements in above patch to address these problems:
> 1. Logging forever every 5 minutes in case of no completion or rescind.
> 2. If we now print warning once, not knowing if completion ever arrived.
> 3. On solving pt. 1 and 2 by adding a print for completion, one should 
> avoid adding a print by default for regular timely completions.
> 
Ack.
> Basically something like this:
> 
> #define PCI_RESPONSE_WARN_TIMEOUT_SEC 300
> 
> static int wait_for_response(struct hv_device *hdev,
>                   struct completion *comp)
> {
>      unsigned long warn_at =
>          jiffies + secs_to_jiffies(PCI_RESPONSE_WARN_TIMEOUT_SEC);
>      bool warned = false;
> 
>      while (true) {
>          if (hdev->channel->rescind) {
>              dev_warn_once(&hdev->device, "The device is gone.\n");
>              return -ENODEV;
>          }
> 
>          if (wait_for_completion_timeout(comp, HZ / 10)) {
>              if (warned || time_after_eq(jiffies, warn_at))
>                  dev_warn(&hdev->device,
>                       "PCI response received after prolonged wait.\n");
>              return 0;
>          }
> 
>          if (!warned && time_after_eq(jiffies, warn_at)) {
>              dev_warn(&hdev->device,
>                   "PCI still waiting for response.\n");
>              warned = true;
>          }
>      }
> }
> 
Thanks for review, I align on repeated warning every 5 minutes for
stalled guest would add to dmesg noise.I will wait for other review
comments as well and share v2 addressing this.
> Regards,
> Naman