Skip to content

Runner scale set listener pods are frequently recreating with \"reason\":\"Error\" #4356

@rajesh-dhakad

Description

@rajesh-dhakad

Checks

Controller Version

0.13.1

Deployment Method

Helm

Checks

  • This isn't a question or user support case (For Q&A and community support, go to Discussions).
  • I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

We started noticing a couple of days ago that runner listener pods are being recreated too frequently. We upgraded the controller version from 0.13.0 to 0.13.1, but we are still seeing the same behavior, so we are unsure which version this issue started in.

I am not sure how we can reproduce it, but I am going to provide the controller and listener logs to check.


Describe the bug

Runner scale set listener pods are frequently recreating with "reason":"Error"

Image

Describe the expected behavior

Listener pod should be in a stable state - frequenty restart cause delay in CI/CD jobs.

Additional Context

if !errors.As(err, &expiredError) {
--> this should only happen when the token expires, but we are not seeing any pattern like 30m, 60m.

In listener/controller pod logs - we are seeing below before it is being recreated. We have debug log level enabled for the controller, but don't see much information except below.

1768712941128	2026-01-18T05:09:01.128Z	{"container":"listener","pod":"general-metadata-755cf869-listener","_entry":"I0118 05:09:00.959725       1 warnings.go:110] \"Warning: unknown field \\\"status.jobId\\\"\""}
1768712944385	2026-01-18T05:09:04.385Z	{"container":"listener","pod":"general-metadata-755cf869-listener","_entry":"2026/01/18 05:09:04 Application returned an error: failed to get message: failed to get next message: actions error: StatusCode 200, AcivityId \"\": EOF"}
1768712945426	2026-01-18T05:09:05.426Z	{"container":"manager","level":"info","pod":"arc-controller-67c9bf8ccb-nb528","_entry":"{\"severity\":\"info\",\"ts\":\"2026-01-18T05:09:05Z\",\"logger\":\"AutoscalingListener\",\"message\":\"Listener pod is terminated\",\"version\":\"0.13.1\",\"autoscalinglistener\":{\"name\":\"general-metadata-755cf869-listener\",\"namespace\":\"arc-system\"},\"namespace\":\"arc-system\",\"name\":\"general-metadata-755cf869-listener\",\"reason\":\"Error\",\"message\":\"\"}"}
1768712945426	2026-01-18T05:09:05.426Z	{"container":"manager","level":"info","pod":"arc-controller-67c9bf8ccb-nb528","_entry":"{\"severity\":\"info\",\"ts\":\"2026-01-18T05:09:05Z\",\"logger\":\"AutoscalingListener\",\"message\":\"Deleting the listener pod\",\"version\":\"0.13.1\",\"autoscalinglistener\":{\"name\":\"general-metadata-755cf869-listener\",\"namespace\":\"arc-system\"},\"namespace\":\"arc-system\",\"name\":\"general-metadata-755cf869-listener\"}"}

Controller Logs

https://gist.github.com/rajesh-dhakad/2c8e2fc07746fab7a3deab71ab99e7bf

Runner Pod Logs

none ( not applicable )

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinggha-runner-scale-setRelated to the gha-runner-scale-set modeneeds triageRequires review from the maintainers

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions