Marcio Cunha

GitLab CI pipeline stuck? How to cancel pending jobs

Are your GitLab pipelines stuck in an infinite loop? Learn how to use the REST API to clear jammed jobs and restore your CI/CD flow.

Marcio Cunha•2 min
Also available in:PortuguêsEspañol
Summary
  • Stuck pipelines often happen due to communication failures between GitLab Runners and the coordinator.
  • The web interface sometimes fails to terminate jobs that have lost their execution signal.
  • REST API access allows for direct intervention to force-terminate hung processes.
  • Personal access tokens with API permissions are required to authenticate cleanup requests.
  • Monitoring Runner status is essential to prevent jobs from becoming orphans during heavy loads.

Why do GitLab CI pipelines get stuck?

Every automation engineer knows the frustration of a pipeline sitting in 'pending' or 'running' indefinitely. In practice, this happens when the GitLab Runner, the agent executing your tasks, loses its connection to the server or the job becomes an orphan after a container failure. Much like when you face pods stuck in Terminating, the orchestration system often fails to realize the task has died.

Understanding orphan jobs

When a job hangs, it consumes valuable concurrency slots. If your simultaneous execution limit is low, one or two ghost jobs can halt your entire delivery flow. The root cause is often intermittent network issues or timeouts that the agent failed to handle gracefully. If you have dealt with a stuck Docker container, you know that force-killing the process is the only immediate fix to resume normal operation.

Using the REST API for cleanup

To fix this without restarting the whole server, we use the GitLab API. You need a Personal Access Token with 'api' scope. With this token, we can list stuck jobs and cancel them individually. The following command uses cURL to target the project ID and the job ID that needs to be cleared.

curl --header "PRIVATE-TOKEN: <your_token>" "https://gitlab.com/api/v4/projects/<PROJECT_ID>/jobs/<JOB_ID>/cancel" --request POST

Automating the pending job sweep

Manual intervention is not sustainable for recurrent issues. We can write a simple script that filters for jobs in the 'running' state with a duration longer than X minutes and issues a cancellation via API. This prevents constant manual dashboard monitoring. Much like managing a slow NGINX instance, proactive telemetry saves hours of maintenance.

Conclusion: keeping your CI/CD healthy

Keeping your pipeline queue clean is vital for a culture of rapid deployment. If jobs are constantly getting stuck, the issue might lie in your Runner provisioning. Consider increasing machine resources or optimizing network communication to prevent instances from becoming isolated. Manual management via API should be a fallback; the root cause should always be addressed in your underlying infrastructure.