Ephemeral Runners with Github Action Runner Controllers and Kubernetes

Mar 24, 2022 kubernetes github bbc

If you're locked down to repository-only GitHub runners and have a Kubernetes environment locally, you can run ephemeral runners using GitHubs Action Runner Controllers solution.

Here's how to get started. You can deploy a similar setup in AWS with Elastic Kubernetes Service, but it can be quite expensive, as you need to run at least one controller node to watch for new jobs, and respond to requests for new workflow runs. Right now this is approx USD 75 per month for an EKS cluster running continuously before charges for vCPU etc.

If you don't have access to the repository, you will need a Personal Access Token, and will have to manage that in secrets.

Installing Kubernetes with Minikube locally

The quickstart on a MacBook pro involves installing the Kubernetes CLI tool kubektl, minikube which is a micro-Kubernetes cluster runner, and then helm which is a CLI and set of tools to manage Kubernetes packages.

I like Podman instead of Docker and I'm switching over to it. They let you manage local Kubernetes and have a one-click install for Minikube. You can also install Minikube via brew which I used in my experimentation.

brew install kubektl
brew install minikube
brew install helm

Generate a GitHub Personal Access Token (PAT) with enough scope to manage your github runners and read your repositories. Remember this PAT will be able to see everything in your world.

Then from the docs we need to spin up a space for the runners and then enable the management node and listener bits:

NAMESPACE="arc-systems"

helm install arc \
    --namespace "${NAMESPACE}" \
    --create-namespace \
    oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller

Then a set of runners for a particular URL:

GITHUB_PAT="<YOUR_PAT_HERE>"
INSTALLATION_NAME="arc-runner-set"
NAMESPACE="arc-runners"
GITHUB_CONFIG_URL="https://github.com/<REPO>"

helm install "${INSTALLATION_NAME}" \
    --namespace "${NAMESPACE}" \
    --create-namespace \
    --set githubConfigUrl="${GITHUB_CONFIG_URL}" \
    --set githubConfigSecret.github_token="${GITHUB_PAT}" \
    oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set

You can repeat the above runners for each repository (see below).

Ensure your "runs-on" for the repository is set to the NAMESPACE above.

Enabling multiple runners in different repositories using tags

If you are using reuseable workflows in GitHub, you can only specify the name of the runs-on argument on the called workflow (in the template), not in the caller (the repo where the workflow is running). You can pass the name of the runner tag via an input param.

Called workflow in repo playground-reuseable-workflows:

name: Reusable workflow template

on:
  workflow_call:
    inputs:
      runner:
        required: true
        type: string

jobs:
  build:
    runs-on: ${{ inputs.runner }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: my-step
        run: echo "Hello World!"

Caller workflow in repo playground-reuseable-workflows-app1:

name: Call a reusable workflow

on:
  push:
  workflow_dispatch:

jobs:
  call-workflow:
    uses: TODO/playground-reuseable-workflows/.github/workflows/build_app.yml@main
    with:
      runner: playground-reuseable-workflows-app1

Then you add more runners, one for each app repository - so your entire setup with three runners for three apps will look like:

GITHUB_PAT=TODO

# The Actions Runner Controller
NAMESPACE="arc-systems"
helm install arc \
    --namespace "${NAMESPACE}" \
    --create-namespace \
    oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller

# The first runner group
NAMESPACE="playground-reuseable-workflows-app1"
INSTALLATION_NAME="playground-reuseable-workflows-app1"
GITHUB_CONFIG_URL="https://github.com/TODO/playground-reuseable-workflows"
helm install "${INSTALLATION_NAME}" \
    --namespace "${NAMESPACE}" \
    --create-namespace \
    --set githubConfigUrl="${GITHUB_CONFIG_URL}" \
    --set githubConfigSecret.github_token="${GITHUB_PAT}" \
    oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set

# The second runner group
NAMESPACE="playground-ephemeral-runners"
INSTALLATION_NAME="playground-ephemeral-runners"
GITHUB_CONFIG_URL="https://github.com/TODO/playground-ephemeral-runners"
helm upgrade --install "${INSTALLATION_NAME}" \
    --namespace "${NAMESPACE}" \
    --create-namespace \
    --set githubConfigUrl="${GITHUB_CONFIG_URL}" \
    --set githubConfigSecret.github_token="${GITHUB_PAT}" \
    oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set

# The third runner group
NAMESPACE="playground-reuseable-workflows"
INSTALLATION_NAME="playground-reuseable-workflows"
GITHUB_CONFIG_URL="https://github.com/TODO/playground-reuseable-workflows"
helm upgrade --install "${INSTALLATION_NAME}" \
    --namespace "${NAMESPACE}" \
    --create-namespace \
    --set githubConfigUrl="${GITHUB_CONFIG_URL}" \
    --set githubConfigSecret.github_token="${GITHUB_PAT}" \
    oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set

Useful links