Merge branch 'add-crashloop-reporter-skill' into 'main'

---

name: otica-crashloop-reporter

description: Install a cluster-wide CrashLoopBackOff -> Slack reporter into an OTICA kube-* maintenance repo. Deploys a CronJob (+ minimal cluster-wide pod-read RBAC) that scans every namespace for pods stuck in CrashLoopBackOff/ImagePullBackOff/ErrImagePull and posts them to a Slack channel (quiet when clean). Use when a team wants proactive alerting for stuck pods.

---

# Install the CrashLoopBackOff → Slack reporter

Adds a self-contained "stuck pod" reporter to an OTICA **kube-* maintenance repo** (one

deployed per cluster you want to watch — e.g. a `kube-sys-maint`). It scans ALL namespaces

and Slacks findings; deliberately quiet when nothing is stuck.

Bundled artifacts live next to this file in `files/`:

- `crashloopbackoff.sh` — the scanner (read-only; `kubectl get pods -A` + `jq`, posts to Slack)

- `crashloop-report.yml` — CronJob + ServiceAccount + ClusterRole + ClusterRoleBinding

- `secrets-slack.yml` — the `slack-hook` Secret rendered from Vault

## How it's invoked

The user runs this from **inside the target repo's env dir** with the cluster context

already prepared:

```bash

cd <kube-maint-repo>/<env>       # e.g. kube-sys-maint/prod

make kc-config                   # selects the cluster + namespace + credentials

# then invoke this skill: "install the crashloop reporter"

```

Given that, this skill copies the bundled `files/` into the repo (Steps 1–4), wires the two

`env.mk` vars, and deploys to the **current** cluster (Step 5). Repeat per cluster you want to

watch (one maintenance repo/env per cluster). Confirm the repo + target cluster from

`make kc-config` output before applying — you're adding cluster-wide RBAC.

## What it deploys

- **CronJob** (default every 15 min) → runs `/scripts/crashloopbackoff.sh`.

- A dedicated **ServiceAccount** + **ClusterRole** (`get`/`list` **pods**, all namespaces) + **ClusterRoleBinding** — least privilege, no `cluster-info` needed.

- Slacks only when something is stuck; `ALWAYS_NOTIFY=true` posts an all-clear too.

## Prerequisites (verify first)

1. An **OTICA kube-* repo** whose `make kc-config` selects the target cluster, and whose

   `common/env.mk` defines `APP_NAMESPACE` + `APP_IMAGE`.

2. `APP_IMAGE` must have **kubectl + jq + curl**. The OTICA public gcloud image works:

   `us-docker.pkg.dev/uit-services/docker-public/gcloud:latest`.

3. A **Slack incoming webhook stored in Vault** (any dir/key you control).

4. The repo ships `common/scripts/` into a **`scripts-sh` configmap** (`make scripts-sh` —

   standard OTICA target); the CronJob mounts it at `/scripts`. (Alternative: bake the

   script into the image and drop the volume.)

5. The cluster/namespace deployer can create **ClusterRole/ClusterRoleBinding** (cluster-scoped).

## Step 1 — add the scanner

Copy `files/crashloopbackoff.sh` → `common/scripts/crashloopbackoff.sh` (keep it executable).

No edits needed; behavior is env-var driven (see Customize).

## Step 2 — add the Slack secret + point it at Vault

Copy `files/secrets-slack.yml` → `common/templates-sec/secrets-slack.yml`, then set in

`common/env.mk`:

```makefile

SLACK_SEC_PATH=secret/projects/<project>/common/slack   # Vault dir with the webhook

SLACK_WEBHOOK_KEY=<key>                                  # the webhook's key (e.g. channel name)

```

(If a `slack-hook` Secret with key `url` already exists in the namespace, skip this and reuse it.)

## Step 3 — add the CronJob + RBAC

Copy `files/crashloop-report.yml` → `common/templates/crashloop-report.yml`. It references

`{{.Env.APP_NAMESPACE}}` + `{{.Env.APP_IMAGE}}` (already in any OTICA kube repo) and the

`slack-hook` secret from Step 2. Adjust the `schedule:` if you want something other than

`*/15 * * * *`.

## Step 4 — (optional) an on-demand make target

Add to the repo's `common/makefile.mk` for manual runs:

```makefile

.PHONY: crashloop-now

crashloop-now: kc-config ## run the crashloop reporter once now (ALWAYS_NOTIFY=true to force a Slack post)

	@kubectl delete job crashloop-now -n ${APP_NAMESPACE} --ignore-not-found=true

	@kubectl create job crashloop-now -n ${APP_NAMESPACE} --from=cronjob/crashloop-report --dry-run=client -o yaml \

		| kubectl set env --local -f - ALWAYS_NOTIFY=${ALWAYS_NOTIFY} -o yaml \

		| kubectl apply -f -

	@echo "Started crashloop-now (ALWAYS_NOTIFY=${ALWAYS_NOTIFY}). Logs: kubectl logs -f job/crashloop-now -n ${APP_NAMESPACE}"

```

## Step 5 — deploy

```bash

make kc-config          # select the target cluster + namespace

make scripts-sh         # ship crashloopbackoff.sh into the scripts-sh configmap

make kc-apply-sec       # render/apply the slack-hook secret (needs Vault login)

make deploy             # or: make kc-apply — applies RBAC + CronJob

```

## Step 6 — verify

```bash

make crashloop-now ALWAYS_NOTIFY=true          # forces a Slack post so you can confirm the webhook

kubectl -n <APP_NAMESPACE> logs -f job/crashloop-now

```

Expect a Slack message and, in the log, either a `:white_check_mark:` all-clear or a

`:rotating_light:` list of stuck pods.

## Customize (CronJob `env:` or per-run)

- `REASONS` — waiting reasons to flag (default `CrashLoopBackOff ImagePullBackOff ErrImagePull`).

- `ALWAYS_NOTIFY=true` — also post when nothing is stuck (default: quiet).

- `CLUSTER_NAME` — label in the Slack message (default: the CronJob sets it from `GCP_ENVIRONMENT`).

- `schedule:` — change the cron cadence.

## Gotchas

- **Reachability check uses the real `kubectl get pods -A`**, NOT `kubectl cluster-info` —

  `cluster-info` reads `kube-system` services, which the pod-read-only ClusterRole does not

  grant, so it would fail even when the API is fine.

- The **ClusterRole/ClusterRoleBinding names are fixed** (`crashloop-pod-reader` /

  `crashloop-reporter-pod-reader`). Separate clusters don't conflict; don't install twice in

  the same cluster.

- Anonymous/limited `kubectl` can't enumerate pods cluster-wide — the ServiceAccount's

  ClusterRole is what makes the scan work; keep it (get/list pods only).

- Slack masking: the webhook is a Secret, never printed by the script.

#!gomplate

# Cluster-wide CrashLoopBackOff reporter: a CronJob that runs

# /scripts/crashloopbackoff.sh (from the scripts-sh configmap) as a dedicated

# ServiceAccount granted read-only get/list on pods across ALL namespaces, and

# posts findings to Slack (webhook from the slack-hook secret).

#

# Deploy: make scripts-sh   (ships the script into the scripts-sh configmap)

#         make deploy        (renders this + the slack-hook secret, applies RBAC+cron)

---

apiVersion: v1

kind: ServiceAccount

metadata:

  name: crashloop-reporter

  namespace: "{{.Env.APP_NAMESPACE}}"

---

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

  name: crashloop-pod-reader

rules:

  - apiGroups: [""]

    resources: ["pods"]

    verbs: ["get", "list"]

---

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

  name: crashloop-reporter-pod-reader

roleRef:

  apiGroup: rbac.authorization.k8s.io

  kind: ClusterRole

  name: crashloop-pod-reader

subjects:

  - kind: ServiceAccount

    name: crashloop-reporter

    namespace: "{{.Env.APP_NAMESPACE}}"

---

apiVersion: batch/v1

kind: CronJob

metadata:

  name: crashloop-report

  namespace: "{{.Env.APP_NAMESPACE}}"

spec:

  # Every 15 min. CrashLoopBackOff is a sustained state, so polling catches it.

  schedule: "*/15 * * * *"

  concurrencyPolicy: "Forbid"

  successfulJobsHistoryLimit: 1

  failedJobsHistoryLimit: 2

  jobTemplate:

    spec:

      activeDeadlineSeconds: 300

      ttlSecondsAfterFinished: 3600

      template:

        spec:

          serviceAccountName: crashloop-reporter

          restartPolicy: Never

          containers:

            - name: crashloop-report

              image: "{{.Env.APP_IMAGE}}"

              imagePullPolicy: Always

              command: ["/scripts/crashloopbackoff.sh"]

              env:

                - name: CLUSTER_NAME

                  value: "{{ getenv "GCP_ENVIRONMENT" "cluster" }}"

                # ALWAYS_NOTIFY=true would also post an all-clear each run; default

                # is quiet (only alert when something is stuck).

                - name: ALWAYS_NOTIFY

                  value: "false"

                - name: SLACK_HOOK

                  valueFrom:

                    secretKeyRef:

                      name: slack-hook

                      key: url

              resources:

                requests: { cpu: "50m", memory: "64Mi" }

                limits:   { cpu: "500m", memory: "256Mi" }

              volumeMounts:

                - name: scripts-sh

                  mountPath: /scripts

          volumes:

            - name: scripts-sh

              configMap:

                name: scripts-sh

                defaultMode: 0755

#!/bin/bash

#

# crashloopbackoff.sh — cheap, cluster-wide "stuck pod" reporter.

#

# Scans ALL namespaces for containers stuck in CrashLoopBackOff (and, by default,

# the image-pull failures that look the same to an operator), prints a summary,

# and posts it to Slack. Runs in-cluster from a CronJob (uses the pod's

# ServiceAccount, which needs cluster-wide get/list pods — see crashloop-report.yml)

# or ad-hoc from a workstation (uses your current kube-context).

#

# READ-ONLY: only `kubectl get pods`. By default it Slacks ONLY when something is

# found (no news = no noise); set ALWAYS_NOTIFY=true to also post an all-clear.

#

# Env:

#   SLACK_HOOK      Slack incoming-webhook URL. If unset, prints to stdout only

#                   (Vault: secret/projects/uit-authnz/common/slack/authnz-alerts).

#   REASONS         space-separated container waiting-reasons to flag

#                   (default: "CrashLoopBackOff ImagePullBackOff ErrImagePull").

#   ALWAYS_NOTIFY   "true" to post even when nothing is found (default: false).

#   CLUSTER_NAME    label used in the message (default: current kube-context).

#

# Exit: 0 = clean run (found or not); 2 = no cluster reachable.

set -uo pipefail

REASONS="${REASONS:-CrashLoopBackOff ImagePullBackOff ErrImagePull}"

CLUSTER_NAME="${CLUSTER_NAME:-$(kubectl config current-context 2>/dev/null || echo cluster)}"

reason_re="$(echo "${REASONS}" | tr -s ' ' '|')"

# The real query doubles as the reachability/permission check. (Don't use

# `kubectl cluster-info` as a preflight: it reads kube-system services, which the

# pod-read-only ClusterRole doesn't grant, so it fails even when the API is fine.)

if ! pods_json="$(kubectl get pods -A -o json 2>/tmp/crashloop_kubeerr)"; then

  echo "ERROR: cannot list pods cluster-wide — $(cat /tmp/crashloop_kubeerr 2>/dev/null)" >&2

  echo "       Set the context ('make kc-config') or check the pod's RBAC." >&2

  exit 2

fi

# One pass over every container in the cluster: emit those Waiting with a flagged

# reason. Columns: NAMESPACE  POD  CONTAINER  REASON  RESTARTS.

rows="$(printf '%s' "${pods_json}" | jq -r --arg re "^(${reason_re})$" '

  .items[] as $p

  | ($p.status.containerStatuses // [])[]

  | select((.state.waiting.reason // "") | test($re))

  | [ $p.metadata.namespace, $p.metadata.name, .name,

      .state.waiting.reason, (.restartCount|tostring) ] | @tsv

')"

if [[ -z "${rows}" ]]; then

  echo "OK: nothing in [${REASONS}] on ${CLUSTER_NAME}"

  [[ "${ALWAYS_NOTIFY:-}" == "true" ]] || exit 0

  text=":white_check_mark: *${CLUSTER_NAME}* — no stuck pods (${REASONS// /, })"

else

  count="$(printf '%s\n' "${rows}" | wc -l | tr -d ' ')"

  echo "FOUND ${count} stuck container(s) on ${CLUSTER_NAME}:"

  { printf 'NAMESPACE\tPOD\tCONTAINER\tREASON\tRESTARTS\n'; printf '%s\n' "${rows}"; } \

    | { column -t -s "$(printf '\t')" 2>/dev/null || cat; }

  list="$(printf '%s\n' "${rows}" \

    | awk -F'\t' '{printf "• `%s/%s` [%s] *%s* (restarts: %s)\n", $1,$2,$3,$4,$5}')"

  text=":rotating_light: *${CLUSTER_NAME}* — ${count} stuck pod container(s)"$'\n'"${list}"

fi

# Post to Slack if a webhook is configured; otherwise leave the stdout report.

if [[ -n "${SLACK_HOOK:-}" ]]; then

  payload="$(jq -n --arg t "${text}" '{blocks:[{type:"section",text:{type:"mrkdwn",text:$t}}]}')"

  if curl -sS -X POST -H 'Content-type: application/json' --data "${payload}" "${SLACK_HOOK}" >/dev/null; then

    echo "(posted to Slack)"

  else

    echo "WARN: Slack post failed" >&2

  fi

else

  echo "(SLACK_HOOK unset — stdout only)"

fi

#!gomplate

# Slack incoming-webhook for the crashloop reporter, sourced from Vault.

# Set in env.mk:

#   SLACK_SEC_PATH    Vault dir holding the webhook (e.g. secret/projects/<proj>/common/slack)

#   SLACK_WEBHOOK_KEY the webhook's key within that dir (e.g. the channel name)

# Consumed by the crashloop-report CronJob via secretKeyRef (SLACK_HOOK).

apiVersion: v1

kind: Secret

metadata:

  name: slack-hook

  namespace: "{{.Env.APP_NAMESPACE}}"

type: Opaque

data:

  url: "{{ tmpl.Exec "vault-kv" (filepath.Join .Env.SLACK_SEC_PATH .Env.SLACK_WEBHOOK_KEY) | base64.Encode }}"