Commit b60c47a4 authored by xuwang's avatar xuwang
Browse files

docs: add GCP-AUTH.md (GCP_LOGIN_METHOD, GOOGLE_APPLICATION_CREDENTIALS, CI setup)



Framework reference for GCP auth: the four GCP_LOGIN_METHOD methods
(browser/device/service/workload-identity), how CI provides
GOOGLE_APPLICATION_CREDENTIALS from Vault via .gitlab-ci.sec + gl-add-sec,
the per-env PHI key pattern, the retired legacy GCP_KEY, and troubleshooting.

Changelog: added
Co-Authored-By: default avatarClaude Opus 4.8 <noreply@anthropic.com>
parent 78cb41be
Loading
Loading
Loading
Loading
+1 −0
Original line number Diff line number Diff line
@@ -400,6 +400,7 @@ All long-form docs live in [`docs/`](docs/):
| [WORKFLOWS.md](docs/WORKFLOWS.md) | Common end-to-end workflows |
| [OTICA_TECHNICAL_REFERENCE.md](docs/OTICA_TECHNICAL_REFERENCE.md) | Internal architecture, variable contracts, extension patterns |
| [GITLAB-RELEASE.md](docs/GITLAB-RELEASE.md) | Releases & versioning module (`gitlab-release.mk`) |
| [GCP-AUTH.md](docs/GCP-AUTH.md) | GCP authentication — `GCP_LOGIN_METHOD`, `GOOGLE_APPLICATION_CREDENTIALS`, CI setup |
| [RENOVATE.md](docs/RENOVATE.md) | Automated dependency updates with Renovate |
| [GKE-IAP-AUTH.md](docs/GKE-IAP-AUTH.md) | Protecting GKE apps with Identity-Aware Proxy |
| [HEADLAMP.md](docs/HEADLAMP.md) | Headlamp Kubernetes dashboard |

docs/GCP-AUTH.md

0 → 100644
+193 −0
Original line number Diff line number Diff line
# GCP Authentication

How OTICA authenticates to Google Cloud — for `gcloud`, GKE (`kubectl`), Terraform
state, and client libraries. Auth is controlled by a single variable,
**`GCP_LOGIN_METHOD`**, and driven by [`scripts/gcp-login.sh`](../scripts/gcp-login.sh),
invoked by the `gcp-login` target in [`makefiles/gcp.mk`](../makefiles/gcp.mk).

Almost every other target that touches GCP depends on `gcp-login` (directly, or via
`gke-login``kc-config` for kube deploys, or `tf-init` for Terraform).

---

## The four methods (`GCP_LOGIN_METHOD`)

| Method | When | Credentials source |
|---|---|---|
| `browser` (default) | local dev | interactive Google SSO (`gcloud auth login --update-adc`) |
| `device` | local dev, headless/remote shell | device-code OAuth (`--no-launch-browser`) |
| `service` | **CI/CD** (irt-as / tds-as) | a **service-account key file** at `$GOOGLE_APPLICATION_CREDENTIALS` |
| `workload-identity` | **CI/CD** on GKE runners (laneweb) | GKE metadata server — **key-less** |

`gcp-login.sh` reads `auth_method="${GCP_LOGIN_METHOD:-browser}"`. If unset it defaults
to `browser`, which **hangs in CI** (no TTY) — so CI must always set `GCP_LOGIN_METHOD`
explicitly (see below).

```
make <target> ──> gcp-login ──> gcp-login.sh

        ┌──────────────┬──────────┴────────┬──────────────────┐
     browser         device             service          workload-identity
   gcloud auth     gcloud auth      activate-service-      metadata server
   login --adc     login --dev      account --key-file     (KSA→GSA binding)
                                    =$GOOGLE_APPLICATION_
                                       CREDENTIALS
```

---

## Local development

Nothing to set — `GCP_LOGIN_METHOD` defaults to `browser`. `make gcp-login` (or any
target that needs GCP) opens a Google SSO prompt for `${GCP_ACCOUNT}` (defaults to
`${USER}@stanford.edu`) and writes Application Default Credentials.

On a remote/headless shell where a browser can't open:

```bash
make gcp-login GCP_LOGIN_METHOD=device
```

**Do not** commit `GCP_LOGIN_METHOD` into a `.mk` file — a committed
`GCP_LOGIN_METHOD=browser` overrides the CI variable and makes CI hang. Set the method
via CI variables only (see below); `gcp-env.mk` should be silent on it.

---

## CI/CD — `service` (irt-as / tds-as deploy & release)

The `irt-as` group sets a single group CI variable:

```
GCP_LOGIN_METHOD = service        # applies to every project under irt-as
```

The `service` branch of `gcp-login.sh` runs:

```bash
gcloud auth activate-service-account --key-file="${GOOGLE_APPLICATION_CREDENTIALS}"
```

so it needs **`GOOGLE_APPLICATION_CREDENTIALS`** to point at a real SA key file. If the
variable is unset it falls back to `~/.config/gcloud/application_default_credentials.json`
(which doesn't exist in CI) and fails with:

```
GOOGLE_APPLICATION_CREDENTIALS=/root/.config/gcloud/application_default_credentials.json not found.
make: *** [.../gcp.mk:9: gcp-login] Error 1
```

### Providing `GOOGLE_APPLICATION_CREDENTIALS` (the key file)

`GOOGLE_APPLICATION_CREDENTIALS` is a **file-type GitLab CI variable** whose contents are
the provisioning service-account key, sourced from Vault. It is **declared in the repo's
`.gitlab-ci.sec`** and created with `make gl-add-sec`:

```
# .gitlab-ci.sec  (rendered by render.sh; #!envsubst on line 1)
GOOGLE_APPLICATION_CREDENTIALS   vault://${GCP_KEY_PATH}   --form variable_type=file
```

- `GCP_KEY_PATH` resolves (in `gcp-env.mk`) to `${SEC_PATH}/common/gcp-provision`.
- `SEC_PATH` is `secret/projects/<project>` for non-PHI, or
  `secret/phi-projects/<project>` for PHI.
- GitLab materializes a **file** var into a temp file at job time and sets
  `GOOGLE_APPLICATION_CREDENTIALS` to its path — exactly what `gcp-login.sh` expects.

### Applying secrets: `make gl-add-sec`

```bash
# from a repo whose common/makefile.mk includes makefiles/gitlab.mk and sets GITLAB_REPO
make gl-add-sec            # reads .gitlab-ci.sec, resolves vault://, creates the CI vars via glab
```

Requirements for `gl-add-sec` to work in a repo:
1. `common/makefile.mk` **includes `${OTICA_DIR}/makefiles/gitlab.mk`**.
2. **`GITLAB_REPO`** is set (e.g. `GITLAB_REPO=irt-as/runtimes/<subgroup>/deploy-<app>`).
   Use the **literal** path — `irt-as/runtimes/${APP}/deploy-${APP}` is only correct when
   the subgroup equals `${APP}` (it does for `faculty-billets`, not for `gst-web`).
3. A valid Vault token (`make vault-login`) and `glab` authenticated to the GitLab host.

`make gl-rm-sec` removes the variables again.

---

## CI/CD — per-env keys (multi-project / PHI)

Some deploy repos deploy the **same app to different GCP projects per environment** — most
importantly the PHI pattern: `dev`/`uat` → non-PHI cluster, `stage`/`prod`**PHI**
cluster (`som-as-frapp-phi-prod`). A single key can't serve both, and a non-PHI key must
never reach a PHI cluster.

These repos use **per-env** variables and select one per environment in `.gitlab-ci.yml`:

```yaml
.deploy_dev:   { variables: { TARGET_GCP_ENVIRONMENT: dev,   GOOGLE_APPLICATION_CREDENTIALS: $DEV_GOOGLE_APPLICATION_CREDENTIALS } }
.deploy_uat:   { variables: { TARGET_GCP_ENVIRONMENT: uat,   GOOGLE_APPLICATION_CREDENTIALS: $UAT_GOOGLE_APPLICATION_CREDENTIALS } }
.deploy_stage: { variables: { TARGET_GCP_ENVIRONMENT: stage, GOOGLE_APPLICATION_CREDENTIALS: $STAGE_GOOGLE_APPLICATION_CREDENTIALS } }   # PHI
.deploy_prod:  { variables: { TARGET_GCP_ENVIRONMENT: prod,  GOOGLE_APPLICATION_CREDENTIALS: $PROD_GOOGLE_APPLICATION_CREDENTIALS } }     # PHI
```

The `<ENV>_GOOGLE_APPLICATION_CREDENTIALS` file vars are declared once in `.gitlab-ci.sec`
and created by running `gl-add-sec` **per environment**, so each env resolves its own
`GCP_KEY_PATH`:

```
${APP_ENVIRONMENT}_GOOGLE_APPLICATION_CREDENTIALS   vault://${GCP_KEY_PATH}   --form variable_type=file
```

- `dev`/`uat` resolve `GCP_KEY_PATH` under `secret/projects/...` (non-PHI key).
- `stage`/`prod` resolve `GCP_KEY_PATH` under `secret/phi-projects/...` (**PHI key**).

> **PHI rule:** never create a stage/prod GAC from the non-PHI vault, and never point a
> non-PHI env at the PHI key. The per-env split is what keeps the boundary intact.

---

## CI/CD — Workload Identity (laneweb)

On GKE-hosted runners whose Kubernetes SA is bound to a Google SA, set:

```
GCP_LOGIN_METHOD = workload-identity
```

`gcp-login.sh` reads the pod's identity from the metadata server and proceeds **key-less**
(no `GOOGLE_APPLICATION_CREDENTIALS`, no Vault key). It is **never auto-detected** — every
GKE pod has *some* metadata identity, so WI must be requested explicitly; if the bound
identity isn't a real `name@PROJECT.iam.gserviceaccount.com` the script fails loudly rather
than run as an unintended SA.

This is the model used by `som-laneweb` (runner GSAs hold the needed roles); `irt-as`
deploys use `service` instead.

---

## Legacy `GCP_KEY` (removed)

`GCP_KEY` was a **ps-cloud-framework** CI variable (env-var type) holding the SA key
contents, consumed by the old `config-gcloud` target. **OTICA does not use it**
`gcp-login.sh` reads only `GOOGLE_APPLICATION_CREDENTIALS`. Leftover `GCP_KEY` variables
were pruned from the migrated deploy repos; do not reintroduce it.

Do not confuse it with the live vars `GCP_KEY_PATH` / `GCP_KEY_FILE` in `gcp-env.mk`
those are the **Vault path** and local cache path for the key, and are still used.

---

## Troubleshooting

| Symptom | Cause | Fix |
|---|---|---|
| `GOOGLE_APPLICATION_CREDENTIALS=…/application_default_credentials.json not found` in CI | `service` method but the GAC file var isn't set | `make gl-add-sec` to create it from `.gitlab-ci.sec` |
| CI job hangs at `gcp-login` | `GCP_LOGIN_METHOD` unset → defaults to `browser` | set `GCP_LOGIN_METHOD=service` (group var) or `workload-identity` |
| `ERROR: GCP_LOGIN_METHOD=workload-identity but metadata SA is '…', not a bound GSA` | runner KSA has no GSA binding | bind the KSA→GSA, or use `service` |
| `make gl-add-sec``No rule to make target 'gl-add-sec'` | `gitlab.mk` not included | add `include ${OTICA_DIR}/makefiles/gitlab.mk` to `common/makefile.mk` |
| `gl-add-sec``Error: GITLAB_REPO … is not set` | `GITLAB_REPO` missing | set the literal `GITLAB_REPO` in `env.mk` |
| local `make` picks the wrong account | stale ADC | `make gcp-login` (browser) or `make gcp-app-login` |

## See also

- [`makefiles/gcp.mk`](../makefiles/gcp.mk) · [`scripts/gcp-login.sh`](../scripts/gcp-login.sh)
- [GITLAB-RELEASE.md](GITLAB-RELEASE.md) — releases & `update-downstream-tag`
- [GKE-IAP-AUTH.md](GKE-IAP-AUTH.md) — end-user auth for GKE apps (IAP)