Google Cloud Storage as an object storage source or target. One connection maps to one bucket. Reads and writes go through the GCS Hadoop connector; metadata browsing and Test Connection use the GCS JSON/REST API directly.
On this page
Connection Fields
Source: GcpConnectorPlugin.ts (connector type GCP_GCS).
| Field | Required | Default | Notes |
|---|---|---|---|
gcp_project_id | Yes | — | The GCP project the bucket lives in. |
auth_method | Yes | — | SERVICE_ACCOUNT_KEY · WORKLOAD_IDENTITY · APPLICATION_DEFAULT. Recommended: WORKLOAD_IDENTITY. |
service_account_email | For WORKLOAD_IDENTITY | — | The service account whose identity is impersonated via the compute metadata server. |
gcp_region | No | — | Default region, used for regional operations. |
storage_bucket | Yes | — | One GCS connection maps to exactly one bucket. |
storage_base_path | No | — | Optional path prefix inside the bucket, applied to every read/write on this connection. |
service_account_key_json (secret) | For SERVICE_ACCOUNT_KEY | — | Full service account key JSON. Stored encrypted. |
Authentication Methods
| Method | Behaviour |
|---|---|
SERVICE_ACCOUNT_KEY | JWT-bearer token exchange against oauth2.googleapis.com/token using the stored key JSON. |
WORKLOAD_IDENTITY | Resolves a token from the GCE/GKE/Dataproc metadata server. Only works when the DataKnits backend itself is running on GCP compute — fails fast with a clear error otherwise. |
APPLICATION_DEFAULT | Reads the operator's own gcloud auth application-default login session on the backend host. Intended for local or on-premises backend deployments, not production. |
Required IAM Roles
Grant the connection's identity one of the following on the target bucket:
roles/storage.objectViewer— read-only pipelines (source node).roles/storage.objectAdmin— read and write pipelines (source or target node).
Test Connection Behaviour
Test Connection performs a real LIST gs://{bucket}/ call. On failure it reports the actual HTTP status (401/403) along with remediation text naming the missing IAM role — it is not a mocked or hardcoded pass.
Spark Read/Write Configuration
Generated PySpark jobs read and write GCS through the GCS Hadoop connector (gcs-connector-hadoop3-shaded.jar):
WORKLOAD_IDENTITYsetsfs.gs.auth.type=COMPUTE_ENGINE_SERVICE_ACCOUNTon the Spark session, so the batch runtime resolves its own credentials from the compute environment.- All other auth methods pass the service account key through as
fs.gs.auth.service.account.json.keyfile.content— the key is injected into Spark config at submit time, never embedded in the generated.pyfile.
Known Limitations
WORKLOAD_IDENTITYwill fail outright if the DataKnits backend is not itself hosted on GCP compute (GCE, GKE, or Dataproc) — there is no fallback.APPLICATION_DEFAULTdepends on a localgcloudsession on the backend host and is not a production-appropriate auth method.