Kubernetes

You can use Phase to sync secrets with applications running in your Kubernetes cluster.

Prerequisites

• Sign up for the Phase Console and create an application.
• Obtain a PHASE_SERVICE_TOKEN for your application, that has access to the environment you are trying to fetch secrets from.

Using the Phase Kubernetes Operator

This section provides a step-by-step guide to help you get started with the Phase Kubernetes Operator. Feel free to adjust the provided options and configurations to suit your specific needs.

1. Install the Operator via Helm

Add the Phase Helm repository and update it:

helm repo add phase https://helm.phase.dev && helm repo update

Install the Phase Secrets Operator:

helm install phase-secrets-operator phase/phase-kubernetes-operator --set image.tag=v1.4.1

Too see the full set of configurable Helm chart options, please see: values.yaml

2. Create a Service Token Secret in Kubernetes

kubectl create secret generic phase-service-token \
  --from-literal=token=<TOKEN> \
  --type=Opaque \
  --namespace=default

3. Deploy the Phase Secrets Operator CR (Custom Resource)

Create a custom resource file: phase-secrets-operator-cr.yaml

A Basic custom resource to sync all secrets from a production environment in an App in the Phase Console to your Kubernetes cluster as my-application-secret, type Opaque.

apiVersion: secrets.phase.dev/v1alpha1
kind: PhaseSecret
metadata:
  name: example-phase-secret
  namespace: default
spec:
  phaseApp: 'your-application-name' # The name of your Phase application
  phaseAppEnv: 'production' # OPTIONAL - The Phase App Environment to fetch secrets from
  phaseAppEnvPath: '/' # OPTIONAL Path within the Phase application environment to fetch secrets from
  phaseHost: 'https://console.phase.dev' # OPTIONAL - URL of a Phase Console instance
  pollingInterval: 5 # OPTIONAL - Interval in seconds to poll for secret updates
  authentication:
    serviceToken:
      serviceTokenSecretReference:
        secretName: 'phase-service-token' # Name of the Phase Service Token with access to your application
        secretNamespace: 'default'
  managedSecretReferences:
    - secretName: 'my-application-secret' # Name of the Kubernetes managed secret that Phase will sync
      secretNamespace: 'default'

Deploy the custom resource:

kubectl apply -f phase-secrets-operator-cr.yaml

Watch for my-application-secret managed secret being created:

watch kubectl get secrets

View the secrets:

kubectl get secret my-application-secret -o yaml

Properties

Secret processors:

Secret key name transformation

The nameTransformer option transforms secret key names from UPPER_SNAKE_CASE to a target format. It can be set at two levels:

1. Secret-level — applies to all keys in the managed secret.
2. Per-key — set within processors for a specific key. Takes precedence over the secret-level transformer.

If both asName and nameTransformer are set on the same key, asName takes precedence.

Example: Transforming all keys to camelCase

managedSecretReferences:
  - secretName: 'my-app-config'
    secretNamespace: 'default'
    nameTransformer: 'camel'

Example: Mixed transformations with per-key overrides

managedSecretReferences:
  - secretName: 'my-app-config'
    secretNamespace: 'default'
    nameTransformer: 'camel'
    processors:
      DATABASE_HOST:
        nameTransformer: upper-camel
      STRIPE_SECRET_KEY:
        asName: stripeKey

Available transformations

Secret value transformation

The processors field can also transform secret values during ingestion:

• asName: Renames the secret key. For instance, PKCS12_PRIVATE_KEY is renamed to tls.crt.
• type: Specifies the encoding type of the secret. base64 indicates that the secret is already base64 encoded. If a secret is set to type: "plain", which is the default, the Kubernetes operator will encode it in base64.

Example: PKCS12 Secret Transformation

Suppose we have the following secrets stored in the Phase service:

1. PKCS12_PRIVATE_KEY: A private key in PKCS12 format, already base64 encoded.
2. PKCS12_CERTIFICATE: A certificate in PKCS12 format, also already base64 encoded.

In the custom resource, these are handled as follows:

apiVersion: secrets.phase.dev/v1alpha1
kind: PhaseSecret
metadata:
  name: example-phase-secret
  namespace: default
spec:
  phaseApp: 'your-application-name'
  phaseAppEnv: 'production'
  phaseAppEnvTag: 'certs'
  phaseHost: 'https://console.phase.dev'
  pollingInterval: 5
  authentication:
    serviceToken:
      serviceTokenSecretReference:
        secretName: 'phase-service-token'
        secretNamespace: 'default'
  managedSecretReferences:
    - secretName: 'pkcs12-cert'
      secretNamespace: 'default'
      secretType: 'kubernetes.io/tls'
      processors:
        PKCS12_PRIVATE_KEY:
          asName: 'tls.crt'
          type: 'base64'
        PKCS12_CERTIFICATE:
          asName: 'tls.key'
          type: 'base64'

Using secrets in Kubernetes Deployments

1. Using Environment Variables

Expose specific secret(s) from your Kubernetes secret to your deployment as environment variables:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-app
          image: my-app-image
          env:
            - name: MY_SECRET_KEY
              valueFrom:
                secretKeyRef:
                  name: my-application-secret
                  key: MY_SECRET_KEY

2. Using envFrom to inject all secrets

Expose all secrets in your Kubernetes secrets to your deployment as environment variables:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-app
          image: my-app-image
          envFrom:
            - secretRef:
                name: my-application-secret

3. Mounting secrets as an in-memory volume

Mount secrets as an in-memory volume for scenarios requiring configuration files:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-app
          image: my-app-image
          volumeMounts:
            - name: secret-volume
              mountPath: /etc/secrets
              readOnly: true
      volumes:
        - name: secret-volume
          secret:
            secretName: my-application-secret

4. Auto-Redeploying Deployments When Secrets Change

To ensure that deployments are automatically redeployed when their associated secrets change, add the secrets.phase.dev/redeploy annotation with a value of "true" to your deployment. This instructs the Phase Kubernetes Operator to redeploy the application whenever the specified secrets are updated.

Here's an example deployment manifest incorporating this approach:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-autoredeploy
  annotations:
    secrets.phase.dev/redeploy: 'true'
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app-autoredeploy
  template:
    metadata:
      labels:
        app: my-app-autoredeploy
    spec:
      containers:
        - name: my-app
          image: my-app-image
          envFrom:
            - secretRef:
                name: my-application-secret

When Operator Triggers Redeployment

The operator triggers redeployment in the following case:

1. Secret Change Detected: When the operator updates a Kubernetes secret (either creates a new one, updates an existing one, or deletes and recreates one due to type change), it checks for deployments in the same namespace that have the REDEPLOY_ANNOTATION set.

2. Annotation Present on Deployment: If a deployment has the annotation secrets.phase.dev/redeploy (the value of REDEPLOY_ANNOTATION), it indicates that this deployment is reliant on the secret(s) being managed by the operator.

3. Patching the Deployment for Redeployment: When such a deployment is found, the operator triggers a redeployment by patching the deployment. This is typically done by updating an annotation on the deployment's pod template, which results in Kubernetes redeploying the pods.

When Operator Does Not Trigger Redeployment

The operator does not trigger redeployment in the following cases:

1. No Secret Change: If there are no changes in the secrets (i.e., the fetched secrets from Phase service are the same as what's already in Kubernetes), the operator does not perform any secret update actions, and consequently, no redeployment is triggered.

2. Deployment Lacks Annotation: If a deployment in the namespace does not have the secrets.phase.dev/redeploy annotation, it will not be considered for redeployment by the operator, even if it uses the secrets being managed.

3. Secret Change Not Affecting Deployments: If the updated secret is not used by any deployment or if the deployments using the secret do not rely on the operator for updates (i.e., they do not have the redeploy annotation), those deployments will not be redeployed.

4. Errors During Processing: If there's an error in processing the secrets (e.g., fetching from Phase service, processing, or updating in Kubernetes), and as a result, the secret update doesn’t happen, then no redeployment will be triggered.

Security Considerations

• Least Privilege: Ensure applications have access only to the secrets they need.
• RBAC Policies: Implement Role-Based Access Control policies to restrict access to secrets.

Troubleshooting and Debugging the Phase Kubernetes Operator

Check Operator Logs

View logs for potential errors or misconfigurations:

kubectl get pods

kubectl logs <operator-pod-name>

Example (revoked Service Token):

λ kubectl logs phase-secrets-operator-phase-kubernetes-operator-8b69db6f-f4m8s -f
[2023-11-20 10:54:07,948] kopf._core.engines.a [INFO    ] Initial authentication has been initiated.
[2023-11-20 10:54:07,950] kopf.activities.auth [INFO    ] Activity 'login_via_client' succeeded.
[2023-11-20 10:54:07,951] kopf._core.engines.a [INFO    ] Initial authentication has finished.
[2023-11-20 10:54:08,047] kopf._core.reactor.o [WARNING ] Not enough permissions to watch for resources: changes (creation/deletion/updates) will not be noticed; the resources are only refreshed on operator restarts.
[2023-11-20 10:54:08,692] kopf._core.reactor.r [WARNING ] Cleanup activity is not executed at all due to cancellation.

🚫 Not authorized. Token expired or revoked.
Failed to fetch secrets: The environment 'dev' either does not exist or you do not have access to it.

Check Sync Status

The Phase Kubernetes Operator only initiates a secret sync when a change is made in your source environment in Phase. The operator will periodically check for changes and store the timestamp of the last sync in the following format {namespace}:{cr_name}:{cr_uid} at /tmp/phase_sync_status.json location. This file is will persist across operator restarts. You may choose to delete this file to manually force a full sync.

View the sync status to see if the operator is trying to sync secrets that are not present in the Phase Console:

cat /tmp/phase_sync_status.json

Example output:

{
  "default:example-phase-secret:a6244420-a712-4ac2-b9c2-eae80ea5cdba": "2025-08-13T09:39:21.940863+00:00"
}

Verify Operator Status

Ensure the operator is running correctly:

kubectl get deployment <operator-deployment-name> -n <operator-namespace>

Inspect the Custom Resource (CR)

Check for correct configuration:

kubectl get phasesecret example-phase-secret -n default -o yaml

Check Events

View Kubernetes events for a timeline of cluster activities:

kubectl describe phasesecret example-phase-secret -n default

Review Kubernetes Secrets

Verify the presence and content of synced secrets:

kubectl get secret my-application-secret -n default -o yaml

Examine RBAC Settings

Ensure appropriate permissions are set:

kubectl get clusterrole <operator-cluster-role>
kubectl get clusterrolebinding <operator-cluster-rolebinding>

Operator Version and Updates

Compare the deployed version with the latest available version:

helm list -n <operator-namespace>

Resource Limits and Quotas

Check for any resource constraints:

kubectl describe deployment <operator-deployment-name> -n <operator-namespace>

Uninstall the Operator

Remove the operator using Helm:

helm uninstall phase-secrets-operator

Using Init Container

You can easily integrate with your kubernetes workload and inject secrets securely using in memory init containers.

1. Create PHASE_SERVICE_TOKEN Kubernetes secret secret.

kubectl create secret generic phase-service-token --from-literal=PHASE_SERVICE_TOKEN=<TOKEN> --namespace=default

2. Here's an example deploy.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      volumes:
        - name: secrets-volume
          emptyDir:
            medium: Memory
      initContainers:
        - name: fetch-secrets
          image: phasehq/cli:latest
          volumeMounts:
            - name: secrets-volume
              mountPath: /secrets
          env:
            - name: PHASE_SERVICE_TOKEN
              valueFrom:
                secretKeyRef:
                  name: phase-service-token
                  key: PHASE_SERVICE_TOKEN
          command:
            [
              '/bin/sh',
              '-c',
              'phase secrets export --app "my application name" --env development > /secrets/secrets.env',
            ]
      containers:
        - name: main-container
          image: alpine:latest
          volumeMounts:
            - name: secrets-volume
              mountPath: /secrets
          command: ['/bin/sh', '-c', 'cat /secrets/secrets.env && sleep 3600']

Security Benefits

1. Isolation of Responsibilities: Init containers allow you to separate the responsibility of setting up the environment from the primary purpose of the main container. By using init containers solely to fetch secrets and populate in-memory volumes, you reduce the surface area for potential attacks targeted at the main application.

2. Transient Nature: By using in-memory volumes, the data doesn't persist beyond the lifecycle of the pod. When the pod is terminated, the data is lost. This reduces the risk of secret data being left behind.

3. No Persistent Storage: Since secrets fetched and placed in an in-memory volume aren't written to any persistent storage, there's no risk of them being exposed if the underlying storage is compromised or improperly decommissioned.

Security Considerations

1. In-Memory Exposure: While the secrets are not persisted to disk, they are still available in the node's RAM. A process or user with sufficient privileges on the node could potentially access data directly from memory.

2. Swap Space: If the Kubernetes node is configured with swap, and the node starts swapping memory contents to disk, in-memory data (including secrets) might end up being written to swap space. It's recommended to disable swap on Kubernetes nodes for this (and other) reasons.

3. Secret Versioning: If you update a secret, the change won't be automatically propagated to running pods. The pods need to be restarted to pick up the new version of the secret.