GCP Troubleshooting Guide

This guide will help you troubleshoot common issues when deploying your Armonik project on Google Cloud Platform (GCP).

1. Deployment Error with New Partition

If you encounter an error during deployment after adding a new partition with the value replicas > 0, follow these steps:

1. Set the replicas to 0.
2. Redeploy the resources.
3. If the deployment succeeds, update the replicas to the desired value.
4. Redeploy the resources again.

2. Pub/Sub Client-Side Issue

If you encounter an issue with the Pub/Sub client and see the error message: FAILED_PRECONDITION: Requested entity was not found., follow the steps below to resolve the issue.

Error Message

When running the application, you may encounter the following error:

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.FAILED_PRECONDITION
    details = "Cloud Pub/Sub did not have the necessary permissions configured to support this operation.
    Please verify that the service account [SERVICE_ACCOUNT] was granted the Cloud KMS CryptoKey Encrypter/Decrypter role
    for the project containing the CryptoKey resource [PROJECT_ID]/locations/[LOCATION]/keyRings/[KEY_RING]/cryptoKeys/[CRYPTO_KEY]."
    debug_error_string = "UNKNOWN:Error received from peer {created_time:"[TIMESTAMP]",
    grpc_status:9,
    grpc_message:"Cloud Pub/Sub did not have the necessary permissions configured to support this operation.
    Please verify that the service account [SERVICE_ACCOUNT] was granted the Cloud KMS CryptoKey Encrypter/Decrypter role
    for the project containing the CryptoKey resource [PROJECT_ID]/locations/[LOCATION]/keyRings/[KEY_RING]/cryptoKeys/[CRYPTO_KEY]."}

Problem Description

This error occurs because Cloud Pub/Sub requires access to the specified CMEK to encrypt or decrypt messages. The service account used by Pub/Sub does not have the Cloud KMS CryptoKey Encrypter/Decrypter role for the specified CryptoKey.

Key Components in the Error

• Service Account: service-[NUMERIC_ID]@gcp-sa-pubsub.iam.gserviceaccount.com
• CryptoKey Resource:
  • Project: [PROJECT_ID]
  • Location: [LOCATION] (e.g., europe-west1)
  • Key Ring: [KEY_RING]
  • CryptoKey: [CRYPTO_KEY]
• Missing Role: roles/cloudkms.cryptoKeyEncrypterDecrypter

Without this role, Cloud Pub/Sub cannot perform encryption or decryption using the CMEK.

Solution

To resolve the issue, grant the Cloud KMS CryptoKey Encrypter/Decrypter role to the Pub/Sub service account for the specified CryptoKey.

Step 1: Identify the Service Account

The service account mentioned in the error typically has the format: service-[NUMERIC_ID]@gcp-sa-pubsub.iam.gserviceaccount.com. This service account is automatically created by Google Cloud to manage Pub/Sub operations.

Step 2: Grant the Necessary Role

You can grant the Cloud KMS CryptoKey Encrypter/Decrypter role to the service account using the Google Cloud Console or the gcloud command-line tool.

Using the Google Cloud Console

1. Open the Google Cloud Console.
2. Navigate to Key Management > CryptoKeys.
3. Locate the CryptoKey resource:
   • Project: [PROJECT_ID]
   • Location: [LOCATION] (e.g., europe-west1)
   • Key Ring: [KEY_RING]
   • CryptoKey: [CRYPTO_KEY]
4. Click on the CryptoKey and go to the Permissions tab.
5. Add the service account as a principal:
   • Principal: service-[NUMERIC_ID]@gcp-sa-pubsub.iam.gserviceaccount.com
   • Role: Cloud KMS CryptoKey Encrypter/Decrypter
6. Save the changes.

After granting the role, Cloud Pub/Sub should be able to access the specified CMEK for encryption and decryption operations. Retry the operation that triggered the error. Ensure that the deployment is successful and verify that the Pub/Sub client can now access the CMEK without any issues.