Filebeat Google Cloud Storage Input requires JSON or file credentials for authentication which causes issues when using workload identity

[bkaznowski]

Describe the enhancement:
Make auth.credentials_json.account_key and auth.credentials_file.path optional so you can configure filebeat to set up the Google Storage client without either option.

Describe a specific use case for the enhancement or feature:
If you are running Filebeat in a Kubernetes cluster and want to use workload identity to authenticate with GCS then the documentation doesn't make it clear how to achieve this and the behaviour would suggest this isn't possible. It would be nice to either specifically call out the workaround in the documentation or to make both config options optional.

This is the guide I was following which says you must provide either a credentials JSON or a credentials file. However, workload identity doesn't require either.

Existing workaround:
There is a workaround that exists for now, which is to set an empty credentials file path. For example, auth.credentials_file.path: "". This makes filebeat think the credentials file path is set and the Google Storage client will set this path. However, the GCS client treats an empty path identically to the scenario where a path isn't provided.

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[andrewkroh]

I think the same logic used by the GCP Pub/Sub input could be implemented by the GCS input. If none of the config options are set then it checks if it can load application default credentials before failing.

beats/x-pack/filebeat/input/gcppubsub/config.go

Lines 45 to 68 in d2c4c2d

	func (c *config) Validate() error {
	// credentials_file
	if c.CredentialsFile != "" {
	if _, err := os.Stat(c.CredentialsFile); os.IsNotExist(err) {
	return fmt.Errorf("credentials_file is configured, but the file %q cannot be found", c.CredentialsFile)
	} else {
	return nil
	}
	}
	// credentials_json
	if len(c.CredentialsJSON) > 0 {
	return nil
	}
	// Application Default Credentials (ADC)
	ctx := context.Background()
	if _, err := google.FindDefaultCredentials(ctx, pubsub.ScopePubSub); err == nil {
	return nil
	}
	return fmt.Errorf("no authentication credentials were configured or detected " +
	"(credentials_file, credentials_json, and application default credentials (ADC))")
	}

Does that sound correct to you @bkaznowski?

[bkaznowski]

Yes, I believe this sounds right. We use the gcp-pubsub input too and we haven't had any issues with using it alongside workload identity. It would also be a nice solution to bring both of these under the same behaviour

[alelevinas]

Hi @efd6

I'm wondering how you are setting up Workload Identity because I'm having the same issue here.
I can authenticate correctly with GCS on filebeat 8.10 and auth.credentials_file.path: "" but when upgrading to 8.15 (and removing that line) I get

{"log.level":"error","@timestamp":"2024-08-29T16:39:43.768Z","log.logger":"input.gcs","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/v2/compat.(*runner).Start.func1","file.name":"compat/compat.go","file.line":139},"message":"Input 'gcs' failed with: input logs-consumer failed: no valid auth specified","service.name":"filebeat","id":"logs-co
nsumer","ecs.version":"1.6.0"}

I'm running on GKE.
Service Account has the iam.gke.io/gcp-service-account: **** annotation and it works with filebeat 8.10

How are you setting up Workload Identity so that now it works???

[efd6]

Thanks @alelevinas, it looks like part of the process was omitted. Sending that now.