Please add exclusions

[kenmoore25]

Please add exclusions as an input variable and pass it along to the resource.

The exclusion block is already available in resources as seen here:

https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/logging_project_sink#exclusions

I believe it would just need to be an additional variable and reference it in the resource section of the sub-module

[bharathkkb]

Hi @kenmoore25
Thanks for the report. IIUC we can also use the resource google_logging_project_exclusion to achieve the same purpose?

[kenmoore25]

@bharathkkb - I was trying to best understand how the exclusions block works on the project vs just on the log sink as I was reading the documentation. I was trying to understand if they are basically the same thing or if one only impacts the specific log sink.

I can give that a try - I've got it working using the standard google_logging_project_sink resource - using the exclusions block.

[kenmoore25]

@bharathkkb - After a little trial, I think they actually do work differently.

I cannot find the place where it actually displays my log exclusions on the project, except through the API (tried the UI and via gcloud command line)

Here is the output of the log export with the project exclusion filter:

logging.sinks.get
{
"name": "",
"destination": "pubsub.googleapis.com/projects//topics/",
"filter": "Change_Me_to_Send_Additional_Logs_to_Splunk; Leave_Exclusions_in_Place_PLEASE",
"writerIdentity": "serviceAccount:p***@gcp-sa-logging.iam.gserviceaccount.com",
"createTime": "2020-11-24T14:32:59.781680406Z",
"updateTime": "2020-11-24T19:14:43.336726821Z"
}

logging.exclusions.get
{
"exclusions": [
{
"name": "Aggregated_Logs_Exclusion",
"description": "Excluding logs already captured by aggregated log sink (at folder). Please DO NOT alter/remove.",
"filter": "LOG_ID("cloudaudit.googleapis.com/activity") OR LOG_ID("externalaudit.googleapis.com/activity") OR LOG_ID("cloudaudit.googleapis.com/system_event") OR LOG_ID("externalaudit.googleapis.com/system_event") OR LOG_ID("cloudaudit.googleapis.com/access_transparency") OR LOG_ID("externalaudit.googleapis.com/access_transparency")",
"createTime": "2020-11-24T19:14:43.313344920Z",
"updateTime": "2020-11-24T19:14:43.313344920Z"
}
]
}

Here is the output of the log export with the exclusions block in the google_logging_project_sink resource:

{
"name": "-log-export",
"destination": "pubsub.googleapis.com/projects//topics/-logs-export",
"filter": "Change_Me_to_Send_Additional_Logs_to_Splunk; Leave_Exclusions_in_Place_PLEASE",
"writerIdentity": "serviceAccount:p@gcp-sa-logging.iam.gserviceaccount.com",
"createTime": "2020-11-24T14:32:59.781680406Z",
"updateTime": "2020-11-24T19:26:17.731837115Z",
"exclusions": [
{
"name": "Aggregated_Logs_Exclusion",
"description": "Excluding logs already captured by aggregated log sink (at folder). Please DO NOT alter/remove.",
"filter": "LOG_ID("cloudaudit.googleapis.com/activity") OR LOG_ID("externalaudit.googleapis.com/activity") OR LOG_ID("cloudaudit.googleapis.com/system_event") OR LOG_ID("externalaudit.googleapis.com/system_event") OR LOG_ID("cloudaudit.googleapis.com/access_transparency") OR LOG_ID("externalaudit.googleapis.com/access_transparency")"
}
]
}

Honestly, I do not know the specific differences between the two, other than it appears that the exclusion block placed on the google_logging_project_sink might only apply to that specific log-sink (which we are pushing through pub/sub) while the other looks like it might be more global - applies at the project level.

Appreciate you looking into this.

[bharathkkb]

Hi @kenmoore25
After looking through the API docs it seems like google_logging_project_exclusion operates on _Default sink, however defining exclusions with the log sink via google_logging_project_sink operates on the same sink. google_logging_*_sink resources also seems to supported. hashicorp/terraform-provider-google#7880 for tracking doc update

[kenmoore25]

Hey @bharathkkb - appreciate your help looking into this.

After trying both the project exclusion and the individual sink exclusion, only the individual exclusion seems to fit what I'm trying to do.

We have an aggregated log sink on a folder that includes the children. We have a log sink in each project that will ship logs via pub/sub to splunk (who subscribes).

When I try to use the project level exclusion with the aggregated log sink and use the same filters on both, I get duplicate messages in splunk (and obviously paying double via pub/sub).

When I use the aggregated with the log exclusion specified on the individual log sink (and I can use the same filters for inclusion on the aggregated sink, inclusion on the specific log sink, and the exclusion filter applied in that same log sink (not via the project logging.exclusion api call), I only get a single log - the one from the aggregated log sink at the folder level.

Again...really appreciate your help in looking into this. I honestly think it's another variable on the logging module and exposing the exclusions on the sub-module - hope it's that simple.

Also...if we can get our CLA in place, we can try to do the pull request and help out when we things like this that we might be able to help on...sorry...

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days

[kravvcu]

@bharathkkb can you take a look at #103? Or tell me who could?