PuppetDeployment Failing in ServiceControlPolicy[Both Complete & SingleAccount Run] after upgrade
Closed this issue · 8 comments
Describe the bug
With no mainfest or IAC template changes. Just upgraded both Factory and Puppet to specific versions factory[0.97] and Puppet[0.213.1]. The failure is observed for scp deployment which were getting deployed successfully earlier.
To Reproduce
Steps to reproduce the behavior:
- Vend a new Account or triggered 'servicecatalog-puppet-pipeline'
- See error below[_ belongs to single account vending operation_]:
Error-1:
INFO scheduler sending: service-control-policies_ttbc-guardrails-scp-security-services-general_-eu-west-1
606 | /root/.pyenv/versions/3.9.12/lib/python3.9/site-packages/luigi/parameter.py:291: UserWarning: Parameter "get_or_create_policy_ref" with value "None" is not of type string.
607 | warnings.warn('Parameter "{}" with value "{}" is not of type string.'.format(param_name, param_value))
608 | INFO worker#1 executing task: service-control-policies_ttbc-guardrails-scp-security-services-general_-eu-west-1
609 | INFO worker#1 DoExecuteServiceControlPoliciesTask:service-control-policies_ttbc-guardrails-scp-security-services-general_-eu-west-1 started
613 | INFO worker#1 service-control-policies_ttbc-guardrails-scp-security-services-general_-eu-west-1: Ensuring attachments for policies
614 | ERROR worker#1 executed task [failure]: service-control-policies_ttbc-guardrails-scp-security-services-general_-eu-west-1 failures: 'NoneType' object has no attribute 'open'
615 | ERROR worker#1 ---- START OF ERROR----
616 | ERROR worker#1 Task DoExecuteServiceControlPoliciesTask:
617 | ERROR worker#1 account_id: ''
618 | ERROR worker#1 cache_invalidator: '2023-01-23 15:20:09.949596'
619 | ERROR worker#1 content:
620 | ERROR worker#1 s3:
621 | ERROR worker#1 bucket: ttbc-scp-artifacts-******
622 | ERROR worker#1 key: ttbc-guardrails-scp-security-services-general.json
623 | ERROR worker#1 dependencies_by_reference:
624 | ERROR worker#1 - create-policies
625 | ERROR worker#1 description: Protecting Security Services across the Organization.
626 | ERROR worker#1 get_or_create_policy_ref: null
627 | ERROR worker#1 manifest_file_path: ./manifest-expanded.yaml
628 | ERROR worker#1 manifest_files_path: .
629 | ERROR worker#1 manifest_task_reference_file_path: ./manifest-task-reference.json
630 | ERROR worker#1 ou_name: ou-mpd*-***
631 | ERROR worker#1 puppet_account_id: '315*****'
632 | ERROR worker#1 region: eu-west-1
633 | ERROR worker#1 requested_priority: 0
634 | ERROR worker#1 service_control_policy_name: ttbc-guardrails-scp-security-services-general
635 | ERROR worker#1 task_reference: service-control-policies_ttbc-guardrails-scp-security-services-general_-eu-west-1
636 | ERROR worker#1
637 | ERROR worker#1 Traceback (most recent call last):
638 | ERROR worker#1
639 | ERROR worker#1 File "/root/.pyenv/versions/3.9.12/lib/python3.9/site-packages/servicecatalog_puppet/waluigi/threads/topological_generations.py", line 184, in worker_task
640 | ERROR worker#1 task.execute()
641 | ERROR worker#1
642 | ERROR worker#1 File "/root/.pyenv/versions/3.9.12/lib/python3.9/site-packages/servicecatalog_puppet/waluigi/tasks.py", line 66, in execute
643 | ERROR worker#1 self.run()
644 | ERROR worker#1
645 | ERROR worker#1 File "/root/.pyenv/versions/3.9.12/lib/python3.9/site-packages/servicecatalog_puppet/workflow/service_control_policies/do_execute_service_control_policies_task.py", line 63, in run
646 | ERROR worker#1 policy_id = self.get_output_from_reference_dependency(
647 | ERROR worker#1
648 | ERROR worker#1 File "/root/.pyenv/versions/3.9.12/lib/python3.9/site-packages/servicecatalog_puppet/workflow/dependencies/tasks.py", line 42, in get_output_from_reference_dependency
649 | ERROR worker#1 with self.input().get("reference_dependencies").get(reference).open("r") as f:
650 | ERROR worker#1
651 | ERROR worker#1 AttributeError: 'NoneType' object has no attribute 'open'
652 | ERROR worker#1
653 | ERROR worker#1 ---- END OF ERROR ----
Mainfest Configuration
Below is the mainfest configuration that being used. [execution mode = async]
ttbc-guardrails-scp-security-services-general: description: "Enforcing *****" execution: async tags: - Key: Category Value: Encryption content: s3: bucket: ttbc-scp-artifacts-****** key: ttbc-guardrails-scp-******.json apply_to: ous: - ou: ou-mp******** - ou: ou-mp********
Expected behavior
The deployment should work without any failure as per last execution state before the upgrade.
@eamonnfaherty I checked this further and its failing for tag policies too whenever the target is multiple ou's. The policy(scp or tagging) with single OU is working fine. Can you please check this as it has high impact ?
Can you share your manifest file please. Just the scp or tag policy section should be enough.
is the issue when you have multiple ous listed in the apply_to section? or is the issue with nested ous?
Tag policy and SCP are failing if multiple ous are listed in the apply_to configuration
generate-task-reference is only generating one task when there are two targets
task reference does not have target in the name when the target is an ou : eg service-control-policies_deny-organizations-leave-organization_-eu-west-1
is the issue when you have multiple ous listed in the apply_to section? or is the issue with nested ous?
The actual project setup is big so I tried to reproduce the same at my own playground org. Haven't tried for nested OU's but failing for sure for mutiple OU's when configured in apply_to section. The same SCP & TaggingPolicy manifest works when i applied them to single OU.