ERROR: You must specify a region (after upgrading to sceptre 4.0.1 version)

Question

ERROR: You must specify a region (after upgrading to sceptre 4.0.1 version)

chenlego opened this issue 2 years ago · 3 comments

Subject of the issue

Previously, I use sceptre version 3.3.0 and everything could work fine. After upgrade sceptre to 4.0.1. I started to encounter the following problem:

botocore.exceptions.NoRegionError: You must specify a region.

Environment

$ sceptre --version
Sceptre, version 4.0.1

$ python3 --version
Python 3.9.8

Steps to reproduce

file structure:

.
|-- config
|   |-- config.yaml
|   `-- us-west-2
|       |-- app
|       |   `-- myapp.yaml
|       `-- config.yaml
`-- templates
    `-- myapp.yaml

4 directories, 4 files

file content:

$ find . -type f
./config/us-west-2/config.yaml
./config/us-west-2/app/myapp.yaml
./config/config.yaml
./templates/myapp.yaml

## ./config/config.yaml
$ cat ./config/config.yaml
project_code: aws

## ./config/us-west-2/config.yaml
$ cat ./config/us-west-2/config.yaml
region: us-west-2

## ./config/us-west-2/app/myapp.yaml
$ cat ./config/us-west-2/app/myapp.yaml
stack_name: myapp
template_path: myapp.yaml
parameters:
  VpcId: !stack_output_external my-vpc::VPCID

## ./templates/myapp.yaml
$ cat ./templates/myapp.yaml
Parameters:
  VpcId:
    Type: String
    Description: VPC ID

Resources:
  NullResource:
    Type: AWS::CloudFormation::WaitConditionHandle

Expected behaviour

In sceptre version 3.3.0, the same config could working fine without the errors below.

Actual behaviour

execution result

The problem I can see in the following debugging result are:

It seems cannot re-use the Boto3 session.
The first Boto3 session could reterive region from config.yaml properly. But, the second one cannot.

$  sceptre --debug launch -y us-west-2
[2023-02-14 16:26:53] - Adding yaml constructors for the entry point groups ['sceptre.hooks', 'sceptre.resolvers']
[2023-02-14 16:26:54] - Added constructor for <class 'sceptre.hooks.asg_scaling_processes.ASGScalingProcesses'> with node tag !asg_scheduled_actions
[2023-02-14 16:26:54] - Added constructor for <class 'sceptre.hooks.cmd.Cmd'> with node tag !cmd
[2023-02-14 16:26:54] - Added constructor for <class 'sceptre.resolvers.environment_variable.EnvironmentVariable'> with node tag !environment_variable
[2023-02-14 16:26:54] - Added constructor for <class 'sceptre.resolvers.file_contents.FileContents'> with node tag !file_contents
[2023-02-14 16:26:54] - Added constructor for <class 'sceptre.resolvers.no_value.NoValue'> with node tag !no_value
[2023-02-14 16:26:54] - Added constructor for <class 'sceptre.resolvers.stack_attr.StackAttr'> with node tag !stack_attr
[2023-02-14 16:26:54] - Added constructor for <class 'sceptre.resolvers.stack_output.StackOutput'> with node tag !stack_output
[2023-02-14 16:26:54] - Added constructor for <class 'sceptre.resolvers.stack_output.StackOutputExternal'> with node tag !stack_output_external
[2023-02-14 16:26:54] - Added constructor for <class 'resolver.file.File'> with node tag !file
[2023-02-14 16:26:54] - Added constructor for <class 'resolver.rcmd.SceptreResolverCmd'> with node tag !rcmd
[2023-02-14 16:26:54] - Reading in 'us-west-2/app/config.yaml' files...
[2023-02-14 16:26:54] - Config: {'project_path': '/root/reproduce', 'stack_group_path': 'us-west-2/app', 'project_code': 'aws', 'region': 'us-west-2'}
[2023-02-14 16:26:54] - Reading in 'us-west-2/app/myapp.yaml' files...
[2023-02-14 16:26:54] - Config: {'project_path': '/root/reproduce', 'stack_group_path': 'us-west-2/app', 'project_code': 'aws', 'region': 'us-west-2', 'stack_name': 'myapp', 'template_path': 'myapp.yaml', 'parameters': {'VpcId': <sceptre.resolvers.stack_output.StackOutputExternal object at 0x7f8fad0afa00>}}
/usr/local/lib/python3.9/site-packages/sceptre/stack.py:426: DeprecatedWarning: template_path is deprecated as of 4.0.0 and will be removed in 5.0.0. Use the template Stack Config key instead.
  setattr(self, deprecated_attribute_name, deprecated_value)
[2023-02-14 16:26:54] - Generate dependencies for stack us-west-2/app/myapp
[2023-02-14 16:26:54] - us-west-2/app/myapp - Launching Stack
[2023-02-14 16:26:54] - No cloudformation client found, creating one...
[2023-02-14 16:26:54] - Getting Boto3 session
('us-west-2', None, None)
[2023-02-14 16:26:54] - No Boto3 session found, creating one...
[2023-02-14 16:26:54] - Using cli credentials...
[2023-02-14 16:26:54] - Using credential set from shared-credentials-file: {'AccessKeyId': '****************DTGX', 'SecretAccessKey': '************************************dZlW', 'Region': 'us-west-2'}
[2023-02-14 16:26:54] - Boto3 session created
[2023-02-14 16:26:55] - us-west-2/app/myapp - Stack is in the PENDING state
[2023-02-14 16:26:55] - us-west-2/app/myapp - Creating Stack
[2023-02-14 16:26:55] - Resolving external Stack output: my-vpc::VPCID
[2023-02-14 16:26:55] - Collecting outputs from 'my-vpc'...
[2023-02-14 16:26:55] - No cloudformation client found, creating one...
[2023-02-14 16:26:55] - Getting Boto3 session
(None, None, None)
[2023-02-14 16:26:55] - No Boto3 session found, creating one...
[2023-02-14 16:26:55] - Using cli credentials...
[2023-02-14 16:26:55] - Using credential set from shared-credentials-file: {'AccessKeyId': '****************DTGX', 'SecretAccessKey': '************************************dZlW', 'Region': None}
[2023-02-14 16:26:55] - Boto3 session created
Traceback (most recent call last):
  File "/usr/local/bin/sceptre", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/sceptre/cli/helpers.py", line 43, in decorated
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/sceptre/cli/launch.py", line 62, in launch_command
    exit_code = launcher.launch(prune)
  File "/usr/local/lib/python3.9/site-packages/sceptre/cli/launch.py", line 105, in launch
    code = code or self._deploy(deploy_plan)
  File "/usr/local/lib/python3.9/site-packages/sceptre/cli/launch.py", line 197, in _deploy
    result = deploy_plan.launch()
  File "/usr/local/lib/python3.9/site-packages/sceptre/plan/plan.py", line 170, in launch
    return self._execute(*args)
  File "/usr/local/lib/python3.9/site-packages/sceptre/plan/plan.py", line 29, in wrapped
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/sceptre/plan/plan.py", line 55, in _execute
    return executor.execute(*args)
  File "/usr/local/lib/python3.9/site-packages/sceptre/plan/executor.py", line 52, in execute
    stack, status = future.result()
  File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 438, in result
    return self.__get_result()
  File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 390, in __get_result
    raise self._exception
  File "/usr/lib64/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.9/site-packages/sceptre/plan/executor.py", line 59, in _execute
    result = getattr(actions, self.command)(*args)
  File "/usr/local/lib/python3.9/site-packages/sceptre/hooks/__init__.py", line 107, in decorated
    response = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/sceptre/plan/actions.py", line 213, in launch
    status = self.create()
  File "/usr/local/lib/python3.9/site-packages/sceptre/hooks/__init__.py", line 107, in decorated
    response = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/sceptre/plan/actions.py", line 73, in create
    "Parameters": self._format_parameters(self.stack.parameters),
  File "/usr/local/lib/python3.9/site-packages/sceptre/resolvers/__init__.py", line 202, in __get__
    container = super().__get__(stack, stack_class)
  File "/usr/local/lib/python3.9/site-packages/sceptre/resolvers/__init__.py", line 94, in __get__
    return self.get_resolved_value(stack, stack_class)
  File "/usr/local/lib/python3.9/site-packages/sceptre/resolvers/__init__.py", line 252, in get_resolved_value
    _call_func_on_values(resolve, container, Resolver)
  File "/usr/local/lib/python3.9/site-packages/sceptre/helpers.py", line 64, in _call_func_on_values
    func_on_instance(key)
  File "/usr/local/lib/python3.9/site-packages/sceptre/helpers.py", line 58, in func_on_instance
    func(attr, key, value)
  File "/usr/local/lib/python3.9/site-packages/sceptre/resolvers/__init__.py", line 226, in resolve
    result = self.resolve_resolver_value(value)
  File "/usr/local/lib/python3.9/site-packages/sceptre/resolvers/__init__.py", line 165, in resolve_resolver_value
    return resolver.resolve()
  File "/usr/local/lib/python3.9/site-packages/sceptre/resolvers/stack_output.py", line 179, in resolve
    return self._get_output_value(
  File "/usr/local/lib/python3.9/site-packages/sceptre/resolvers/stack_output.py", line 42, in _get_output_value
    outputs = self._get_stack_outputs(stack_name, profile, region, sceptre_role)
  File "/usr/local/lib/python3.9/site-packages/sceptre/resolvers/stack_output.py", line 70, in _get_stack_outputs
    response = connection_manager.call(
  File "/usr/local/lib/python3.9/site-packages/sceptre/connection_manager.py", line 54, in decorated
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/sceptre/connection_manager.py", line 460, in call
    client = self._get_client(service, region, profile, stack_name, sceptre_role)
  File "/usr/local/lib/python3.9/site-packages/sceptre/connection_manager.py", line 370, in _get_client
    self._clients[key] = self._get_session(
  File "/usr/local/lib/python3.9/site-packages/boto3/session.py", line 299, in client
    return self._session.create_client(
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 884, in create_client
    client = client_creator.create_client(
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 102, in create_client
    client_args = self._get_client_args(
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 384, in _get_client_args
    return args_creator.get_client_args(
  File "/usr/local/lib/python3.9/site-packages/botocore/args.py", line 71, in get_client_args
    final_args = self.compute_client_args(
  File "/usr/local/lib/python3.9/site-packages/botocore/args.py", line 148, in compute_client_args
    endpoint_config = self._compute_endpoint_config(
  File "/usr/local/lib/python3.9/site-packages/botocore/args.py", line 234, in _compute_endpoint_config
    return self._resolve_endpoint(**resolve_endpoint_kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/args.py", line 320, in _resolve_endpoint
    return endpoint_bridge.resolve(
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 465, in resolve
    resolved = self.endpoint_resolver.construct_endpoint(
  File "/usr/local/lib/python3.9/site-packages/botocore/regions.py", line 196, in construct_endpoint
    result = self._endpoint_for_partition(
  File "/usr/local/lib/python3.9/site-packages/botocore/regions.py", line 230, in _endpoint_for_partition
    raise NoRegionError()
botocore.exceptions.NoRegionError: You must specify a region.

Workaround

export AWS_DEFAULT_REGION="us-west-2" instead of defining region in config.yaml.
downgrade to 3.3.0

Answer 1 · 2023-02-14T17:13:55.000Z

Hey @chenlego! Thanks for posting this issue. Looks like there's some sort of conflict between stack-output-external and the recent changes to the ConnectionManager. I'll need to dive in, which probably won't happen today. If you wanted to do some investigation yourself and post your findings, that might speed things along.

I'm pretty certain the issue relates to the ConnectionManager's call() method and v4's overhaul of how it resolves values when combined with stack_output_external and how it resolves values.

Answer 2 · 2023-02-18T16:01:14.000Z

I've figured this issue out and have a fix that appears to work. Our integration tests didn't catch this because our integration tests on !external_stack_output...

Didn't test using that resolver without specifying explicit profile and region (which you're not doing)
Apparently to replicate this issue, it looks like either (1) you cannot have a profile specified in your Stack Configs or via environment variable (which I alway do) OR (2) If you have a profile specified, it cannot specify a default region (which I do).

In either case, it somehow made it through. I've added an integration test that should cover this situation in my branch. I should get a fix up today, I think.

Answer 3 · 2023-02-24T15:43:51.000Z

Thank you @jfalkenstein. I do very appreciated with your help. I have validated the new version and it could work properly.