Failed to create cluster behind proxy.

Question

Failed to create cluster behind proxy.

Closed this issue 6 months ago · 11 comments

Issue description

I am following first steps (local) on TDX enabled kernel, but constellation create fails.

constellation version:

Version:        v2.13.0 (Enterprise build; see documentation for license agreement)
GitCommit:      ea1fe82682889056d1b5ede058927ed5960ccb01
GitTreeState:   clean
BuildDate:      2023-11-14T08:51:53
GoVersion:      go1.21.4
Compiler:       bazel/gc
Platform:       linux/amd64

os:
Ubuntu 22.04.3 LTS
kernel:
5.19.17

Steps to reproduce the behavior

Run constellation mini up --debug

Creating cluster in QEMU
Error: creating cluster: creating terraform variables: fetching image reference: sending request for versionsapi.ImageInfo: Get "https://cdn.confidential.cloud/constellation/v2/ref/-/stream/stable/v2.13.0/image/info.json": context canceled

I'm using my company machine, it connects to internet through http proxy, and I configured the proxy correctly before running the command, will constellation client tool uses the proxy？

export https_proxy=http://proxy-host:proxy-port
export http_proxy=http://proxy-host:proxy-port

but I still got the error, here is the details with debug:

2023-11-23T06:40:32Z    DEBUG   cmd/miniup_linux_amd64.go:35    Checked arch and os
2023-11-23T06:40:32Z    DEBUG   cmd/miniup_linux_amd64.go:40    Checked that /dev/kvm exists
2023-11-23T06:40:32Z    DEBUG   cmd/miniup_linux_amd64.go:48    Checked CPU cores - there are 192
2023-11-23T06:40:32Z    DEBUG   cmd/miniup_linux_amd64.go:66    Scanned for available memory
2023-11-23T06:40:32Z    DEBUG   cmd/miniup_linux_amd64.go:74    Checked available memory, you have 1006GB available
2023-11-23T06:40:32Z    DEBUG   cmd/miniup_linux_amd64.go:84    Checked for free space available, you have 117GB available
A config file already exists in the configured workspace.
2023-11-23T06:40:40Z    DEBUG   cmd/miniup.go:187       Creating mini cluster
Error: creating cluster: creating terraform variables: fetching image reference: sending request for versionsapi.ImageInfo: Get "https://cdn.confidential.cloud/constellation/v2/ref/-/stream/stable/v2.13.0/image/info.json": dial tcp 13.225.103.76:443: connect: connection timed out

However i can download the json file with wget

Proxy request sent, awaiting response... 200 OK
Length: 1937 (1.9K) [application/octet-stream]
Saving to: ‘info.json’

info.json                          100%[================================================================>]   1.89K  --.-KB/s    in 0s

2023-11-23 07:21:11 (854 MB/s) - ‘info.json’ saved [1937/1937]

iptables:

-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-N LIBVIRT_FWI
-N LIBVIRT_FWO
-N LIBVIRT_FWX
-N LIBVIRT_INP
-N LIBVIRT_OUT
-A INPUT -j LIBVIRT_INP
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -j LIBVIRT_FWX
-A FORWARD -j LIBVIRT_FWI
-A FORWARD -j LIBVIRT_FWO
-A OUTPUT -j LIBVIRT_OUT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
-A LIBVIRT_FWI -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A LIBVIRT_FWI -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWO -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A LIBVIRT_FWO -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWX -i virbr0 -o virbr0 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p tcp -m tcp --dport 68 -j ACCEPT

Version

v2.13.0

Constellation Config

auto generated

Answer 1 · 2023-11-23T09:27:34.000Z

I'd like to make a temp fix by add proxy in constellation source code locally, but how to rebuild constellation?

bazel build //...
INFO: Reading rc options for 'build' from ${SRC_DIR}/constellation/bazel/bazelrc/convenience.bazelrc:
  Inherited 'common' options: --enable_platform_specific_config --heap_dump_on_oom
ERROR: --heap_dump_on_oom :: Unrecognized option: --heap_dump_on_oom

Answer 2 · 2023-11-23T09:48:57.000Z

Hey @michaelrun,

First of all thanks for pointing out this issue! We've implemented a fix in #2635.

You can check out the branch with the fix and build the open source CLI with Bazel from there using the following commands:

bazel build //cli:cli_oss

And then execute your built CLI with

bazel-bin/cli/cli_oss_/cli_oss

As a sidenote, you will not need a TDX-enabled machine to test with MiniConstellation, as it only relies on a virtual TPM by KVM / QEMU for attestation.

Feel free to reach out here in case anything should be unclear!

Answer 3 · 2023-11-23T12:24:07.000Z

Hi, @msanft , I have no knowledge about bazel, got trouble as below:

bazel build //cli:cli_oss

INFO: Reading rc options for 'build' from ${SRC_DIR}/constellation/bazel/bazelrc/convenience.bazelrc:
  Inherited 'common' options: --enable_platform_specific_config --heap_dump_on_oom
ERROR: --heap_dump_on_oom :: Unrecognized option: --heap_dump_on_oom

bazel version:

[bazel release 3.5.1- (@non-git)]

I don't know how to continue, any idea? Thanks!

Answer 4 · 2023-11-23T12:35:48.000Z

bazel version:

[bazel release 3.5.1- (@non-git)]

The code in this repository requires a more recent Bazel version.
It looks like you are already using nix. If so, you can enter a shell with a more recent Bazel version by typing: nix develop, followed by bazel version, which should now return Build label: 6.4.0- (@non-git).

Answer 5 · 2023-11-23T12:44:27.000Z

Actually i din't install nix yet, however I just tried to install it by apt install nix-bin, unfortunately, other error appear:

#nix develop bazel version

error: unexpected argument 'version'
Try 'nix --help' for more information.

Answer 6 · 2023-11-23T12:46:18.000Z

The first command nix develop enters a new shell session. In there you can run build commands or query the Bazel version.

EDIT: this document should also help you with setting up the development environment: https://github.com/edgelesssys/constellation/blob/main/dev-docs/workflows/build-develop-deploy.md

Answer 7 · 2023-11-23T13:15:58.000Z

Hi, @malt3 It is not easy for me to use constellation build system in a short time.

build with nix, it seems the nix has proxy issue too:

# nix develop
warning: error: unable to download 'https://cache.nixos.org/nix-cache-info': Timeout was reached (28); retrying in 271 ms

error: flake 'git+file://${SRC_DIR}/constellation' does not provide attribute 'devShells.x86_64-linux.devShell.x86_64-linux', 'packages.x86_64-linux.devShell.x86_64-linux', 'legacyPackages.x86_64-linux.devShell.x86_64-linux', 'devShell.x86_64-linux' or 'defaultPackage.x86_64-linux'

download latest bazel(bazel_nojdk-8.0.0-pre.20231030.2-linux-x86_64), then build with that, it reports error as below:

../bazel_nojdk-8.0.0-pre.20231030.2-linux-x86_64 build //cli:cli_oss

Starting local Bazel server and connecting to it...
WARNING: Option 'experimental_action_cache_store_output_metadata' is deprecated: Use --action_cache_store_output_metadata instead
WARNING: Option 'experimental_allow_tags_propagation' is deprecated: Use --incompatible_allow_tags_propagation instead
WARNING: --enable_bzlmod is set, but no MODULE.bazel file was found at the workspace root. Bazel will create an empty MODULE.bazel file. Please consider migrating your external dependencies from WORKSPACE to MODULE.bazel. For more details, please refer to https://github.com/bazelbuild/bazel/issues/18958.
WARNING: Option 'experimental_action_cache_store_output_metadata' is deprecated: Use --action_cache_store_output_metadata instead
WARNING: Option 'experimental_allow_tags_propagation' is deprecated: Use --incompatible_allow_tags_propagation instead
ERROR: ${SRC_DIR}/constellation/cli/BUILD.bazel:12:10: While resolving toolchains for target //cli:cli_oss (cc16af8): invalid registered toolchain '@go_sdk_toolchains//:toolchain_go_aix_ppc64': error loading package '@go_sdk_toolchains//': at /root/.cache/bazel/_bazel_root/3bcc67957bca0315e769105f06e58335/external/go_sdk_toolchains/toolchain.bzl:2:6: Unable to find package for @rules_go//go:def.bzl: The repository '@rules_go' could not be resolved: Repository '@rules_go' is not defined.
INFO: Analyzed target //cli:cli_oss (72 packages loaded, 11 targets configured).
WARNING: errors encountered while analyzing target '//cli:cli_oss': it will not be built
INFO: Found 0 targets...
ERROR: command succeeded, but not all targets were analyzed
INFO: Elapsed time: 269.409s, Critical Path: 0.06s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully

Is it possible to provide a temp constellation client binary?

Answer 8 · 2023-12-04T01:30:59.000Z

Is there any plan for new release?

Answer 9 · 2023-12-04T09:21:29.000Z

Ho @michaelrun. The next release will still take some time.
Sorry for keeping you waiting. Here is a CLI for Linux, amd64 with the patch applied:

https://uploadthing.com/f/7e287508-18ff-4f5c-bc87-ba01bcc7c4ba-ygt2jx

EDIT: also see this alternative URL, in case the other one is not accessible: https://cdn.confidential.cloud/constellation/customer-support/2634/constellation

Answer 10 · 2023-12-06T08:57:17.000Z

thanks @malt3 , i will try later

Answer 11 · 2024-01-09T13:21:01.000Z

Versions >= 2.14.0 include this fix