mtouloup/stellar-docker-testnet

Stellar private network fails to upgrade to latest version

Opened this issue · 7 comments

mtouloup commented
  1. Private Network of n number nodes is deployed
  2. Peers are connected and added in the list of quorum within the cfg file of each validator
  3. Horizon instance is up and running, getting the history of the validator-genesis served by a local nginx server
  4. Using the stellar python sdk, account balance and other requests seems to work, however, creating a new account and fund it through the root account returns 400 error (tx_not_supported)

PS1: cfg file to be used by each node/validator here: https://github.com/mtouloup/stellar-docker-testnet/blob/main/stellar-genesis/stellar-core.cfg
PS2: Here is the "info" of one of the validators within the network

 {
    "info" : {
       "build" : "stellar-core 18.3.0 (2f9ce11b2e7eba7d7d38b123ee6da9e0144249f8)",
       "ledger" : {
          "age" : 2,
          "baseFee" : 100,
          "baseReserve" : 100000000,
          "closeTime" : 1645479069,
          "hash" : "b22961c8ac0fe21547ce9afe0e682ca454c46faea08c923eb18a218a811171d4",
          "maxTxSetSize" : 100,
         "num" : 1058,
         "version" : 0
      },
      "network" : "Standalone UNIC_IFF Network ; 2022",
      "peers" : {
         "authenticated_count" : 4,
         "pending_count" : 0
      },
      "protocol_version" : 18,
      "quorum" : {
         "node" : "self",
         "qset" : {
            "agree" : 5,
            "cost" : 17200,
            "delayed" : 0,
            "disagree" : 0,
            "fail_at" : 1,
            "hash" : "132ebb",
            "lag_ms" : 193,
            "ledger" : 1057,
            "missing" : 0,
            "phase" : "EXTERNALIZE",
            "validated" : true
         },
         "transitive" : {
            "critical" : null,
            "intersection" : true,
            "last_check_ledger" : 2,
            "node_count" : 5
         }
      },
      "startedOn" : "2022-02-21T20:00:09Z",
      "state" : "Synced!"
   }
}

PS3: Here is what Horizon returns:

{
  "_links": {
    "account": {
      "href": "http://10.10.70.21:8000/accounts/{account_id}",
      "templated": true
    },
    "accounts": {
      "href": "http://10.10.70.21:8000/accounts{?signer,sponsor,asset,liquidity_pool,cursor,limit,order}",
      "templated": true
    },
    "account_transactions": {
      "href": "http://10.10.70.21:8000/accounts/{account_id}/transactions{?cursor,limit,order}",
      "templated": true
    },
    "claimable_balances": {
      "href": "http://10.10.70.21:8000/claimable_balances{?asset,sponsor,claimant,cursor,limit,order}",
      "templated": true
    },
    "assets": {
      "href": "http://10.10.70.21:8000/assets{?asset_code,asset_issuer,cursor,limit,order}",
      "templated": true
    },
    "effects": {
      "href": "http://10.10.70.21:8000/effects{?cursor,limit,order}",
      "templated": true
    },
    "fee_stats": {
      "href": "http://10.10.70.21:8000/fee_stats"
    },
    "ledger": {
      "href": "http://10.10.70.21:8000/ledger/{sequence}",
      "templated": true
    },
    "ledgers": {
      "href": "http://10.10.70.21:8000/ledgers{?cursor,limit,order}",
      "templated": true
    },
    "liquidity_pools": {
      "href": "http://10.10.70.21:8000/liquidity_pools{?reserves,account,cursor,limit,order}",
      "templated": true
    },
    "offer": {
      "href": "http://10.10.70.21:8000/offers/{offer_id}",
      "templated": true
    },
    "offers": {
      "href": "http://10.10.70.21:8000/offers{?selling,buying,seller,sponsor,cursor,limit,order}",
      "templated": true
    },
    "operation": {
      "href": "http://10.10.70.21:8000/operations/{id}",
      "templated": true
    },
    "operations": {
      "href": "http://10.10.70.21:8000/operations{?cursor,limit,order,include_failed}",
      "templated": true
    },
    "order_book": {
      "href": "http://10.10.70.21:8000/order_book{?selling_asset_type,selling_asset_code,selling_asset_issuer,buying_asset_type,buying_asset_code,buying_asset_issuer,limit}",
      "templated": true
    },
    "payments": {
      "href": "http://10.10.70.21:8000/payments{?cursor,limit,order,include_failed}",
      "templated": true
    },
    "self": {
      "href": "http://10.10.70.21:8000/"
    },
    "strict_receive_paths": {
      "href": "http://10.10.70.21:8000/paths/strict-receive{?source_assets,source_account,destination_account,destination_asset_type,destination_asset_issuer,destination_asset_code,destination_amount}",
      "templated": true
    },
    "strict_send_paths": {
      "href": "http://10.10.70.21:8000/paths/strict-send{?destination_account,destination_assets,source_asset_type,source_asset_issuer,source_asset_code,source_amount}",
      "templated": true
    },
    "trade_aggregations": {
      "href": "http://10.10.70.21:8000/trade_aggregations?base_asset_type={base_asset_type}\u0026base_asset_code={base_asset_code}\u0026base_asset_issuer={base_asset_issuer}\u0026counter_asset_type={counter_asset_type}\u0026counter_asset_code={counter_asset_code}\u0026counter_asset_issuer={counter_asset_issuer}",
      "templated": true
    },
    "trades": {
      "href": "http://10.10.70.21:8000/trades?base_asset_type={base_asset_type}\u0026base_asset_code={base_asset_code}\u0026base_asset_issuer={base_asset_issuer}\u0026counter_asset_type={counter_asset_type}\u0026counter_asset_code={counter_asset_code}\u0026counter_asset_issuer={counter_asset_issuer}",
      "templated": true
    },
    "transaction": {
      "href": "http://10.10.70.21:8000/transactions/{hash}",
      "templated": true
    },
    "transactions": {
      "href": "http://10.10.70.21:8000/transactions{?cursor,limit,order}",
      "templated": true
    }
  },
  "horizon_version": "devel",
  "core_version": "stellar-core 18.3.0 (2f9ce11b2e7eba7d7d38b123ee6da9e0144249f8)",
  "ingest_latest_ledger": 1050,
  "history_latest_ledger": 1050,
  "history_latest_ledger_closed_at": "2022-02-21T21:30:29Z",
  "history_elder_ledger": 256,
  "core_latest_ledger": 1050,
  "network_passphrase": "Standalone UNIC_IFF Network ; 2022",
  "current_protocol_version": 0,
  "core_supported_protocol_version": 18
}

PS3: This is the source code used with python and stellar python SDK

from stellar_sdk import Keypair, Network, Server, TransactionBuilder

server = Server(horizon_url="http://10.10.70.21:8000")
from_account = server.load_account(account_id="GCDCEUV7IQSV47A75M3ZOP6GTELCLAPPZXXZD4POX4A2E77JKDS4NXJL")

destination = Keypair.random()
transaction = (
    TransactionBuilder(
        source_account=from_account,
        network_passphrase="Standalone UNIC_IFF Network ; 2022",
        base_fee=100,
    )
    .append_create_account_op(
        destination=destination.public_key, starting_balance="100"
    )
    .set_timeout(30)
    .build()
)
transaction.sign("SBJK323PKCHUBGUHA674RQZ6NVCHWRUNKPJMXREAJCL2CXJNVVZWZUPW")

print(transaction)
response = server.submit_transaction(transaction)
print(f"Transaction hash: {response['hash']}")
print(f"New Keypair: \n\taccount id: {destination.public_key}\n\tsecret seed: {destination.secret}")

PS4: This is the response from the network

stellar_sdk.exceptions.BadRequestError: {
  "type": "https://stellar.org/horizon-errors/transaction_failed",
  "title": "Transaction Failed",
  "status": 400,
  "detail": "The transaction failed when submitted to the stellar network. The `extras.result_codes` field on this response contains further details.  Descriptions of each code can be found at: https://developers.stellar.org/api/errors/http-status-codes/horizon-specific/transaction-failed/",
  "extras": {
    "envelope_xdr": "AAAAAgAAAACGIlK/RCVefB/rN5c/xpkWJYHvze+R8e6/AaJ/6VDlxgAAAGQAAAAAAAAAAQAAAAEAAAAAAAAAAAAAAABiE/8cAAAAAAAAAAEAAAAAAAAAAAAAAABLw6f/ZYT6bptt+x3ZkNDi91hBks/DTA+RCxmb6QPkCQAAAAA7msoAAAAAAAAAAAHpUOXGAAAAQHzUGVBuTyGfBpXRfFPR+ggGk3Y5QyYvo+8DdYyrb0wuxVnOI+gxWCN3E/M2nq5Q9/ZFqGupJvHUQXZaGvXZPAU=",
    "result_codes": {
      "transaction": "tx_not_supported"
    },
    "result_xdr": "AAAAAAAAAGT////0AAAAAA=="
  }
}
rice2000 commented

"current_protocol_version": 0
means you need to upgrade the protocol version of your network. On a new network, it starts at 0. You want it to be 18.

You can use the stellar-core http-command to upgrade the network (see "upgrades" here: https://developers.stellar.org/docs/run-core-node/commands/#http-commands)

Or you can use a request to port 11626:
http://localhost:11626/upgrades?mode=set&upgradetime=1970-01-01T00:00:00Z&protocolversion=18&basereserve=5000000

mtouloup commented

Hi Justin and thanks for your reply. I have tested what you have proposed but it seems "current_protocol_version" from my horizon instance is not upgraded. I can see from the validators logs that they get the upgrade request however nothing changes within the horizon's response. Let me paste some logs of the moment when the validator gets the upgrade request:

2022-03-21T13:12:05.915154132Z 2022-03-21T13:12:05.914 GBJJM [Herder INFO] Armed with network upgrades: upgradetime=1970-01-01T00:00:00Z, protocolversion=18
2022-03-21T13:12:06.172409229Z 2022-03-21T13:12:06.171 GBJJM [Overlay TRACE] OverlayManagerImpl tick
2022-03-21T13:12:06.172453133Z 2022-03-21T13:12:06.172 GBJJM [Overlay TRACE] Peer 172.21.0.8:11635 is not preferred
2022-03-21T13:12:06.172464135Z 2022-03-21T13:12:06.172 GBJJM [Overlay TRACE] Peer 172.21.0.7:11635 is not preferred
2022-03-21T13:12:06.172473035Z 2022-03-21T13:12:06.172 GBJJM [Overlay TRACE] Peer 172.21.0.5:11625 is not preferred
2022-03-21T13:12:06.579505118Z 2022-03-21T13:12:06.579 GBJJM [Overlay DEBUG] TCPPeer::startRead 0 from 172.21.0.5:11625
2022-03-21T13:12:06.581375902Z 2022-03-21T13:12:06.581 GBJJM [Overlay TRACE] No quorum set: 2c72ef
2022-03-21T13:12:06.581396204Z 2022-03-21T13:12:06.581 GBJJM [Overlay TRACE] send: DONTHAVE 10:2c72ef to : GDDEM
2022-03-21T13:12:06.581406305Z 2022-03-21T13:12:06.581 GBJJM [Overlay DEBUG] messageSender 172.21.0.5:11625 - b:88 n:1/1
2022-03-21T13:12:06.601825621Z 2022-03-21T13:12:06.601 GBJJM [Overlay DEBUG] TCPPeer::startRead 0 from 172.21.0.8:11635
2022-03-21T13:12:06.603384175Z 2022-03-21T13:12:06.603 GBJJM [Overlay TRACE] No quorum set: 1aa1b9
2022-03-21T13:12:06.603406677Z 2022-03-21T13:12:06.603 GBJJM [Overlay TRACE] send: DONTHAVE 10:1aa1b9 to : GAFNA

Also, let me add some questions for some clarity:

  1. As you can see from my first comment, in the "info" respponse of the validators, their protocol version is 18. Thus, why does horizon instance returns 0 as current protocol version?

  2. As you may have understood, we are running a private network with n number of nodes. Thus, to which validator shall i send the upgrade protocol version request? To all of them or just to the one the horizon instance is communicating with?

  3. What should i receive as a response on the upgrade request in order to understand that my request succesfully passed through?

Many thanks for your time :)

rice2000 commented

Sorry for the slow response here!

A quorum of validators need to agree to upgrade the network (since network upgrades are voted on as part of consensus). In your case, you should just arm every validator to upgrade. Once you do, they should agree to the upgrade, and the network should switch to protocol 18.

For more info, see: https://developers.stellar.org/docs/run-core-node/network-upgrades/

jcx120 commented

hi - did you also upgrade your Horizon to a version that matches Core? I see "current_protocol_version" is 0 where it needs to be at least 18 to match the Core version.

Note that Horizon is a separate package from Core Validators and should be installed and run as an independent standalone system. Horizon has an "embedded" non-validating Core within it (called "Captive Core") that needs to be enabled. Captive Core allows Horizon to interact with the consensus network (and will need to be configured to point to your private network).

mtouloup commented

I've enabled captive core through horizon, everything seems to be okay with no errors in my logs, however validators fail to upgrade the network protocol version to 18. Let me attach some screenshots from my setup.

Firstly, the private network is consisted of 3 validators. Below we have the "info" command of the first one:
stellar-genesis
As you can see the validator seems to be synced with the network and part of the quorum set.

Then we have the 2nd validator:
validator-0
Again, the 2nd validator seems to be synced with the network and part of the quorum set.

Finally, the third validator:
validator-1
Same here, the 3rd validator seems to be synced with the network and part of the quorum set.

In another screenshot of my logs you can see that the networks closes new ledgers succesfully and normally proceeds...
ledger_closing

However, still after executing the command of upgrading the network to all the validators (i can see the log from each validator "armed with network upgrade", the validators do not seem to agree on the network upgrade....

Here is a screenshot of the horizon instance where you can also see that the ledgers are parsed through the published history etc... Still the current protocol version remains to 0
horizon

Is there anything else you believe i should do??

jcx120 commented

Horizon might not be configured correctly. Please take a look at these configuration steps. On that thread, you should review every step from installation, configuring, ingesting to running. Horizon should be treated as a completely separate system from validators. It will however, need to know the references to your validators in the configuration.

mtouloup commented

thanks for those, thats what i've used to configure my horizon instance etc. Can you please have a look at my configurations? Maybe you will find something odd...

Here are the configurations of my Horizon instance in a docker-compose file: https://github.com/mtouloup/stellar-docker-testnet/blob/main/templates/docker-compose-network-template.yml#L7

Here is the toml file for the captive core: https://github.com/mtouloup/stellar-docker-testnet/blob/main/templates/stellar-captive-core.toml

Here is the cfg file of the validator who publishes its history: https://github.com/mtouloup/stellar-docker-testnet/blob/main/stellar-genesis/stellar-core.cfg

Here is the config file used for the rest of the validators in the network: https://github.com/mtouloup/stellar-docker-testnet/blob/main/templates/stellar_template.cfg

Many thanks :)