blockchain

Fundamentals of Blockchain and beyond.

ULTIMATE RESOURCES - http://www.dappuniversity.com/articles/the-complete-blockchain-developer-resource-list

Exercises

Hashing - https://github.com/siddeshpillai/blockchain/tree/hashing-exercise
Block - https://github.com/siddeshpillai/blockchain/tree/block-exercise
Bitcoin Message - https://github.com/siddeshpillai/blockchain/tree/bitcoin-message
Block Explorer - https://github.com/siddeshpillai/blockchain/tree/block-explorer
Create Simple Block Chain - https://github.com/siddeshpillai/blockchain/tree/simple-chain
Private Blockchain - https://github.com/siddeshpillai/blockchain/tree/private-blockchain
Express JS Block Exercise - https://github.com/siddeshpillai/blockchain/tree/express-js-exercise
Hapi JS Block Exercise - https://github.com/siddeshpillai/blockchain/tree/hapi-js-exercise
Webservices - https://github.com/siddeshpillai/blockchain/tree/blockchain-webservice
DAPP Secret Messenger - https://github.com/siddeshpillai/blockchain/tree/dapp-secret-messenger
Create and deploy an ERC20 token in local & rinkeby network - https://github.com/siddeshpillai/blockchain/tree/erc20tokens
Create ERC20 tokens using Webpack - https://github.com/siddeshpillai/blockchain/tree/ERC20-tokens-webpack
Create ERC721 Star Notary tokens - https://github.com/siddeshpillai/blockchain/tree/ERC-721-star-notary

Good Reads

How to timestamp digital document - https://www.anf.es/pdf/Haber_Stornetta.pdf
Bitcoin: A peer to peer cash system - https://bitcoin.org/bitcoin.pdf
Genesis Block - https://en.bitcoin.it/wiki/Genesis_block
The Complete Satoshi - https://satoshi.nakamotoinstitute.org/

Workshops

Resources

Bitcoin Stats - https://bitcoinvisuals.com/stats
Importing bitcoins via private keys - https://bitcoinelectrum.com/importing-your-private-keys-into-electrum/
Generate your own address - https://bitaddress.org
Bitoin Developer Glossary - https://bitcoin.org/en/developer-glossary#a
Bitcoin Improvement Proposals (BIP) = https://github.com/bitcoin/bips
Bitcoin book - https://github.com/bitcoinbook/bitcoinbook
Certifications- https://cryptoconsortium.org/certifications/CBP
Address - https://www.bitaddress.org
Hyperledger explorer - https://github.com/hyperledger/blockchain-explorer
Blockchain demo - https://github.com/anders94/blockchain-demo
Bitcoin Testnet block explorer - https://live.blockcypher.com/btc-testnet/
Block Verification - https://bitcoin-message-validation.firebaseapp.com/index.html
Udacity: Creating unique metadata with solidity - https://medium.com/@alexpanas/udacity-blockchain-nanodegree-creating-unique-metadata-with-solidity-daccded333ef

Projects

Create a Private Blockchain - https://github.com/siddeshpillai/blockchain/tree/private-blockchain
Create a Block Chain WebService - https://github.com/siddeshpillai/blockchain/tree/blockchain-webservice
Create a Block Chain Notary Service - https://github.com/siddeshpillai/blockchain/tree/blockchain-notary-service
Create a CryptoStar DAPP using Ethereum - https://github.com/siddeshpillai/blockchain/tree/CryptoStar-DAPP-Ethereum

Future Projects

Blockchain Identity
Blockchain data
WebServices
Identify and Notary Service
Identify & Smart Contracts (DAPP)
Architecture
Supply Chain

Consensus Algorithms

Proof of Work (PoW)

Bitcoin figured out how to use the Proof of Work algorithm to solve this issue.

The main innovation that Satoshi Nakamoto introduced in Bitcoin’s white paper is using Proof of Work (PoW) to achieve consensus without a central authority and solve the double-spend problem.

How Does It Work? PoW involves miner nodes, or miners, to solve a math puzzle that requires a lot of computation power. Whichever miner is able to solve the puzzle the fastest is able to add a block of transactions to the blockchain, and in return, they are paid the transaction fees from all the transactions included in the block as well as paid by the network with bitcoins that were newly created upon the “mining” of the block.

Potential Issues 2 Commonly discussed issues with Proof of Work are:

Extremely High-Energy Consumption A Monopoly of Miners which Leads to a Concern for System Centralizations

Proof of Stake

In the Proof-of-Stake Consensus Protocol, there are no more miners; instead, there are Validators. These validators, or stakeholders, determine which block makes it onto the blockchain. In order to validate transactions and create blocks, validators put up their own coins as “stake”. Think of it as placing a bet - if they validate a fraudulent transaction, they lose their holdings as well as their rights to participate as a validator in the future. In theory, this check incentivizes the system to validate only truthful transactions.

Potential Issues We discussed the “Nothing At Stake” problem in which a bad acting Validator places bets on multiple forks so they theoretically always win out in the end.

One issue that can arise is the "nothing-at-stake" problem, wherein block generators have nothing to lose by voting for multiple blockchain histories, thereby preventing consensus from being achieved. Because unlike in proof-of-work systems, there is little cost to working on several chains.

Many have attempted to solve these problems:

Peercoin is the first cryptocurrency that applied the concept of PoS. In its early stages, it used centrally broadcast checkpoints signed under the developer's private key. No blockchain reorganization was allowed deeper than the last known checkpoints. Checkpoints are opt-in as of v0.6 and are not enforced now that the network has reached a suitable level of distribution.
Ethereum's suggested Slasher protocol allows users to "punish" the cheater who forges on top of more than one blockchain branch. This proposal assumes that one must double-sign to create a fork and that one can be punished for creating a fork while not having stake. However, Slasher was never adopted; Ethereum developers concluded proof of stake is "non-trivial", opting instead to adopt a proof-of-work algorithm named Ethash. It is planned to be replaced by a different PoS protocol called "Casper".
Nxt's protocol only allows reorganization of the last 720 blocks. However, this merely rescales the problem: a client may follow a fork of 721 blocks, regardless of whether it is the tallest blockchain, thereby preventing consensus.
Hybrid "proof of burn" and proof of stake. Proof-of-burn blocks act as checkpoints, have higher rewards, contain no transactions, are more secure, and anchor both to each other and to the PoS chain but are more expensive.
Decred's hybrid proof-of-work and proof-of-stake, in which proof of stake is an extension dependent on the proof-of-work timestamping, based on the "proof of activity" proposal, which aims to solve the nothing-at-stake problem by having proof-of-work miners mining blocks and proof-of-stake acting as a second authentication mechanism.

Proposed Solutions

Slasher Strategy - which entails penalizing validators if they simultaneously create blocks on multiple chains.
Punisher Strategy - which simply punishes validators for creating blocks on the wrong chain. In this method, Validators will be motivated to be selective and conscious about the blockchain in which they put their stake.

Delegated Byzantine Fault Tolerance (dBFT)

dBFT uses a system similar to a democracy where Ordinary Nodes the system vote on representative Delegate Nodes to decide which blocks should be added to the blockchain. When it’s time to add a block, a Speaker is randomly assigned from the group of Delegates to create a new block and propose the new block. 66.66% of delegates need to approve on the block for it to pass.

Potential Issues Two issues we explored were the case of the Dishonest Speaker and the Dishonest Delegate.

Dishonest Speaker There is always a chance the Speaker, who is randomly selected from the Delegates, could be dishonest or malfunction. In this situation, the network needs to rely on honest delegates to vote the proposed block down so it doesn’t reach 66% approval. It is up to users of protocol who vote on Delegates, to find out which delegates are not trustworthy and vote on other delegates that are truthful.

Dishonest Delegate In this case, the chosen Speaker is honest but there are Dishonest Delegates in the system meaning even if they receive a proposal for new block that is faulty, they can say it is valid. If it is a minority of delegates that are dishonest, the block will not make it and new speaker is elected.

Proof of Existence

The concept that publicly proving and authenticating any digital asset on the blockchain by verifying its hash.

Helps you demonstrate data ownership without revealing actual data. This is useful for things like copyrighted material or patents.
Checks for the integrity of your digital asset. Any proof of existence will recognize your document FOREVER. Even the slightest difference will be recognized allowing you to be sure your asset hasn’t changed.
Provides document Time stamping. You can use this to prove certain information existed at a certain time.
Certifies the existence of the document without the need for a central authority.

Wallets

Blockchain identities are made up of a few important tools like wallets, addresses, and keys. Not only are there a few of these different tools creating our identity, it's also possible to implement them in different ways.

Non-deterministic: (random wallets) A wallet where private keys are generated from random numbers.
Sequentia deterministic: A wallet where addresses, private keys, and public keys can be traced back to their original seed words.
Hierarchical Deterministic Wallet: An advanced type of deterministic wallet that contains keys derived in a tree structure.

Block

A typical Block is comprised of 2 things:

Block Header
Body

Block Header

Block header is composed of the following things:

Block Number : A number to identify where it is located in the chain. 1st block in the chain is also known as genesis block
Block Hash : A hash representation of the data. SHA 256 is used to produce the hash.
Number Of Transactions
Height : The number of blocks preceding a particular block on a block chain. For example, the genesis block has a height of zero because zero block preceded it.
Timestamp
Merkle Root : Hash that represents every transaction in the block. This is also used in reverse to reconstruct the netire transaction data.
Previous Block : To identify what comes before the current block.
Difficulty : It is defined by number of zeros in the hash value. More zero's means more difficulty.
Bits
Size (bytes) : Amount of space the block has to hold the data. Also remember, the size of all the block remains the same for every block in a given blockchain.
Nonce : An arbitrary number that can be used once to find the hash
Version
Next Block : To identify what comes after the current block

Body

Body is made up of transaction data that comprises of the following:

Hash
Size
Received Time
Mined Time
Included in Block
LockTime
Coinbase
Amount send : Usually the money sent + miner's transaction fee
Amount returned back to sender

To see what a block is made up of visit https://blockexplorer.com/ and enter 520607 in the search and see for yourself.

Bitcoin and Bitcoin core

Main distinctions are Bitcoin as a network and Bitcoin as a software.

Bitcoin: Network of bitcoin users creating and validating transactions
Bitcoin Core: Implementation of bitcoin that encompasses all of the software behind bitcoin
Debug Console: Tool that allows you to interact with data on the bitcoin blockchain

Bitcoin Core

Connect to the Network
Validate the blockchain
Send and Receive bitcoins

Bitcoin Core Networks

Bitcoin Mainnet: Primary Network where live transactions take place
Bitcoin Testnet: Alternative Bitcoin blockchain that provides a test environment for applications
Bitcoin Regnet: Alternative test network for testing bitcoin applications

	Mainnet	Testnet	Regnet
Purpose	Live	Testing	Testing
Coins	Value	No Value	No Value
Peers	Yes	Yes	Yes

More focussed comparison

	Mainnet	Testnet	Regnet
Purpose	Production	Testing	Testing
Environment	Public	Public	Private
Peers	Entire Network	Testers	None
Size	~200 GB	~14 GB	~0 GB
Block Creation	10 minutes	10 minutes	Instantly
Value	Full Value	No Value	No Value
Public key prefix	1	m or n	m or n
Block difficuly	Full	Half of mainnet	None

Setting up Bitcoin Core

You need to have atleat 200 GB of storage if you want to install the entire bitcoin transaction data. Alternatively, you can also configure the bitcoin core as a testnet which requires only 20 GB of storage space.

If you do this, rather than the software syncing to the mainnet, which can be up to 200GB or more, you’ll sync to the testnet which is approximately 15GB, or the regnet which is 0GB.

Reduce Storage

It is possible to configure your node to to run in pruned mode in order to reduce storage requirements. This can reduce the disk usage from over 200GB to around 5GB.

To enable block pruning set prune=N on the command line or in bitcoin.conf, where N is the number of MiB to allot for raw block and undo data.

A value of 0 disables pruning. The minimal value above 0 is 550. Your wallet is as secure with high values as it is with low ones. Higher values merely ensure that your node will not shut down upon blockchain reorganizations of more than 2 days - which are unlikely to happen in practice. In future releases, a higher value may also help the network as a whole because stored blocks could be served to other nodes.

As a quick summary here are the steps you’ll go through.

create a bitcoin.conf file in your bitcoin installation directory.
In this file, write prune=550.

Some useful links

https://en.bitcoin.it/wiki/Data_directory
https://jlopp.github.io/bitcoin-core-config-generator/
Parameters in bitcoin.conf - https://www.youtube.com/watch?v=W54aRszkEOI&t=32s

Debug Console

This is a powerful tool that helps you explore data within the bitcoin blockchain.

After setting up the bitcoin core you have 2 ways to navigate and interact via terminal.

Via Client : Go to the help in the menu bar and open Debug Console. Enter help
Via Cli : Go to terminal and type bitcoin-cli --help

You notice several categories:

Blockchain
Control
Generating
Mining
Network
Rawtransactions
Util
Wallet
Zmq

For a list of available commands and their definitions, you can also go to the bitcoin wiki https://en.bitcoin.it/wiki/Original_Bitcoin_client/API_calls_list

1. Block Chain Commands

getblockchaininfo: Returns various state information about blockchain processing.
getblockcount: Returns the number of blocks in the blockchain.
verifychain: Verifies blockchain database.

Key Terms:
- Block Chain: Digital ledger that contains the entire history of transactions made on the network.

2. Hash Commands

getblockhash: Returns hash of a block at the block number provided
getnetworkhashps: Returns an estimated network hashes per second based on a specified number of recent blocks.
getbestblockhash: Returns the hash of the best block.

Key Terms:
- Hash Value: A digital fingerprint for information
- Best Block: Most recent block that you’ve synced to with your local copy of the blockchain.

3. Block Commands

getblock: Returns details of block information.
getblockheader: Returns information about the block header.
generate: Immediately mines the specified number of blocks to an address in the wallet.

Key Terms:
- Block: A container that holds a list of transactions to be added to the blockchain.

4. Wallet Commands

getwalletinfo: Returns an object containing various information about a wallet’s state.
listwallets: Returns a list of currently loaded wallets.
walletpassphrasechange: Change the wallet passphrase.

Key Terms:
- Wallet: Software that stores private keys that give access to a bitcoin balance

5. Mempool Commands

getmempoolinfo: Returns details on the active state of the transaction memory pool.
getrawmempool: Returns all transaction details in the memory pool.
getmempoolentry: Returns mempool data for a given transaction.

Key Terms:
- Mempool: Waiting place for all unconfirmed transactions before they are added to the blockchain.

6. Transaction Commands

getchaintxstats: Compute statistics about the total number and rate of transactions in the chain
getrawtransaction: Returns raw transaction data
listtransactions: Returns a list of transactions for a given account

Key Terms:
- Transaction: Record of any movement of funds that takes place on the network.

7. Signature Commands

signrawtransaction: Sign inputs for a raw transaction.
signmessage: Sign message with the private key of an address

Resources:
Learn more about the Raw transaction format

Key Terms:
- Signature: Establishes proof of ownership for each transaction on the blockchain.

Changes in Bitcoin Core version v0.17.0.1
signrawtransaction is deprecated and will be fully removed in v0.18. 
To use signrawtransaction in v0.17, restart bitcoind with -deprecatedrpc=signrawtransaction.

Projects should transition to using signrawtransactionwithkey and signrawtransactionwithwallet before upgrading to v0.18.

Try those commands using the help command:

help signrawtransactionwithkey
Sign inputs for raw transaction (serialized, hex-encoded). 
The second argument is an array of base58-encoded private keys that will be the only keys used to sign the transaction. 
The third optional argument (may be null) is an array of previous transaction outputs that this transaction depends on but may not yet be in the block chain.

help signrawtransactionwithwallet
Sign inputs for raw transaction (serialized, hex-encoded). 
The second optional argument (may be null) is an array of previous transaction outputs that this transaction depends on but may not yet be in the block chain

8. Network Commands

getnetworkinfo: Returns information about the state of the peer-to-peer network.
getpeerinfo: Returns data about each connection network node.
getconnectioncount: Returns the number of connections to other nodes.

Key Terms:
- Peer-to-peer network: A network of computers that allows information to be shared across users.

9. Mining Commands

getmininginfo: Returns an object that contains mining-related information.
getblocktemplate: Returns data needed to construct a block.
prioritisetransaction: Accepts the transaction into mined blocks at a higher or lower priority.

Transactions

Transactions encode the transfer of value between participants in the system. In more detail, a transaction is a data structure that encodes a transfer of value from a source of funds called an “input” to a destination called an “output”.

Inputs in one transaction are just the unspent outputs from another transaction. All inputs reference back to an output. Unspent Outputs is sometimes short-handed to UTXO.

Transaction Data Model

In a blockchain transaction are stored in double hashed form i.e. the initial raw information is put to SHA 256 twice.

For e.g. SHA 256 ( SHA 256 (raw) ) => b138360800cdc72248c3ca8dfd06de85913d1aac7f41b4fa54eb1f5a4a379081

Raw Transaction Data: (hex decimal format) 0100000001f3f6a909f8521adb57d898d2985834e632374e770fd9e2b98656f1bf1fdfd427010000006b48304502203a776322ebf8eb8b58cc6ced4f2574f4c73aa664edce0b0022690f2f6f47c521022100b82353305988cb0ebd443089a173ceec93fe4dbfe98d74419ecc84a6a698e31d012103c5c1bc61f60ce3d6223a63cedbece03b12ef9f0068f2f3c4a7e7f06c523c3664ffffffff0260e31600000000001976a914977ae6e32349b99b72196cb62b5ef37329ed81b488ac063d1000000000001976a914f76bc4190f3d8e2315e5c11c59cfc8be9df747e388ac00000000

Breakdown of Raw Transaction:

Version - All transactions include information about the Bitcoin Version number so we know which rules this transaction follows.
Input count - Which is how many inputs were used for this transaction
Input info - At a high level, it provides where the input is coming from and checks whether inputs can be used

Previous output hash - All inputs reference back to an output (UTXO). This points back to the transaction containing the UTXO that will be spent in this input. The hash value of this UTXO is saved in a reverse order here.
Previous output index - The transaction may have more than one UTXO which are referenced by their index number. The first index is 0.
Unlocking Script Size - This is the size of the Unlocking Script in bytes.
Unlocking Script - This is the hash of the Unlocking Script that fulfills the conditions of the UTXO Locking Script.
Sequence Number - This is a deprecated feature of bitcoin, currently set to ffffffff by default.

Output count - which tells us how many outputs were produced from this transaction.
Output info - At a high level, it provides how many BTC outputted and conditions for future spending.

Amount - The amount of Bitcoin outputted in Satoshis (the smallest bitcoin unit). 10^8 Satoshis = 1 Bitcoin.
Locking Script Size - This is the size of the Locking Script in bytes.
Locking Script - This is the hash of the Locking Script that specifies the conditions that must be met to spend this output.

Locktime - The locktime field indicates the earliest time or the earliest block a transaction can be added to the blockchain. If the locktime is non-zero and less than 500 million, it is interpreted as a block height and miners have to wait until that block height is reached before attempting to add it to a block. If the locktime is above 500 million, it is read as a UNIX timestamp which means the number of seconds since the date January 1st 1970. It is usually 0 which means confirm as soon as possible.

Create a raw transaction

View all unspent confirmed UTXO in the wallet
listunspent - Show all the unspent confirmed outputs in our wallet)
View Details about a Specific UTXO
gettxout - Get the details of this unspent output above
Create a Raw Transaction
createrawtransaction - Create a transaction
Decode the Raw Transaction (to double-check it went through correctly)
decoderawtransaction - View raw hex string that encodes the transaction details we supplied
Sign the Raw Transaction
signrawtransactionwithwallet
signrawtransaction - older versions than 0.17
Submit the Raw Transaction to the Network
sendrawtransction - Takes the raw hex string produced by signrawtransaction and returns a transaction hash (txid) as it submits the transaction on the network.
Query the TxID of the Transaction we sent
gettransaction - Query the TxID and view details. Similar to online block explorer

To build a simple blockchain

Some of the core ideas behind building a simple private blockchain are as follows:

Block Data Model
Create New Blocks
Store Blocks
Linking Blocks
Block Height and Timestamp
Level DB to Persist Data

First, lets discuss some key differences between public and private blockchain

	Public Blockchain	Private Blockchain
Permissions	Permissionless	Permissioned
Scalability	More Difficult	Simpler
Vulnerability	Less Vulnerable	More Vulnerable
Compliance	More Difficult	Simpler

Cross Chain Functionality: Allows a series of protocols that work in harmony to deliver decentralized applications

Do You Really Need A Blockchain?

Some things to consider before deciding on need for having blockchain in your app

Problem of Value Identification

Is there a need to share information, credentials or value with others?
Is trust a critical requirement to the process?
Do you need to prove to others you are transacting/reporting accurately?
Is there potential to monetize the data or digital asset in the value chain?
Who owns the problem? Individual or industry wide challenges?

Stakeholders buy in

Is there a network of stakeholders (i.e. more than 2)?
Is there a dependency on others for information?
Does more than one participant need to update the data?
Is there scope to open up the ecosystem to ancillary parties in the future?
Are you working with other industry players on any activities?

Technical considerations

Is there any ongoing need or future requirements for high data throughput?
Do you rely or use public data sources to make decisions?
Do you need to store a particularly rich/complex data structure?
Do you need to digitize assets in your value chain?
Do you need transaction privacy? Do you need anonymity?

Ethereum

Ethereum Virtual Machine

A VM that executes code
A runtime for smart contracts

Refer: http://ethdocs.org/en/latest/introduction/what-is-ethereum.html?highlight=EVM#ethereum-virtual-machine

Ethereum Acoounts

Externally owned accounts (EOA)

Tied to private key
Doesn't hold code
Maintains ether balance
Can send transactions
Call smart contracts
Transfer balance between accounts

Certificate accounts (CA)

Has code
Maintains ether balance
Executes code when triggered by transaction or messages
Manipulate storage

Account State Variables

Nonce : number of transactions on the account EOA - # transanction send to account address CA - # of smart contracts initiated by an account
Account balance : Total value of ether in wei
Storage hash : Root node of particia tree contains the contents in the account
Code hash : execute calls made to the account

Smart Contracts

Smart Contract Life Cycle

Develop - Write the contract and compile.
Deploy - Once deployed, the smart contract is un-editable. After deploying on any network (e.g. mainnet, testnet), you get an address specific to where contract was deployed.
Invoke - Use the address created after the smart contract is deployed to invoke contract to call functions on it.
Destroy - Once a contract has finished its purpose, it is important to destroy it so no one can call functions on it anymore. Remember that smart contracts cannot be edited once deployed. Also, often smart contracts are tied to funds, so keeping it around if it is no longer used puts these funds at risk.

Tools

Etherscan - Etherscan is a Block Explorer and Analytics Platform for Ethereum, a decentralized smart contracts platform.
Metamask - MetaMask is a bridge that allows you to visit the distributed web of tomorrow in your browser today. It allows you to run Ethereum dApps right in your browser without running a full Ethereum node.
Remix - A browser based IDE that allows you to write Solidity and deploy solidity contracts.
web3js - A collection of libraries that allow you to interact with a local or remote Ethereum node.
Infura - Gives you access to the Ethereum network without running your own local node.
Ganache - Application for launching local Ethereum blockchain instances.
Truffle - A development environment, testing framework, and asset pipeline meant to make your life easier as an Ethereum developer.
Geth - A golang implementation of the Ethereum blockchain used to set up a local full Ethereum node.

Solidity

Solidity is an object-oriented, high-level language for implementing smart contracts. Smart contracts are programs which govern the behaviour of accounts within the Ethereum state. API Reference: https://solidity.readthedocs.io/en/v0.5.4/ Use remix to write smart contracts on browser - https://remix.ethereum.org Video tutorial to get started with Solidity - https://www.youtube.com/playlist?list=PLS5SEs8ZftgUq-aMMYeKf8nPqHrNqa3Iu

Tokens

The idea that Ethereum not only has its own currency (Ether) but also has tokens on top of that which can act as currency themselves.

good read - https://blockgeeks.com/guides/ethereum-token/

Truffle commands

truffle develop- to run Truffle Development environment.
compile - to compile any changes in the Smart Contract.
migrate or migrate --reset - to migrate and create a fresh and clean Smart contract session.
migrate --reset --network rinkeby - to deploy in rinkeby network
test - to run the test cases.

Bullish

Rinkeby - Send test coins
Rinkeby.io - Rinkeby Live Stats
Metamask faucet - Metamask test coins

Community

https://forum.ethereum.org/categories/ethereum-js

SidoPillai/blockchain