Tricircle

Tricircle is a project for Openstack cascading solution, including the source code of Nova Proxy, Cinder Proxy, Neutron L2/L3 Proxy, Glance sync manager and Ceilometer Proxy(not implemented yet).

The project name "Tricircle" comes from a fractal. See the blog "OpenStack cascading and fractal" for more information.

Important to know

Only about 15k code lines developed for OpenStack cascading.
The source code now based on Juno is for PoC only. Refactory will be done constantly to reach OpenStack acceptance standard.
The Neutron cascading using the feature of provider network. But horizon doen't support provider network very well. So you have to use Neutron CLI to create a network.
Support L2 networking(VxLAN) across cascaded OpenStack, but only point2point remote host IP tunneling supported now.L2 networking through L2GW to reduce population traffic and simplify networking topology will be developed in the near future.
The L3 networking across casacaded OpenStack will set up tunneling network for piggy data path, useing GRE tunneling over extra_route to brige the router in different cascaded OpenStack.Therefore, the loca L2 network (VLAN,VxLAN) in one cascaded OpenStack can reach L2 network(VLAN,VxLAN) located in another cascaded OpenStack.
Glance cascading using Glance V2 API. Only CLI/pythonclient support V2 API, the Horizon doesn't support that version. So image management should be done through CLI, and using V2 only. Otherwise, the glance cascading cannot work properly.
Glance cascading is not used by default, eg, useing global Glance by default. If Glance cascading is required, configuration is required.

Key modules

Nova proxy

The hypervisor driver for Nova running on Nova-Compute node. Transfer the VM operation to cascaded Nova. Also responsible for attach volume and network to the VM in the cascaded OpenStack.
Cinder proxy

The Cinder-Volume driver for Cinder running on Cinder-Volume node.. Transfer the volume operation to cascaded Cinder.
Neuton proxy

Including L2 proxy and L3 proxy, Similar role like OVS-Agent/L3-Agent. Finish L2/L3-networking in the cascaded OpenStack, including cross OpenStack networking.
Glance sync

Synchronize image among the cascading and policy determined Cascaded OpenStacks

Patches required

Juno-Patches

Pacthes for OpenStack Juno version, including patches for cascading level and cacscaded level.

Feature Supported

Nova cascading Launch/Reboot/Terminate/Resize/Rescue/Pause/Un-pause/Suspend/Resume/VNC Console/Attach Volume/Detach Volume/Snapshot/KeyPair/Flavor
Cinder cascading Create Volume/Delete Volume/Attach Volume/Detach Volume/Extend Volume/Create Snapshot/Delete Snapshot/List Snapshots/Create Volume from Snapshot/Create Volume from Image/Create Volume from Volume (Clone)/Create Image from Volume
Neutron cascading Network/Subnet/Port/Router. Including L2/L3 networking across cascaded OpenStacks
Glance cascading Only support V2 api. Create Image/Delete Image/List Image/Update Image/Upload Image/Patch Location/VM Snapshot/Image Synchronization

Known Issues

Launch VM only support "boot from image", "boot from volume", "boot from snapshot"
Flavor only support new created flavor synchronized to the cascaded OpenStack, does not support flavor update synchronization to cascaded OpenStack yet.

Installation without Glance cascading

Prerequisites
- the minimal installation requires three OpenStack Juno installated to experience across cascaded OpenStacks L2/L3 function. The minimal setup needs four nodes, see the following picture:
- the cascading OpenStack needs two node, Node1 and Node 2. Add Node1 to AZ1, Node2 to AZ2 in the cascading OpenStack for both Nova and Cinder.
- It's recommended to name the cascading Openstack region to "Cascading" or "Region1"
- Node1 is all-in-one OpenStack installation with KeyStone and Glance, Node1 also function as Nova-Compute/Cinder-Volume/Neutron OVS-Agent/L3-Agent node, and will be replaced to be the proxy node for AZ1.
- Node2 is general Nova-Compute node with Cinder-Volume, Neutron OVS-Agent/L3-Agent function installed. And will be replaced to be the proxy node for AZ2
- the all-in-one cascaded OpenStack installed in Node3 function as the AZ1. Node3 will also function as the Nova-Compute/Cinder-Volume/Neutron OVS-Agent/L3-Agent in order to be able to create VMs/Volume/Networking in this AZ1. Glance is only required to be installed if Glance cascading needed. Add Node3 to AZ1 in the cascaded OpenStack both for Nova and Cinder. It's recommended to name the cascaded Openstack region for Node3 to "AZ1"
- the all-in-one cascaded OpenStack installed in Node4 function as the AZ2. Node3 will also function as the Nova-Compute/Cinder-Volume/Neutron OVS-Agent/L3-Agent in order to be able to create VMs/Volume/Networking in this AZ2. Glance is only required to be installed if Glance cascading needed.Add Node4 to AZ2 in the cascaded OpenStack both for Nova and Cinder.It's recommended to name the cascaded Openstack region for Node4 to "AZ2"
Make sure the time of these four nodes are synchronized. Because the Nova Proxy/Cinder Proxy/Neutron L2/L3 Proxy will query the cascaded OpenStack using timestamp, incorrect time will lead to VM/Volume/Port status synchronization not work properly.

Register all services endpoint in the global shared KeyStone.

Make sure the 3 OpenStack can work independently before cascading introduced, eg. you can boot VM with network, create volume and attach volume in each OpenStack. After verify that 3 OpenStack can work independently, clean all created resources VM/Volume/Network.

After all OpenStack installation is ready, it's time to install Juno pathces both for cascading OpenStack and cascaded OpenStack, and then replace the Nova-Compute/Cinder-Volume/Neutron OVS-Agent/L3-Agent to Nova Proxy / Cinder Proxy / Neutron l2/l3 Proxy.
Juno pachtes installation step by step

Node1

Patches for Neutron - neutron_cascading_l3_patch

This patch is to enable cross cascaded OpenStack L3 routing over extra route.The mapping between cascaded OpenStack and it's onlink external network which is used for GRE tunneling data path

Navigate to the folder
```
cd ./tricircle/juno-patches/neutron/neutron_cascading_l3_patch
```
follow README.md instruction to install the patch

Node3

Patches for Cinder - timestamp-query-patch

This patch is to make the cascaded Cinder being able to execute query with timestamp filter, but not to return all objects.

Navigate to the folder
```
cd ./tricircle/juno-patches/cinder/timestamp-query-patch
```
follow README.md instruction to install the patch
Patches for Neutron - neutron_timestamp_cascaded_patch

This patch is to make Neutron being able to provide timestamp based port query.

Navigate to the folder
```
cd ./tricircle/juno-patches/neutron/neutron_timestamp_cascaded_patch
```
follow README.md instruction to install the patch
Patches for Neutron - neutron_cascaded_l3_patch

This patch is to enable cross cascaded OpenStack L3 routing over extra route..

Navigate to the folder
```
cd ./tricircle/juno-patches/neutron/neutron_cascaded_l3_patch
```
follow README.md instruction to install the patch

Node4

Patches for Cinder - timestamp-query-patch

This patch is to make the cascaded Cinder being able to execute query with timestamp filter, but not to return all objects.

Navigate to the folder
```
cd ./tricircle/juno-patches/cinder/timestamp-query-patch
```
follow README.md instruction to install the patch
Patches for Neutron - neutron_timestamp_cascaded_patch

This patch is to make Neutron being able to provide timestamp based port query.

Navigate to the folder
```
cd ./tricircle/juno-patches/neutron/neutron_timestamp_cascaded_patch
```
follow README.md instruction to install the patch
Patches for Neutron - neutron_cascaded_l3_patch

This patch is to enable cross cascaded OpenStack L3 routing over extra route..

Navigate to the folder
```
cd ./tricircle/juno-patches/neutron/neutron_cascaded_l3_patch
```
follow README.md instruction to install the patch

Proxy installation step by step

Node1

Nova proxy

Navigate to the folder
```
cd ./tricircle/novaproxy
```
follow README.md instruction to install the proxy. Please change the configuration value in the install.sh according to your environment setting
Cinder proxy

Navigate to the folder
```
cd ./tricircle/cinderproxy
```
follow README.md instruction to install the proxy. Please change the configuration value in the install.sh according to your environment setting
L2 proxy

Navigate to the folder
```
cd ./tricircle/neutronproxy/l2-proxy
```
follow README.md instruction to install the proxy. Please change the configuration value in the install.sh according to your environment setting
L3 proxy

Navigate to the folder
```
cd ./tricircle/neutronproxy/l3-proxy
```
follow README.md instruction to install the proxy. Please change the configuration value in the install.sh according to your environment setting

Node2

Nova proxy

Navigate to the folder
```
cd ./tricircle/novaproxy
```
follow README.md instruction to install the proxy. Please change the configuration value in the install.sh according to your environment setting
Cinder proxy

Navigate to the folder
```
cd ./tricircle/cinderproxy
```
follow README.md instruction to install the proxy. Please change the configuration value in the install.sh according to your environment setting
L2 proxy

Navigate to the folder
```
cd ./tricircle/neutronproxy/l2-proxy
```
follow README.md instruction to install the proxy. Please change the configuration value in the install.sh according to your environment setting
L3 proxy

Navigate to the folder
```
cd ./tricircle/neutronproxy/l3-proxy
```
follow README.md instruction to install the proxy. Please change the configuration value in the install.sh according to your environment setting

Upgrade to Glance cascading

Prerequisites
- To experience the glance cascading feature, you can simply upgrade the current installation with several step, see the following picture:

Node1

Patches for Glance - glance_location_patch

This patch is to make the glance being able to handle http url location. The patch also insert the sync manager to the chain of responsibility.

Navigate to the folder
```
cd ./tricircle/juno-patches/glance/glance_location_patch
```
follow README.md instruction to install the patch
Patches for Glance - glance_store_patch

This patch is to make the glance being able to handle http url location.

Navigate to the folder
```
cd ./tricircle/juno-patches/glance_store/glance_store_patch
```
follow README.md instruction to install the patch
Sync Manager

Navigate to the folder
```
cd ./tricircle/glancesync
```
modify the storage scheme configuration for cascading and cascaded level
```
vi ./tricircle/glancesync/etc/glance/glance_store.yaml
```
follow README.md instruction to install the sync manager. Please change the configuration value in the install.sh according to your environment setting, espeically for configuration: sync_enabled=True sync_server_port=9595 sync_server_host=127.0.0.1

Node3

Glance Installation

Please install Glance in the Node3 as the casacded Glance. Register the service endpoint in the KeyStone. Change the glance endpoint in nova.conf and cinder.conf to the Glance located in Node3

Node4

Glance Installation

Please install Glance in the Node4 as the casacded Glance. Register the service endpoint in the KeyStone Change the glance endpoint in nova.conf and cinder.conf to the Glance located in Node4

Configuration

Change Nova proxy configuration on Node1, setting the "cascaded_glance_flag" to True and add "cascaded_glance_url" of Node3 configurantion according to Nova-proxy README.MD instruction
Change Cinder proxy configuration on Node1, setting the "glance_cascading_flag" to True and add "cascaded_glance_url" of Node3 configurantion according to Nova-proxy README.MD instruction
Change Nova proxy configuration on Node2, setting the "cascaded_glance_flag" to True and add "cascaded_glance_url" of Node4 configurantion according to Nova-proxy README.MD instruction
Change Cinder proxy configuration on Node2, setting the "glance_cascading_flag" to True and add "cascaded_glance_url" of Node4 configurantion according to Nova-proxy README.MD instruction

Experience Glance cascading

Restart all related service
Use Glance V2 api to create Image, Upload Image or patch location for Image. Image should be able to sync to distributed Glance if sync_enabled is setting to True
Sync image only during first time usage but not uploading or patch location is still in testing phase, may not work properly.
Create VM/Volume/etc from Horizon

BruceChiZhang/tricircle