Elijah: Cloudlet Infrastructure for Mobile Computing

A cloudlet is a new architectural element that arises from the convergence of mobile computing and cloud computing. It represents the middle tier of a 3-tier hierarchy: mobile device - cloudlet - cloud. A cloudlet can be viewed as a "data center in a box" whose goal is to "bring the cloud closer". Please visit our website at Elijah page.

License

All source code and documentation except modified-QEMU listed below are under the Apache License, Version 2.0.

To enable on-demand fetching of the virtual machine, we use modified-QEMU, which is distributed under GPLv2.
The binary of this modified-QEMU will be automatically downloaded at installation script upon your agreement.
- Repository for modified-QEMU
- Download link for the binary

A copy of this license is reproduced in the LICENSE file.

Before you start

This code is about rapid provisioning of a custom VM(Virtual Machine) to cloudlet using VM synthesis. This does not include any code for mobile applications, rather it provides functions to create VM overlay and perform VM Synthesis that will rapidly reconstruct your custom VM at an arbitrary computer.

Please read Just-in-Time Provisioning for Cyber Foraging to understand what we do here and find the detail techniques.

The key to rapid provisioning is the recognition that a large part of a VM image is devoted to the guest OS, software libraries, and supporting software packages. The customizations of a base system needed for a particular application are usually relatively small. Therefore, if the base VM already exists on the cloudlet, only its difference relative to the desired custom VM, called a VM overlay, needs to be transferred. Our approach of using VM overlays to provision cloudlets is called VM synthesis. A good analogy is a QCOW2 file with a backing file. You can consider VM overlay as a QCOW2 file and Base VM as a backing file. The main difference is that VM synthesis includes both disk and memory state and it is much more efficient in generating diff and reconstructing suspended state.

Installing

You will need:

qemu-kvm
libvirt-bin
gvncviewer
python-libvirt
python-xdelta3
python-bsdiff4
python-dev (to pip install msgpack)
liblzma-dev (for pyliblzma)
apparmor-utils (for disable apparmor for libvirt)
libc6-i386 (for extracting free memory of 32 bit vm)
libxml2-dev libxslt1-dev (for overlay packaging)
python libraries at requirements.txt

To install

run a installation script

$ sudo apt-get install fabric openssh-server
$ fab install (Type your Ubuntu account password when it is asked)

Tested platforms

We have tested at Ubuntu 16.04 LTS 64-bit (as well as their derivatives such as Kubuntu).

This version of Cloudlet has several dependencies on other projects for further optimization, and currently we include this dependency as a binary. Therefore, we recommend you to use Ubuntu 16.04 LTS 64-bit.

How to use

Import Base VM

First thing you need is a Base VM. Base VM is a pre-loaded element and you can think it as a golden image similar to Amazon AMI. We provide sample base VM of Ubuntu 12.04 32-bit server for easybootstrapping. Download sample base VM at:

For Ubuntu 16.04: Base VM for Ubuntu-12.04.01-i386-Server (Ubuntu account for VM : cloudlet, password: cloudlet)

Then, you can import this base VM using command line tool, cloudlet.

> $ cloudlet import-base ./precise-hotplug-new.zip  
> INFO     create directory for base VM  
> INFO     Decompressing Base VM to temp directory at /tmp/cloudlet-base-k7ANqB  
> INFO     Place base VM to the right directory  
> INFO     Register New Base to DB  
> INFO     ID for the new Base VM: abda52a61692094b3b7d45c9647d022f5e297d1b788679eb93735374007576b8
> INFO     Success

You can check the imported base VM by

> $ cloudlet list-base  
> hash value<code>&nbsp;&nbsp;&nbsp;&nbsp;</code>path  
> \------------------------------------------------------------------------------------------  
> abda52a<code>&nbsp;&nbsp;&nbsp;&nbsp;</code>/home/krha/.cloudlet/abda52a/precise.raw  
> \------------------------------------------------------------------------------------------

Perform VM synthesis using a sample VM overlay.

First, launch the VM synthesis server at Cloudlet.

> $ synthesis_server  
> INFO     --------------------------------------------------  
> INFO     * Base VM Configuration  
> INFO      0 : /home/krha/.cloudlet/abda52a61692094b3b7d45c9647d022f5e297d1b788679eb93735374007576b8/precise.raw (Disk 8192 MB, Memory 1040 MB)  
> INFO     --------------------------------------------------  
> INFO     * Server configuration  
> INFO      - Open TCP Server at ('0.0.0.0', 8021)  
> INFO      - Disable Nagle(No TCP delay)  : 1  
> INFO     --------------------------------------------------

Then, you can test VM synthesis using a sample VM overlay URL of htop command. This process will resume a backend VM that is running a htop command. VM overlay URL synthesis_client -s localhost -u https://storage.cmusatyalab.org/cloudlet-vm/overlay-htop.zip

Demo for the Fluid simulation's back-end server (YouTube) is temporarily unavailable. Instead, you can create your own VM overlay following How To Create VM Overlay.

You can perform VM synthesis using sample VM overlay

> $ synthesis_client -s localhost -u https://storage.cmusatyalab.org/cloudlet-vm/overlay-htop.zip

For more details, check out help command synthesis_client -h

If VM synthesis is successful, you will see a screen-shot like

The synthesized VM shows a running htop command that was saved in the VM overlay. Note that synthesis_client is a TCP client, so you can execute it at a different machine.

If the VM synthesis is failed showing a screen-shot like below, it is due to the memory snapshot compatibility of the base VM.

This kernel panic is causes by CPU flag incompatibility coming from QEMU side. We do our best to make a VM memory snapshot work across heterogeneous host machine, but it can failed in an old machine. In this case, you need to create your own base VM (See more at How to create your own Base VM)

VM synthesis in different ways

In addition to using a synthesis server and a client, you can perform VM synthesis 1) using command line tool and 2) using an Android client.

1) Command line tool

    > $ cloudlet synthesis /path/to/base_disk.img /path/to/overlay.zip

2) Android client

We have sample Android client at $HOME/android/android and you can import
it to ``Eclipse`` as an Android project. This client program will
automatically find nearby Cloudlet using UPnP if both client and Cloudlet
are located in same broadcasting domain (e.g. sharing WiFi access point).
Otherwise, you can manually specify IP address of the Cloudlet by clicking
cancel in discovery pop-up.

Once installing application at your phone, you have to copy your VM overlay
to Android phone. For example with Fluid overlay example, download VM
overlay from <a
href=https://storage.cmusatyalab.org/cloudlet-vm/overlay-fluid-portable.zip
target="_blank">https://storage.cmusatyalab.org/cloudlet-vm/overlay-fluid-portable.zip</a>
and **unzip it**, then copy both overlay-meta and overlay-blob_1.xz file to
/sdcard/Cloudlet/overlay/fluid/ directory. This directory name, _fluid_,
will be appeared to your Android application when you're asked to select
``Overlay VM``. Right directory name is important since the directory name
will be	saved as appName in internal data structure and being used to
launch associated mobile application after finishing ``VM synthesis``.

Please recall that the VM synthesis client is designed for provisioning a custom back-end server program at cloudlet and you need to launch your mobile application after finishing VM synthesis. This client application will communicate with the back-end server you just provisioned. To launch mobile application after VM synthesis, we use Android Activity launcher and the directory name is used as an index for associated mobile application. See more details at handleSucessSynthesis() method at CloudletConnector.java file.

Sample application: Fluid Simulation

Fluid Simulation is an interactive fluid dynamics simulation, that renders a liquid sloshing in a container on the screen of a phone based on accelerometer inputs. The application back-end runs on Linux and performs a smoothed particle hydrodynamics physics simulation using 2218 particles, generating up to 50 frames per second. The structure of this application is representative of real-time (i.e., not turn-based) games.
Doyub Kim is a primary contributor of this application.
Video demo
- <a href=https://www.youtube.com/watch?v=f9MN-kvG_ko target="_blank">Using Cloudlet
- <a href=https://www.youtube.com/watch?v=hWc2fpejfiw target="_blank">Using Amazon EC2 West
- <a href=https://www.youtube.com/watch?v=aSjQnfkUoU8 target="_blank">Using Amazon EC2 Asia
~~Code (temporarily unavailable)~~
- ~~Binary back-end server: $ HOME/test/app-server/fluid-bin32/~~
- Android client source code: $HOME/android/android_fluid/
- Python client source code: $HOME/test/app-client/scripts/graphics_client.py
  
  $ ./graphics_client.py -s localhost -p 9093 -i acc_input_50sec
- ~~VM overlay for the back-end server: https://storage.cmusatyalab.org/cloudlet-vm/overlay-fluid-portable.zip~~

Details Usage

How to create your own Base VM

You can also create your own base VM from a regular VM disk image. Here the regular VM disk image means a raw format virtual disk image you normally use for KVM/QEMU or Xen.

> $ cloudlet base /path/to/base_disk.img
> % Use raw file format virtual disk

This command will launch GUI (VNC) connecting to your guest OS and the code will start creating base VM right after you close VNC window. So, please close the VNC window when you think it's right point to snapshot the VM as a base VM (typically you close it after booting up). Then, it will generate snapshot of the current states for both memory and disk and save that information to DB. You can check list of base VM by

> $ cloudlet list-base

How to create VM overlay

Now you can create your customized VM based on top of base VM

> $ cloudlet overlay /path/to/base\_disk.img  
> % Path to base_disk is the path for virtual disk you used earlier  
> % You can check the path by "cloudlet list-base"

This will launch VNC again with resumed base VM. Now you can start making any customizations on top of this base VM. For example, if you're a developer of face recognition back-end server, we will install required libraries, binaries and finally start your face recognition server. After closing the GUI windows, cloudlet will capture only the change portion between your customization and base VM to generate VM overlay that is a minimal binary for reconstructing your customized VM.

VM overlay is using zip container and inside of the zip file there are 2 types of files; 1) overlay-meta file ends with .overlay-meta, 2) compressed overlay blob files ends with .xz

If your application need specific TCP/UDP port to communicate with a client, then you can make a port forwarding using -redir parameter as below.

> $ cloudlet overlay /path/to/base_disk.img -- -redir tcp:2222::22 -redir tcp:8080::80

This will forward client connection at host port 2222 to VM's 22 and 8080 to 80, respectively.

Note: If you have experience kernel panic error like this, You should upgrade your Linux kernel to at least 3.13.0 version. This bug is fixed since kernel version 3.13.0. If you cannot upgrade the kernel, then please avoid using EPT. You can disable EPT as follows:

> $ sudo modprobe -r kvm_intel
> $ sudo modprobe kvm_intel "ept=0"

Directories

HOME
  ├── bin: executable binaries such as command line tool, VM synthesis server and client
  │   
  ├── elijah: Cloudlet provisioning code using VM synthesis
  │     └─ test: unittest
  │   
  ├── cloudletfs: FUSE file system for creating VM overlay and performing early start 
  │               optimization at VM Synthesis
  │   
  ├── android: Android client
  │     ├─ android: main android client for VM synthesis
  │     ├─ android_fluid: fluid simulation client used in demo videos
  │     │	├─ YouTube: Using Amazon EC2 West
  │     │	└─ YouTube: Using Cloudlet
  │     └─ android_ESVMRecogn, android_ESVMTrainer: under development
  │
  ├── test: Test applications' code
  │     ├─ app-client 
  │     │	├─ scripts: client codes for each test application
  │     │	└─ batch_files: batch scripts to test application using VM synthesis on x86 (not Android)
  │     └─ app-server: server binary for each test application
  │
  ├── examples: Example application for VM Synthesis
  │     └─ gabriel-lego: Lego Wearable Cognitive Assistance
  │
  └── fabric.py: installation script using Fabric

Troubleshooing

KVM permission error at the first run

failed to initialize KVM: Permission denied No accelerator found!

Error, make sure previous VM is closed and check QEMU_ARGUMENT

Failed to create overlay

Please logout your session and re-login.

Fuse permission error at the first run

Make sure the current user to have fuse permission. The qemu-kvm library changes fuse access permission while it's being installed, and the permission is recovered if you reboot the host. This is a known bug in qemu-kvm installation script (https://bugs.launchpad.net/ubuntu/+source/udev/+bug/1152718), so you can either reboot the machine to have valid fuse permission of just revert the permission manually as bellow.

      > $ sudo chmod 644 /etc/fuse.conf  
      > $ sod sed -i 's/#user_allow_other/user_allow_other/g' /etc/fuse.conf

blakerouse/elijah-provisioning