BOINCVM ======= BOINC + VMController + VirtualBox + CERNVM = Volunteer Cloud Computing 1. Setup ======== In this section, we discuss how to setup the testing env. I use Ubuntu Linux 10.04. All cmds below shall work fine on Ubuntu... 1.1 VirtualBox ============== Install VirtualBox 3.1+ with VBoxGuestAdditions.iso (To install drivers etc.). And, with following specs provided here: http://cernvm.cern.ch/cernvm/index.cgi?page=VirtualBoxQuickStart Pl. find my VirtualBox configuration files in ./share (These config files live in ~/.VirtualBox on Linux...) 1.2 CERNVM ========== CernVM can be downloaded here: http://cernvm.cern.ch/cernvm/ I use CernVM 2.0.7 1.3 Dependencies ================ First install python 2.4+ with easy_install: sudo apt-get install python ipython easy_install Install VMController dependencies: sudo easy_install twisted stomper netifaces coilmq simplejson 1.3.1 Chirp =========== You are required to download chirp (http://www.cse.nd.edu/~ccl/software/chirp/) binaries from: http://www.cse.nd.edu/~ccl/software/download.shtml I've used: http://www.cse.nd.edu/~ccl/software/files/cctools-3.0.1-i686-linux-2.6.tar.gz The binaries can be found here: ./cctools-3.0.1-i686-linux-2.6 The chirp bin path is to configured in the HostConfig.cfg file (host/boincvm_host/HostConfig.cfg) (use relative path as ../../cctools-3.0.1-i686-linux-2.6/bin ) 1.3.2 BOINC source code ======================= On unix, the boinc (client and server) source can be downloaded from the svn repo @ berkeley: svn co http://boinc.berkeley.edu/svn/tags/boinc_core_release_6_10_56 svn co http://boinc.berkeley.edu/svn/branches/server_stable Compiled binaries of client can be downloaded from the boinc website: http://boinc.berkeley.edu/ BOINC Server VM image can be used to skip compile/build/deloy steps: http://boinc.berkeley.edu/trac/wiki/VmServer In case, Boinc Server is to be compiled, make sure you have following dependencies: m4 make autoconf automake1.9 gcc-4.1 gcc g++-4.1 pkg-config libtool subversion vim apache2-mpm-prefork libapache2-mod-php5 mysql-client-5.0 mysql-server-5.0 php5-mysql php5-cli php5-gd phpmyadmin python-mysqldb libmysql++-dev libssl-dev freeglut3-dev libxmu-dev libxi-dev libjpeg62-dev --- Check out the stable source code && cd <to it> Do: ./_autosetup ./configure make 1.3.3 BOINC standalone mode =========================== BOINC applications can be compiled and tested in the BOINC client's standalone mode. This does not require a developer to setup the server etc. To do that, boinc core libraries must be compiled (in static mode) from source. This requires packages, install the dependencies as (for graphics): sudo apt-get install libglut3-dev libjpeg-dev build-essential On Linux, do: svn co http://boinc.berkeley.edu/svn/tags/boinc_core_release_6_10_56 ./_autosetup ./configure --disable-client --disable-server --enable-static LDFLAGS=-static-libgcc --disable-manager make After successful completion, goto samples folder: cd ./samples/example_app make In this directory, create a file called "in", with some text written. Now, run the compiled program: ./uc2 This will create an output file "out" with all the characters uppercase and a error file "stderr.txt" When uc2 runs for the first time, it will create three files: boinc_finish_called, boinc_uppercase_0 and init_data.xml (Info about system etc) You may try out the graphics example as well, do: make uc2_graphics; ./uc2_graphics etc... 1.4 PYTHONPATH ============== PYTHONPATH is a environment variable that python searches while loading modules. To run the VMController, the PYTHONPATH must be setup correctly. Find the configuration file, pypath.conf in the root. To load the var, run: source pypath.conf If you want to reset PYTHONPATH, run the script with any arg: source pypath.conf 1 1.5 Dependencies for the VM ========================== Run CERNVM image, on VirtualBox and make sure it (the vm config) has two network adapters. (See /share/VirtualBox.xml and /share/CERN VM 2.0.7.xml for pointers...) On the terminal (as root) install boinc-group: su conary install boinc-group 1.6 Distribute ============== Distribute (a fork of setup tools) is suggested to be used to make installers. On Ubuntu install, distribute as: curl -O http://python-distribute.org/distribute_setup.py sudo python distribute_setup.py sudo easy_install pip 1.7 Virtual Env ================ Virtual Env lets us create a virtual sandbox containing our python app... Install virtualenv: (you may use pip) sudo easy_install virtualenv To create such a folder: virtualenv --distribute <Name of the Folder> To create a sandbox without inheriting any of the installed packages on your system: virualenv --no-site-packages --distribute <dir-name> This will create a distribute enable virtualenv... Now, activate it using: Linux/Mac: source ./bin/activate Windows: .\Scripts\activate.bat To deactivate: Linux/Mac: deactivate Windows: .\Scripts\deactivate.bat To install py eggs: easy_install <path to eggs...> 2. Manual Testing ================= This section will try to describe methods used in testing. 2.1 Running VMController ======================== cd to ./host/boincvm_host and run: python boinc-host-controller.py HostConfig.cfg 2.2 XMLRPC Object ================= Run ipython or a python script that does something like following: import xmlrpclib p = xmlrpclib.ServerProxy('http://localhost:8080') p.help() #list all interfaces... vm = p.listRunningVMs()[0] p.ping(vm) p.getState(vm) p.listRunningVMs() #Copy files.... id = p.cpFileToVM(vm, '~/test.txt', '') NOTE: It was found that files got copied to /var/boinc in the VirtualMachine (if the third parameter was left blank) 3. VMWrapper ============ The VMWrapper is a python program, which runs on the lines of BOINC wrapper (which is for supporting legacy apps). The VMWrapper, is the executable which will run on the host, (forked by the BOINC client). This python program, will then bridge between the hypervisor (via the VMController) and the BOINC client. This will read the job.xml file and do the needful... The wrapper (written by Jarno Rantala) requires, boinc python bindings written by David Wier. The Python Boinc bindings are available here: http://plato.tp.ph.ic.ac.uk/~djw03/boinc-python/ The original VMWrapper can be found here: https://svnweb.cern.ch/trac/vmwrapper/browser References ========== * Starting point, of everything: http://boinc.berkeley.edu/trac/wiki/ProjectMain * BOINC app hello world: http://www.spy-hill.net/~myers/help/boinc/hello.html * Building BOINC apps on Linux: http://www.pkids.net/help/boinc/boinc-on-linux.html * Unofficial BOINC development: http://www.boinc-wiki.info/BOINC_Development * Distributed Application With Boinc: http://w3.linux-magazine.com/issue/71/Distributed_Applications_With_BOINC.pdf * Creating Python Eggs: http://mrtopf.de/blog/python_zope/a-small-introduction-to-python-eggs/ * Python Egg formats: http://peak.telecommunity.com/DevCenter/EggFormats * Hello world python egg tutorial: http://mxm-mad-science.blogspot.com/2008/02/python-eggs-simple-introduction.html * A lot of stuff: http://www.swaroopch.com * Standalone BOINC debugging stuff: http://boinc.berkeley.edu/trac/wiki/AppDebug * BOINC Basic APIs (standalone etc.): http://boinc.berkeley.edu/trac/wiki/BasicApi * Some BOINC notes: http://www.boinc-wiki.info/Eric_Myers'_BOINC_Developers'_Notes * How to create a BOINC app: http://www.pkids.net/help/boinc/BOINC_Apps.html * Hello, BOINC world: http://www.pkids.net/help/boinc/hello.html * Create and configure a BOINC project: http://www.pkids.net/help/boinc/Create_Project.html * A wonderful introduciton to create Python Eggs: http://mrtopf.de/blog/python_zope/a-small-introduction-to-python-eggs/ * Writing a package in Python: http://www.packtpub.com/article/writing-a-package-in-python * Virtual Python Env: http://pypi.python.org/pypi/virtualenv * VirtualEnv: http://virtualenv.openplans.org/ * VirtualEnv wrapper: http://www.doughellmann.com/articles/pythonmagazine/completely-different/2008-05-virtualenvwrapper/index.html * iPython and VirtualEnv: http://www.doughellmann.com/articles/pythonmagazine/completely-different/2008-02-ipython-and-virtualenv/index.html BOINC credit stuff ================== * http://www.boinc-wiki.info/Trickle * http://www.boinc-wiki.info/BBC_Climate_Change_Experiment_FAQ Extra: ====== * USING VIRTUALENV: http://showmedo.com/videotutorials/video?name=2910000&fromSeriesID=291 * Zero Install(Cool Build n deploy tool): http://0install.net/ Python Tutorials and Stuff ========================== * http://www.python-visual-sandbox.de/ UNTRACKED: ========== http://svn.pythonmac.org/py2app/py2app/trunk/doc/index.html http://packages.python.org/distribute/setuptools.html http://peak.telecommunity.com/DevCenter/PythonEggs http://svn.pythonmac.org/py2app/py2app/trunk/doc/index.html