We use Giraph to model large-scale neural network on the Hadoop cluster deployed at the University of St. Thomas by Department of Graduate Programs in Software.
Note: The code has been fully tested on CDH4.4, but it is not working on CDH5 yet.
- Maven 3
- CDH4
- Giraph 1.0.0
- Java 1.6
- Python 2.7 or higher
- R
python xml_templates_parser.py
A directory called 'xml_input4Hadoop' will be generated under the project home.
Modify upload.sh first with appropriate user name and Hadoop cluster name.
mvn install
This command will compile the project and upload necessary files to Hadoop cluster in the directory giraph under $HOME
ssh into your Hadoop cluster:
cd giraph
chmod +x *.sh
./input.sh
The neural network will be generated in neuron_input on HDFS.
Delete _logs files generated at the previous step, otherwise Giraph will take it as input and an error would occur.
hadoop fs -rm -r neuron_input/_logs
Run Giraph job
./run.sh
- post-analysis/hive/create_table.hql: creating hive table from Giraph's output
- post-analysis/hive/firing_pattern.hql: extracting fired neurons
- post-analysis/hive/firing_rates.hql: extracting firing rates
- post-analysis/plot/graphs.R: plotting firing patterns
- post-analysis/plot/channel_plot.R: plotting firing patterns for multiple channels