Infrastructure for implementing benchmarx: benchmarks for bidirectional transformation (bx) tools. Also contains a collection of example benchmarx and test runners for various and diverse bx tools.
We have a plug and play (via remote desktop) Share virtual machine available from: http://is.ieis.tue.nl/staff/pvgorp/share/?page=ConfigureNewSession&vdi=Ubuntu12LTS_BenchmarX.vdi
- Clone this repo:
git clone https://github.com/eMoflon/benchmarx.git benchmarx - Download the latest version of the Eclipse Modeling Tools for your platform. Currently tested for: http://www.eclipse.org/downloads/packages/eclipse-modeling-tools/neon3
- Start Eclipse in a workspace of your choice and import the Eclipse project BenchmarxFamiliesToPersons (
benchmarx/examples/familiestopersons/BenchmarxFamiliesToPersons) from the working tree of the benchmarx git repository you just cloned. - Choose the tools you want to execute by appropriately manipulating
/BenchmarxFamiliesToPersons/src/org/benchmarx/examples/familiestopersons/testsuite/FamiliesToPersonsTestCase.java/tools(). - (Optional) If you want to setup the bx tool BiGUL, then you'll have to work through
/BenchmarxFamiliesToPersons/src/org/benchmarx/examples/familiestopersons/implementations/bigul/README-SETUPto do this. - Choose the project BenchmarxFamiliesToPersons and select "Run As/JUnit Test" to execute the benchmarx "Families to Persons" for all tools chosen in Step 5.
- You can compare your results with
/BenchmarxFamiliesToPersons/results/TestResults.xlsx.
To add an implementation of an existing benchmarx example, e.g., FamiliesToPersons with your favourite bx tool, do the following (taking FamiliesToPersons as an example, substitute as appropriate for another example):
- Add a folder for your tool to
/examples/familiestopersons/BenchmarxFamiliesToPersons/src/org/benchmarx/examples/familiestopersons/implementations/<your_bx_tool>. - Add a Java test runner that is responsible for invoking and interfacing with your bx tool. If your bx tool is EMF-based, take a look at
emoflon/EMoflonFamiliesToPersons.javaandmedini/MediniQVTFamiliesToPersons.java. If your bx tool does not use EMF, then take a look atbigul/BiGULFamiliesToPersons.java. - If your implementation needs extra artefacts (such as Eclipse projects, etc) then add these to
/examples/familiestopersons/implementationsArtefacts. - Extend
/examples/familiestopersons/BenchmarxFamiliesToPersons/src/org/benchmarx/examples/familiestopersons/testsuite/FamiliesToPersonsTestCase.javato include your new test runner. - Feel free to add new JUnit tests that demonstrate the strengths of your tool (and reveal weaknesses of others). Please try to classify your new tests using the existing structure in the benchmarx.
- If your bx tool requires additional setup steps in addition to compiling a standard Eclipse project, add a README-SETUP file to this effect in your bx tool's implementation folder (cf.
bigul/README-SETUP). - Please remember to create and send us a pull request from your fork so we can update the benchmarx!
- Copy the existing structure for
/examples/familiestopersons. This consists of a foldermetamodelsfor the source and target metamodels in an EMF representation, a folderBenchmarxFamiliesToPersonsfor the actual testsuite, and a folderimplementationArtefactsfor any further implementation artefacts. - The folder
BenchmarxFamiliesToPersonsis an Eclipse project that can be run as a JUnit test suite and contains all test cases as JUnit tests, all resources (input and expected models), any implementations of the benchmarx example, and auxiliary classes for both metamodels used to simplify the comparison of output and expected models (or model representations). - Please remember to create and send us a pull request from your fork so we can update the benchmarx!