Benchmark Study Runs

Question

Benchmark Study Runs

msdogan opened this issue 8 years ago · 12 comments

@jdherman we would like to perform multiple (so many) runs to compare solver and total run times for each solver we use. If we create 5 different data sets (different # of decision variables) for each solver (4 total) and 10 run for each to create error bars (originally 30), it makes 200 runs. Only changing thing in command in batch file would be --solver and data.dat paths.

pyomo solve --solver=cbc --solver-suffix=dual --solver-options="threads=32" pyvin.py $DIR/data.dat --json --report-timing --stream-solver

Is there a way to run job script recursively? Maybe a for loop in batch file :)

Answer 1 · 2016-10-12T22:00:55.000Z

Sure, 200 runs, why not :)

The question is whether to do this all in one job, or multiple jobs.

To do it all in one job, it would look something like this:

solvers="glpk cbc cplex gurobi"
paths="a b c d e"
trials=$(seq 1 10)

for solver in $solvers
do
  for path in $paths
  do
    for trial in $trials
    do

      pyomo solve --solver=$solver pyvin.py $path/data.dat --report-timing > $path/output_trial_$trial.txt

    done
  done
done

(The > will redirect the command-line output into a file, and we want a different file for each value of $trial).

Of course, this will take forever to run. So if we want to divide up some of this into separate jobs, we can have another script that submits multiple versions of job-slurm.sh. I have some old examples here: https://github.com/jdherman/hpc-submission-scripts/blob/master/example-slurm-script.sh

The "outer loop" script could look like this:

#!/bin/bash

# Example submission for SLURM system
# Loop through multiple job submissions
# Pass environment variables to job script
NUMBERS=$(seq 1 4)

for NUM in ${NUMBERS}
do
    sbatch -J JOB_NAME_${NUM} --export=VARIABLE=${NUM} job-slurm.sh
done

So this is NOT a job script -- it submits multiple job scripts in a loop. The important thing is that a variable called $VARIABLE will be passed into job-slurm.sh, where it can be used. So that variable might be the solver name, for example. Then if you run ./myfile.sh on the command line, it will loop and submit a bunch of jobs.

Option (1) is definitely easier to understand, but I think it will be too slow. Maybe experiment with all 1-year examples first, and build up to the longer ones?

I can help with this -- the syntax is weird.

Answer 2 · 2016-10-12T22:46:34.000Z

Yes, I agree. Let's start with the first method for 1-year data only. This will reduce one for loop. I will send you the job-slurm.sh before submitting it.

Answer 3 · 2016-10-28T00:21:28.000Z

@jdherman I created five datasets: 1-year, 5-year, 20-year, 40-year and 82-year. I have decided to send multiple jobs rather than one job because we can use .out to look at results. results.yaml takes so much space so I will delete it in each job and postprocess results in .out. I think it has enough for us; # of decision variables, runtime, solver time.

Each time, I will change data path and solver name in job-slurm.sh and submit trial.sh, which will send job-slurm.sh ten times. I just need to do that 20 times. Not that bad.

So, do you think below job files work?

trial.sh

#!/bin/bash

# Example submission for SLURM system
# Loop through multiple job submissions
# Pass environment variables to job script
NUMBERS=$(seq 1 10)

for NUM in ${NUMBERS}
do
    sbatch -J JOB_NAME_${NUM} --export=VARIABLE=${NUM} job-slurm.sh
done

job-slurm.sh

#!/bin/bash
#SBATCH -D /home/msdogan/pyvin
#SBATCH -o /home/msdogan/pyvin/benchmark/job.%j.%N.out
#SBATCH -e /home/msdogan/pyvin/benchmark/job.%j.%N.err
#SBATCH -n 32            # Total number of processors to request (32 cores per node)
#SBATCH -p med           # Queue name hi/med/lo
#SBATCH -t 200:00:00        # Run time (hh:mm:ss) - 24 hours
#SBATCH --mail-user=msdogan@ucdavis.edu              # address for email notification
#SBATCH --mail-type=ALL                  # email at Begin and End of job

# you can define your own variables. Access them later with dollar signs ($DIR, etc.)
DIR=1-year
GDIR=/group/hermangrp

# IMPORTANT: Python3/Pyomo/CBC solver are all installed in group directory. Add it to the path.
export PATH=$GDIR/miniconda3/bin:$GDIR/cbc/bin:$PATH

# Export CPLEX and GUROBI solvers
export CPLEX_HOME=$GDIR/cplex1263/cplex
export GUROBI_HOME=$GDIR/gurobi652/linux64
export PATH=$PATH:$GUROBI_HOME/bin:$CPLEX_HOME/bin/x86-64_linux:$GDIR/cbc/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$GUROBI_HOME/lib:$CPLEX_HOME/lib/x86-64_linux/static_pic
export GRB_LICENSE_FILE=$GDIR/gurobi652/gurobi.lic

# if you want GLPK (serial)
# pyomo solve --solver=glpk --solver-suffix=dual /home/msdogan/pyvin/pyvin.py $DIR/data.dat --report-timing --stream-solver

# if you want CBC (parallel, can only use up to 32 cores)
# if you want to use CPLEX or GUROBI solvers, change "--solver=cbc" to "--solver=cplex" or "--solver=gurobi"
pyomo solve --solver=cplex --solver-suffix=dual --solver-options="threads=32" /home/msdogan/pyvin/pyvin.py $DIR/data.dat --report-timing --stream-solver

# the --stream-solver flag gives verbose output if you want to see what's going on. If you don't care about this, remove it.

# At this point the results.json file should exist. Now we can postprocess. (This does not run in parallel; you could do it separately after the job finishes)

# mv results.json /home/msdogan/pyvin/glpk && cd $DIR

rm results.yaml

Answer 4 · 2016-10-28T17:13:01.000Z

Sure, this is great. A couple of thoughts.

Should we wait until we can run the non-debug version for the runtime benchmarking? Maybe we should focus on that first, because it would make more sense to report runtimes for the real-size network (not 2x the size including debug links). We could use the debug-flow-finder script on the 82-year results to see where some problems might be.
The script job-slurm.sh does not use $VARIABLE that is passed to it. Which means that all of the jobs (in this case, 10 of them) will be the same. You want to set it up so that the variable passed in somehow changes which job is being run (either the solver and/or the number of years). See example at the bottom.
I like the idea of only saving the *.out files to look at runtimes. But at some point we will want to compare the actual results (time series of flows, etc.) from the different solvers, at least for the full 82-year version. These can be separate runs later, though.

I haven't tested this but something like this should work (call it submit-all-jobs.sh):

#!/bin/bash

# Example submission for SLURM system
# Loop through multiple job submissions
# Pass environment variables to job script
SOLVERS=( "cplex" "gurobi" "cbc" "glpk" )
DIRS=( "1-year" "5-year" "20-year" "40-year" "82-year" )

for SOLVER in ${SOLVERS[@]}
do
  for DIR in ${DIRS[@]}
  do
    sbatch -J benchmark_${SOLVER}_${DIR} --export=SOLVER,DIR job-slurm.sh
  done
done

Then inside job-slurm.sh, you can use $DIR and $SOLVER without needing to define them each time. Does that make sense? Again I haven't tested this so there may be some bugs :)

Answer 5 · 2016-10-28T17:49:09.000Z

okay sounds good. I will wait until we fix debug flows.

I was thinking to change solvers and paths manually but your way of doing is much smarter :) We want to make 10 run for each right. If we sent the same job 10 times, wouldn't it work?

Okay we can store the result files. I was concerned with the file sizes. But, if we are to compare results, then yes, saving those is a good idea.

Answer 6 · 2016-10-28T17:52:55.000Z

But we send the same job 10 times, how will it know what variables to change? :) (unless you change them manually, right).

About the file storage, one idea might be to run postprocess.py and then delete the big JSON file, only keep the CSV results after that. They should be much smaller. Don't worry about this right now, though.

Answer 7 · 2017-04-27T20:10:47.000Z

@jdherman I have created benchmark run script and sent the job. It is one job file rather than multiple jobs. 1-year run (total 40 = 4 solvers X 10 trials) has just finished in 25 minutes, and now runnig 5-year dataset. I have created runs for 1-year, 5-year, 10-year, 40-year and 82-year. So, 5 datasets that makes 200 runs in total. I put a time limit of 400 hours but I think we are going to hit that especially with glpk solver. Anyway, if we hit the time limit bound, maybe I can create a separate job file only for glpk. I am not keeping results.json. I am only keeping track of time and objective function.

Here what the job file looks like:

#!/bin/bash
#SBATCH -D /home/msdogan/pyvin
#SBATCH -o /home/msdogan/pyvin/job.%j.%N.out
#SBATCH -e /home/msdogan/pyvin/job.%j.%N.err
#SBATCH -n 32            # Total number of processors to request (32 cores per node)
#SBATCH -p med           # Queue name hi/med/lo
#SBATCH -t 400:00:00        # Run time (hh:mm:ss) - 24 hours
#SBATCH --mail-user=msdogan@ucdavis.edu              # address for email notification
#SBATCH --mail-type=ALL                  # email at Begin and End of job

# you can define your own variables. Access them later with dollar signs ($DIR, etc.)
paths="benchmark/1-year benchmark/5-year benchmark/10-year benchmark/40-year benchmark/82-year"
# paths=benchmark/1-year
GDIR=/group/hermangrp
# solvers
solvers="cplex gurobi cbc glpk"
# number of trials
trials=$(seq 1 10)

# IMPORTANT: Python3/Pyomo/CBC solver are all installed in group directory. Add it to the path.
export PATH=$GDIR/miniconda3/bin:$GDIR/cbc/bin:$PATH

# Export CPLEX and GUROBI solvers
export CPLEX_HOME=$GDIR/cplex1263/cplex
export GUROBI_HOME=$GDIR/gurobi652/linux64
export PATH=$PATH:$GUROBI_HOME/bin:$CPLEX_HOME/bin/x86-64_linux:$GDIR/cbc/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$GUROBI_HOME/lib:$CPLEX_HOME/lib/x86-64_linux/static_pic
export GRB_LICENSE_FILE=$GDIR/gurobi652/gurobi.lic

# if you want GLPK (serial)
# pyomo solve --solver=glpk --solver-suffix=dual pyvin.py $DIR/data.dat --json --report-timing --stream-solver

# if you want CBC (parallel, can only use up to 32 cores)
# if you want to use CPLEX or GUROBI solvers, change "--solver=cbc" to "--solver=cplex" or "--solver=gurobi"
# if you want to define number of cores use "--solver-options="threads=32""
for path in $paths
do
	for solver in $solvers
	do
		for trial in $trials
		do
			pyomo solve --solver=$solver --solver-suffix=dual pyvin.py $path/data.dat --json --report-timing > $path/$solver/output_trial_$trial.txt --stream-solver
		done
	done
done
# the --stream-solver flag gives verbose output if you want to see what's going on. If you don't care about this, remove it.

# At this point the results.json file should exist. Now we can postprocess. (This does not run in parallel; you could do it separately after the job finishes)

# mv results.json $DIR && cd $DIR

Answer 8 · 2017-04-28T16:35:53.000Z

@msdogan awesome!! Couple of thoughts:

You might have to make sure all of the $path/$solver folders exist before it tries to save those files. You could mkdir $path/$solver inside the loop before running pyomo solve
Does results.json just get overwritten every time a new solver runs? If so that's fine, I agree we don't care about it. But if it's saving copies that are ~1 GB each you might just want to rm results.json after every call to pyomo solve.

Let's see what these things can do!

Answer 9 · 2017-05-01T16:40:08.000Z

@jdherman benchmark runs are in debug mode because any time period less than 82-year results in infeasibility without debug mode. Would that be a problem for benchmark runs? It will take a bit longer to run those because there is more decision variables.

Status update:

1, 5, 10-year runs are done for all solvers
40-year run is done for cplex, gurobi and cbc. Glpk is still running
82-year, only cplex finished 8 runs out of 10.

Answer 10 · 2017-05-01T18:42:18.000Z

It's fine to run them in debug mode, as long as we report the true number of decision variables, including debug links. For the benchmark timings we're less interested in the actual flows in the system anyway.

Sounds good! Run them for as long as you need to run them. Let me know if I can help write a scraper for the results files.

Thanks again for running the shortcourse on Friday, it gave me some ideas for how we could revise this code in the future ... but not right now, it's working fine :)

Answer 11 · 2017-05-10T23:16:16.000Z

@jdherman I am excited to share benchmark results. Based on results, new champion is gurobi.
I created a new branch for benchmark runs but if you think that it should be in master branch, I can create a pull request. I plotted two results: solver reported time (or solver runtime), and pyomo finished (total PyVIN runtime). All results are coming from text files and benchmark.py retrieves them, summarizes outputs (https://github.com/msdogan/pyvin/blob/benchmark/benchmark/output.csv) and creates plots. I also created some statistics, such as mean, median and standard deviation because there is some outliars.

Following figure shows solver reported times for each run size (# of decision variables:
solver reported time

stats:
solver reported time

Total model runtime depending on number of decision variables with trendlines and equations:
total model runtime

stats:
total runtime

Answer 12 · 2017-05-12T17:34:37.000Z

Hi Mustafa, this is great to have multiple trials. I guess our CPLEX result from last time was just a lucky trial. So we can trust this result more.

I agree we can keep a separate branch for these results. Let's focus on the solver time only (not the total time), because the output file writing operations could be affected by other things happening on the filesystem at the same time.

@mfefer want to use these updated solver results for your EWRI poster? :)