BayAreaMetro/travel-model-two

Hard-coded output matrix type and incomplete MatrixType definition

i-am-sijia opened this issue · 1 comments

i-am-sijia commented

Describe the bug

Failed to write out trip table matrices from CTRAMP, using the latest mtctm2.jar and common-base.jar.

08-Jun-2021 10:55:16:440, ERROR, InvocationTargetException exception making RMI method call: //172.22.78.49:1191/com.pb.mtctm2.abm.ctramp.MatrixDataServer.writeMatrixFile().
java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at gnu.cajo.invoke.Remote.invoke(Unknown Source)
	at gnu.cajo.invoke.Remote.invoke(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
	at sun.rmi.transport.Transport$1.run(Unknown Source)
	at sun.rmi.transport.Transport$1.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$240(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$1/1876363557.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(Unknown Source)
	at sun.rmi.transport.StreamRemoteCall.executeCall(Unknown Source)
	at sun.rmi.server.UnicastRef.invoke(Unknown Source)
	at gnu.cajo.invoke.Remote_Stub.invoke(Unknown Source)
	at gnu.cajo.invoke.Remote.invoke(Unknown Source)
	at com.pb.mtctm2.abm.ctramp.UtilRmi.method(UtilRmi.java:123)
	at com.pb.mtctm2.abm.ctramp.MatrixDataServerRmi.writeMatrixFile(MatrixDataServerRmi.java:41)
	at com.pb.mtctm2.abm.application.MTCTM2TripTables.writeMatricesToFile(MTCTM2TripTables.java:546)
	at com.pb.mtctm2.abm.application.MTCTM2TripTables.writeTrips(MTCTM2TripTables.java:503)
	at com.pb.mtctm2.abm.application.MTCTM2TripTables.createTripTables(MTCTM2TripTables.java:282)
	at com.pb.mtctm2.abm.application.MTCTM2TripTables.main(MTCTM2TripTables.java:717)
Caused by: java.lang.RuntimeException: Could not determine type of matrix file, D:\MTC\travel-model-two\ctramp_output\auto_EA_SOV_GP_EA.mat
	at com.pb.common.matrix.MatrixReader.determineMatrixType(MatrixReader.java:193)
	at com.pb.common.matrix.MatrixWriter.createWriter(MatrixWriter.java:115)
	at com.pb.mtctm2.abm.ctramp.MatrixDataServer.writeMatrixFile(MatrixDataServer.java:113)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at gnu.cajo.invoke.Remote.invoke(Unknown Source)
	at gnu.cajo.invoke.Remote.invoke(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
	at sun.rmi.transport.Transport$1.run(Unknown Source)
	at sun.rmi.transport.Transport$1.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$240(Unknown Source)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$1/1876363557.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

To Reproduce

Steps to reproduce the behavior:
Run CTRAMP/runtime/runMTCTM2ABM.cmd

rem ### create demand matrices in Cube matrix format - must restart mtx manager before running?
java -Xmx200g -cp "%CLASSPATH%" -Dproject.folder=%PROJECT_DIRECTORY% com.pb.mtctm2.abm.application.MTCTM2TripTables mtctm2 -iteration %iteration% -sampleRate %sampleRate%

Triggering line of code

  1. output matrix file extension is hard-coded as "mat". Matrix writer uses filename extension to determine matrix type:

travel-model-two/core/src/java/com/pb/mtctm2/abm/application/MTCTM2TripTables.java

Line 70 in 2d650b5

private String matrixFileExtension = "mat";

travel-model-two/core/src/java/com/pb/mtctm2/abm/application/MTCTM2TripTables.java

Lines 534 to 549 in 2d650b5

public void writeMatricesToFile(String fileName, Matrix[] m){
//todo - currently write one matrix to each file since there is a problem
//calling RemoteMatrixDataServer.writeMatrices from RMI if the matrix array has more than 1 matrix
for (int i=0; i<m.length; i++) {
String matFileName = fileName + "_" + m[i].getName() + "." + matrixFileExtension;
Matrix[] temp = new Matrix[1];
temp[0] = m[i];
logger.info( m[i].getName() + " has " + m[i].getRowCount() + " rows, " + m[i].getColumnCount() + " cols, and a total of " + m[i].getSum() );
logger.info("Writing " + matFileName);
ms.writeMatrixFile(matFileName, temp);
}
}

  1. ".mat" is not defined in the common-base DetermineMatrixType method

travel-model-two/core/cmf/common-base/src/java/com/pb/common/matrix/MatrixReader.java

Lines 153 to 197 in 2d650b5

public static MatrixType determineMatrixType(File file) {
MatrixType type;
String fileName = file.getName();
if (fileName.indexOf(".bin") > 0 || fileName.indexOf(".BIN") > 0 ||
fileName.indexOf(".binary") > 0 || fileName.indexOf(".BINARY") > 0)
{
type = MatrixType.BINARY;
}
else if (fileName.indexOf(".zip") > 0 || fileName.indexOf(".ZIP") > 0
|| fileName.indexOf(".zmx") > 0
|| fileName.indexOf(".compressed") > 0
|| fileName.indexOf(".COMPRESSED") > 0) {
type = MatrixType.ZIP;
}
else if (fileName.indexOf(".csv") > 0 || fileName.indexOf(".CSV") > 0) {
type = MatrixType.CSV;
}
else if (fileName.indexOf(".emme2") > 0 || fileName.indexOf(".EMME2") > 0 ||
fileName.indexOf(".e2ban") > 0 || fileName.indexOf(".E2BAN") > 0 ||
fileName.indexOf(".em2") > 0 || fileName.indexOf(".EM2") > 0 ||
fileName.indexOf("emme2ban") >= 0 || fileName.indexOf("EMME2BAN") >= 0 ||
fileName.equalsIgnoreCase("emme2ban") || fileName.equalsIgnoreCase("emmebank"))
{
type = MatrixType.EMME2;
}
else if (fileName.indexOf(".tpp") > 0 || fileName.indexOf(".TPP") > 0 ||
fileName.indexOf(".tpplus") > 0 || fileName.indexOf(".TPPLUS") > 0)
{
type = MatrixType.TPPLUS;
}
else if (fileName.indexOf(".mtx") > 0 || fileName.indexOf(".MTX") > 0 ||
fileName.indexOf(".transcad") > 0 || fileName.indexOf(".TRANSCAD") > 0)
{
type = MatrixType.TRANSCAD;
}
else if (fileName.indexOf(".omx") > 0 || fileName.indexOf(".OMX") > 0) {
type = MatrixType.OMX;
}
else {
throw new RuntimeException("Could not determine type of matrix file, " + file.getAbsolutePath());
}
return type;
}

Thoughts on resolution

  1. Remove the hard-coded matrix file extension, replace it with reading from properties file
  2. Add if condition for ".mat" in DetermineMatrixType()

Environment

windows 10
java 1.8.0_31

fyi @jfdman @lmz @DavidOry

i-am-sijia commented

I see in transit-ccr branch, the hard-coded matrix file extension is changed to omx, instead of mat. This is a quick solution to get the omx working, but in the long run, maybe it's better to define the matrix type in properties file.

travel-model-two/core/src/java/com/pb/mtctm2/abm/application/MTCTM2TripTables.java

Line 70 in 8f582db

private String matrixFileExtension = "omx";