Add Windows (MSVC) support

Question

Add Windows (MSVC) support

bchretien opened this issue 10 years ago · 30 comments

bchretien commented 10 years ago

Some people are interested in using RobOptim on Windows, so this is worth investigating.

@kayusawa could you make a summary of your tests regarding the compilation of RobOptim on Windows? Thanks! :-)

Answer 1 · 2014-10-08T07:46:25.000Z

👍
Some details on the installation of the dependencies would also be appreciated.

Answer 2 · 2014-10-08T09:56:40.000Z

Indeed, we could then make a detailed installation process on the website.

Answer 3 · 2014-10-08T10:35:35.000Z

Dear Dr.Crétien

Thank you for the e-mail.
I could compile RobOpitim on Windows by using VisualStudio 2012 and Intel
Compiler few month ago.
For the compilation, I prepared several windows tools and libraries as
follows:

CMake (2.8.12)
pkg_config
doxgen
eigen (3.2.2)
boost (1.55)
log4cxx (0.10.0)
libtool (1.5.26)

Among them, I compiled boost, log4cxx, and libtool myself.
I'm sorry that I don't really remember the details, but a few minor
modifications are required for the compilation of log4cxx and libtool.

After preparing the libraries, the file encoding of RobOptim codes has to
be changed from "utf8-without BOM" to "utf8-with BOM".
(It enables MSVC/Intel to read utf8 characters.)

If Intel Compiler is available, the project files generated by CMake will
work properly without modification.
However, if you plan to use VisualStudio, there may be a few conflicts
about MSVC definitions.
For example, "boost::mpl::size_t" and "unsigned int size_t" will be
ambiguous when calling Eigen sparse matrix.
Actually there are few errors in my case, so I modified Eigen codes
directly.

If you have any question, please contact me.

Best regards,
Ko Ayusawa

Ko AYUSAWA, researcher
CNRS-AIST JRL(Joint Robotics Laboratory), UMI3218/CRT,
Intelligent Systems Research Institute,
National Institute of Advanced Industrial Science and Technology(AIST)
Tsukuba Central 2, 1-1-1 Umezono, Tsukuba, Ibaraki 305-8568 JAPAN
TEL: +81-29-861-3576 FAX: +81-29-862-6507

e-mail: k.ayusawa@aist.go.jp

2014-10-07 20:20 GMT+09:00 Benjamin Chrétien notifications@github.com:

Some people are interested in using RobOptim on Windows, so this is worth
investigating.

@kayusawa https://github.com/kayusawa could you make a summary of your
tests regarding the compilation of RobOptim on Windows? Thanks! :-)

—
Reply to this email directly or view it on GitHub
#76.

Answer 4 · 2014-10-08T10:43:27.000Z

Dear Dr. Chrétien

I'm so sorry that I misspelled your name.

If you want to need the pre-compiled binaries or modified codes of the
third-party libraries, I can prepare and send you.

Best regards,

Ko AYUSAWA, researcher
CNRS-AIST JRL(Joint Robotics Laboratory), UMI3218/CRT,
Intelligent Systems Research Institute,
National Institute of Advanced Industrial Science and Technology(AIST)
Tsukuba Central 2, 1-1-1 Umezono, Tsukuba, Ibaraki 305-8568 JAPAN
TEL: +81-29-861-3576 FAX: +81-29-862-6507

e-mail: k.ayusawa@aist.go.jp

2014-10-08 19:35 GMT+09:00 Ko AYUSAWA k.ayusawa@aist.go.jp:

Dear Dr.Crétien

Thank you for the e-mail.
I could compile RobOpitim on Windows by using VisualStudio 2012 and Intel
Compiler few month ago.
For the compilation, I prepared several windows tools and libraries as
follows:

CMake (2.8.12)

pkg_config

doxgen

eigen (3.2.2)

boost (1.55)

log4cxx (0.10.0)

libtool (1.5.26)

Among them, I compiled boost, log4cxx, and libtool myself.
I'm sorry that I don't really remember the details, but a few minor
modifications are required for the compilation of log4cxx and libtool.

After preparing the libraries, the file encoding of RobOptim codes has to
be changed from "utf8-without BOM" to "utf8-with BOM".
(It enables MSVC/Intel to read utf8 characters.)

If Intel Compiler is available, the project files generated by CMake will
work properly without modification.
However, if you plan to use VisualStudio, there may be a few conflicts
about MSVC definitions.
For example, "boost::mpl::size_t" and "unsigned int size_t" will be
ambiguous when calling Eigen sparse matrix.
Actually there are few errors in my case, so I modified Eigen codes
directly.

If you have any question, please contact me.

Best regards,
Ko Ayusawa

Ko AYUSAWA, researcher
CNRS-AIST JRL(Joint Robotics Laboratory), UMI3218/CRT,
Intelligent Systems Research Institute,
National Institute of Advanced Industrial Science and Technology(AIST)
Tsukuba Central 2, 1-1-1 Umezono, Tsukuba, Ibaraki 305-8568 JAPAN
TEL: +81-29-861-3576 FAX: +81-29-862-6507

e-mail: k.ayusawa@aist.go.jp

2014-10-07 20:20 GMT+09:00 Benjamin Chrétien notifications@github.com:

Some people are interested in using RobOptim on Windows, so this is worth
investigating.

@kayusawa https://github.com/kayusawa could you make a summary of your
tests regarding the compilation of RobOptim on Windows? Thanks! :-)

—
Reply to this email directly or view it on GitHub
#76.

Answer 5 · 2014-10-08T11:48:51.000Z

@kayusawa Hi, I would be interested in testing roboptim on windows. Could
you please send me (or put on a server) those pre-compiled libraries?

Thanks.
Le 8 oct. 2014 12:43, "kayusawa" notifications@github.com a écrit :

Dear Dr. Chrétien

I'm so sorry that I misspelled your name.

If you want to need the pre-compiled binaries or modified codes of the
third-party libraries, I can prepare and send you.

Best regards,

Ko AYUSAWA, researcher
CNRS-AIST JRL(Joint Robotics Laboratory), UMI3218/CRT,
Intelligent Systems Research Institute,
National Institute of Advanced Industrial Science and Technology(AIST)
Tsukuba Central 2, 1-1-1 Umezono, Tsukuba, Ibaraki 305-8568 JAPAN
TEL: +81-29-861-3576 FAX: +81-29-862-6507

e-mail: k.ayusawa@aist.go.jp

2014-10-08 19:35 GMT+09:00 Ko AYUSAWA k.ayusawa@aist.go.jp:

Dear Dr.Crétien

Thank you for the e-mail.
I could compile RobOpitim on Windows by using VisualStudio 2012 and
Intel
Compiler few month ago.
For the compilation, I prepared several windows tools and libraries as
follows:

CMake (2.8.12)

pkg_config

doxgen

eigen (3.2.2)

boost (1.55)

log4cxx (0.10.0)

libtool (1.5.26)

Among them, I compiled boost, log4cxx, and libtool myself.
I'm sorry that I don't really remember the details, but a few minor
modifications are required for the compilation of log4cxx and libtool.

After preparing the libraries, the file encoding of RobOptim codes has
to
be changed from "utf8-without BOM" to "utf8-with BOM".
(It enables MSVC/Intel to read utf8 characters.)

If Intel Compiler is available, the project files generated by CMake
will
work properly without modification.
However, if you plan to use VisualStudio, there may be a few conflicts
about MSVC definitions.
For example, "boost::mpl::size_t" and "unsigned int size_t" will be
ambiguous when calling Eigen sparse matrix.
Actually there are few errors in my case, so I modified Eigen codes
directly.

If you have any question, please contact me.

Best regards,
Ko Ayusawa

Ko AYUSAWA, researcher
CNRS-AIST JRL(Joint Robotics Laboratory), UMI3218/CRT,
Intelligent Systems Research Institute,
National Institute of Advanced Industrial Science and Technology(AIST)
Tsukuba Central 2, 1-1-1 Umezono, Tsukuba, Ibaraki 305-8568 JAPAN
TEL: +81-29-861-3576 FAX: +81-29-862-6507

e-mail: k.ayusawa@aist.go.jp

2014-10-07 20:20 GMT+09:00 Benjamin Chrétien notifications@github.com:

Some people are interested in using RobOptim on Windows, so this is
worth
investigating.

@kayusawa https://github.com/kayusawa could you make a summary of
your
tests regarding the compilation of RobOptim on Windows? Thanks! :-)

—
Reply to this email directly or view it on GitHub
#76.

—
Reply to this email directly or view it on GitHub
#76 (comment)
.

Answer 6 · 2014-10-08T12:40:34.000Z

Dear Ayusawa-san,

Thanks for the quick answer. Don't worry about my name, I know it's a difficult one :-)

For UTF-8 with BOM, this may be something that can be solved by CMake when generating the MSVC project files. If the compiler is the issue, a fix similar to this one may be what we need, especially if that means we do not need to add a BOM to all the files that contain UTF-8 characters.

For pre-compiled libraries, these could be released for our next major release. For the dependencies, I guess we could provide them (log4cxx/libtool), especially if some patching is involved.

For the Eigen error, is it something that happened in RobOptim or Eigen itself?

Answer 7 · 2014-10-08T14:13:19.000Z

Dear Chrétien-san, Keith-san

For UTF-8 with BOM, this may be something that can be solved by CMake when

generating the MSVC project files. If the compiler is the issue, a fix similar
to this one
https://github.com/chewing/libchewing/blob/master/CMakeLists.txt#L42 may
be what we need, especially if that means we do not need to add a BOM to
all the files that contain UTF-8 characters.

Thank you for the advice.
I agree that we should solve it by the configuration of CMake.

For pre-compiled libraries, these could be released for our next major

release. For the dependencies, I guess we could provide them (log4cxx/
libtool), especially if some patching is involved.

I attached the pre-compiled log4cxx and libtool (both x86 and x64).
Please rename ".lib.dat" to ".lib" inside "install.zip"
(This is because the security problem of AIST mailing service.)
I'm sorry that the path setting of the package config file of log4cxx has
to be modified manually.

I also attached the modified code of them.
Almost all the modification of log4cxx is just the definition of namespace.
The original windows libtool does not call loadlibrary properly on VC2012,
so I tried to modify and create the windows solution file.
I also attach the simple memo of those installation.

For the Eigen error, is it something that happened in RobOptim or Eigen

itself?

I'm sorry that I almost forgot about where the problem occured.
But I guess "boost/mpl/size_t" and "unsigned int size_t" were ambiguous
when creating Eigen matrix array in Function of RobOptim .
When I used Intel Compiler, there are no such kind of problems.
I guess there is a way to solve it by using MSVC macro to control them.
But at that time, I had to compile RobOptim immediately, so I modified
"size_t" to "::size_t" in several places in Eigen codes and planed to
revert later.

If there are problems, please don't hesitate to contact me.

Best regards,
Ko ayusawa

Ko AYUSAWA, researcher
CNRS-AIST JRL(Joint Robotics Laboratory), UMI3218/CRT,
Intelligent Systems Research Institute,
National Institute of Advanced Industrial Science and Technology(AIST)
Tsukuba Central 2, 1-1-1 Umezono, Tsukuba, Ibaraki 305-8568 JAPAN
TEL: +81-29-861-3576 FAX: +81-29-862-6507

e-mail: k.ayusawa@aist.go.jp

2014-10-08 21:40 GMT+09:00 Benjamin Chrétien notifications@github.com:

Dear Ayusawa-san,

Thanks for the quick answer. Don't worry about my name, I know it's a
difficult one :-)

For UTF-8 with BOM, this may be something that can be solved by CMake when
generating the MSVC project files. If the compiler is the issue, a fix similar
to this one
https://github.com/chewing/libchewing/blob/master/CMakeLists.txt#L42
may be what we need, especially if that means we do not need to add a BOM
to all the files that contain UTF-8 characters.

For pre-compiled libraries, these could be released for our next major
release. For the dependencies, I guess we could provide them (log4cxx/
libtool), especially if some patching is involved.

For the Eigen error, is it something that happened in RobOptim or Eigen
itself?

—
Reply to this email directly or view it on GitHub
#76 (comment)
.

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

open VisualStudio comand prompt (x86)
change directory to boost top directory
type as follows:
$ bootstrap
$ b2 --build-dir=_build\x86 --stagedir=stage\x86 address-model=32 link=static,shared runtime-link=shared -j 4
if you have intel compiler...
$ "c:\Program Files (x86)\Intel\Composer XE\bin\compilervars.bat" ia32 vs2012
$ b2 --toolset="intel" --build-dir=_build\x86 --stagedir=stage\x86 address-model=32 link=static,shared runtime-link=shared -j 4

1. open VisualStudio comand prompt (x64) 2. change directory to boost top directory 3. type as follows: $ bootstrap $ b2 --build-dir=_build\x64 --stagedir=stage\x64 address-model=64 link=static,shared runtime-link=shared -j 4 4. if you have intel compiler... $ "c:\Program Files (x86)\Intel\Composer XE\bin\compilervars.bat" intel64 vs2012 $ b2 --toolset="intel" --build-dir=_build\x64 --stagedir=stage\x64 address-model=64 link=static,shared runtime-link=shared -j 4

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
<log4cxx(MSVC2012)>

download as follows:
- apache-log4cxx-0.10.0.zip
- apr-1.5.1-win32-src.zip
- apr-iconv-1.2.1-win32-src-r2.zip
- apr-util-1.5.3-win32-src.zip
from
http://logging.apache.org/log4cxx/
http://ftp.yz.yamagata-u.ac.jp/pub/network/apache/apr/
extract them as follows:
log4cxx\appatch-log4cxx-0.10.0
log4cxx\apr
log4cxx\apr-util
log4cxx\apr-iconv
call as follows:
- configure.bat
- configure-aprutil.bat
open "log4cxx\apache-log4cxx-0.10.0\projects\log4cxx.dsw"
add "log4cxx\apr-iconv\apriconv.dsp" in the solution file.
replace files as follows:
- loggingevent.h
- loggingevent.cpp
- propertiespatternconverter.cpp
- xmllayout.cpp
build log4cxx
liblog4cxx.pc has to be modified manually.

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

1. open libltdl.sln 2. build libltdl

Answer 8 · 2015-05-21T14:24:30.000Z

Note regarding CI support for Windows: I recently noticed that some open-source projects now use AppVeyor, which could very roughly be seen as Travis for Windows.

Answer 9 · 2016-03-01T13:21:31.000Z

In the latest stable there seems alot of issues with a windows build, that aren't dependencies.

Answer 10 · 2016-03-01T13:23:19.000Z

Windows support is not available yet, although we are slowly getting there (cc @aescande).

Answer 11 · 2016-03-01T13:26:06.000Z

We have worked out quite few issues, but are still working on a lot of templating issues, e.g.:

40>C:\git\roboptim-core\include\roboptim/core/problem.hxx(118): error C2491: 'roboptim::Problem<T>::function' : definition of dllimport function not allowed

Will keep you informed. Is there an old binary of an older commit?

Answer 12 · 2016-03-01T13:39:04.000Z

Thanks for looking into this. Alas I don't think anyone shared any binary yet.

Answer 13 · 2016-03-01T13:46:09.000Z

Just putting my 2 cents here.

I have tried to compile roboptim-core on windows very recently (I must be 1-2 commits late wrt master). Here are the errors I found (as I remember them, so this is not as precise as I'd like it to be).

The error you're describing definition of dllimport function not allowed is due to the fact that some methods / classes are now templated (but weren't in the last version tested). The error can be suppressed by removing this macro for those classes.
Some errors will also pop up because of the macro ROBOPTIM_CORE_**_DEPRECATED, that must be placed *before the function declaration in windows.
Some of the unit tests will not compile in VS2008 because of template issues (I cannot tell you which ones right here).
The format of the output is not the same as in unix (e.g., infinity is not written inf, but #INF, and the E notation takes an extra 0: 1e03 versus 1e003), so some unit tests may fail.

Those are some leads.
I will push that on my fork (it is still a work in progress) tonight if you want to have a look.

Answer 14 · 2016-03-01T13:58:54.000Z

Awesome, this is exactly what we are seeing.
I think to solve the 1st issue the ROBOPTIM_DLLAPI has to moved to a pre-declaration of the class., as:

// function definition
void __declspec(dllimport) funcB() {}   // C2491

// function declaration
void __declspec(dllimport) funcB();   // OK

number 2:
We set

#define ROBOPTIM_CORE_**_DEPRECATED

Answer 15 · 2016-03-01T14:06:58.000Z

I don't think it is required to keep the ROBOPTIM_DLLAPI macro for the templated class (e.g. Problem) since the definition of the method is in the .hxx and those files are installed as well.
So there is no point AFAIK in exporting the symbols.

I compiled the unit tests as well as roboptim/roboptim-core-plugin-eigen without having any symbol issue.

Answer 16 · 2016-03-01T14:40:14.000Z

Indeed, I'll clean this up a bit. I recently added the possibility to precompile most of the templated classes, since they are now only templated on the matrix traits (dense/sparse). However this is disabled by default since explicit template instantiation is a C++11 feature. I guess the error will appear in some other classes.

Answer 17 · 2016-03-01T15:12:27.000Z

ROBOPTIM_CORE_DEPRECATED (which is equivalent to __declspec (deprecated) on Windows with MSVC) has to be at the beginning of function declarations/typedefs?

Answer 18 · 2016-03-01T15:13:27.000Z

Yes, otherwise some compilation issues may raise.

Answer 19 · 2016-03-01T15:18:11.000Z

Ok, we can easily fix that then. I was also considering making a PR to jrl-cmakemodules to have the deprecated macros support version numbers, e.g.:

#define ROBOPTIM_CORE_DEPRECATED(X) __attribute__ ((deprecated))
...
typedef Foo Bar ROBOPTIM_CORE_DEPRECATED(3.2);

They use this for Gazebo, that way you can see exactly when a given function was deprecated when you get the warning while compiling.

Answer 20 · 2016-03-01T15:20:35.000Z

And talking about Gazebo, they had the exact same problem:

#if defined(__GNUC__)
#define GAZEBO_DEPRECATED(version) __attribute__((deprecated))
#define GAZEBO_FORCEINLINE __attribute__((always_inline))
#elif defined(_WIN32)
// GAZEBO_DEPRECATED should be defined as something like
// __declspec(deprecated), but it needs to go *before* the function name,
// and we're putting GAZEBO_DEPRECATED *after* the function.
#define GAZEBO_DEPRECATED(version)
#define GAZEBO_FORCEINLINE __forceinline
#else
#define GAZEBO_DEPRECATED(version) ()
#define GAZEBO_FORCEINLINE
#endif

Answer 21 · 2016-03-01T16:14:36.000Z

Bear in mind, getting Gazebo on windows is still not an easy experience. But this looks good.
That commit (992e734) looks very similar to what we got.

Answer 22 · 2016-03-02T00:04:52.000Z

@phuicy I pushed on https://github.com/francois-keith/roboptim-core the current state of my windows version.
I tested it on MSVC 2012, with the following deps:

boost 1.55
libtools 1.5.26
eigen 3.2.0

@bchretien No Pull request for now, mainly because of the test issue and the fact that the result file is different between the Win32 and UNIX systems.

Answer 23 · 2016-03-02T01:04:33.000Z

@francois-keith good catch on the bug recently added to boost.cmake (see jrl-umi3218/jrl-cmakemodules#76). As for the differences when printing, there are some ideas there. The _set_output_format trick could be added to our test fixtures, thus it would be automatically applied to all the RobOptim tests (core, plugins...) by simply changing a single file.

Answer 24 · 2016-03-02T01:11:25.000Z

@francois-keith well apparently _set_output_format is obsolete, still any alternative workaround should probably be applied in our test fixtures.

Answer 25 · 2016-03-02T04:10:02.000Z

Complementary information here:

The removal of _set_output_format is by design. The standards-conforming two-digit exponent mode is now the default and the non-conforming, previously-default, three-digit exponent format is no longer supported. This is noted in the draft Visual C++ 2015 breaking changes documentation: https://msdn.microsoft.com/en-us/library/bb531344(v=vs.140).aspx. We have not yet completed the full updates to the documentation for Visual Studio 2015.

So we can just check the Visual C++ version, and use _set_output_format if an old version is detected.

Answer 26 · 2016-03-02T09:15:37.000Z

Awesome. Thank you

Answer 27 · 2016-03-07T12:04:21.000Z

Does anyone know how MSVC behaves with explicit template instantiations and symbol exports? The weird symbol exports I had all around the code solved this problem, i.e. gcc and clang behave differently (either one returns a type attributes ignored after type is already defined error, or the other one does not export the symbols).

Answer 28 · 2016-03-10T01:06:09.000Z

AFAIK, you can export the specialization of templated methods or methods of templated class.
I've put an example of the syntax in this repo : francois-keith/test_template_export@34e2e12

To be honest, it was much more tricky in my memory (cf stack-of-tasks/dynamic-graph@5721ccb98), so maybe I'm missing the point / the difficulty here.

Answer 29 · 2016-03-10T01:32:00.000Z

The problem is that gcc and clang behave quite differently, so I was wondering if msvc exhibited the same problem.

The following version works (at least the export) in clang but not gcc:

// In the header
template <typename T>
class Problem { ... };
...
extern template class ROBOPTIM_CORE_DLLAPI Problem<EigenMatrixDense>;
extern template class ROBOPTIM_CORE_DLLAPI Problem<EigenMatrixSparse>;

// In the implementation cpp
template class Problem<EigenMatrixDense>;
template class Problem<EigenMatrixSparse>;

With clang and the explicit template instantiation:

$ nm -CD src/libroboptim-core.so | grep "W roboptim::Problem<roboptim::EigenMatrixDense>::initialize()"
00000000000f5170 W roboptim::Problem<roboptim::EigenMatrixDense>::initialize()

Without it, the symbol is not exported as expected.

gcc error:

roboptim-core/include/roboptim/core/problem.hxx:656:46: error: type attributes ignored after type is already defined [-Werror=attributes]
   extern template class ROBOPTIM_CORE_DLLAPI Problem<EigenMatrixSparse>;
                                              ^

The following version works with both gcc and clang (at least the export), but will fail with msvc:

// In the header
template <typename T>
class ROBOPTIM_CORE_DLLAPI Problem { ... };
...
extern template class Problem<EigenMatrixDense>;
extern template class Problem<EigenMatrixSparse>;

// In the implementation cpp
template class Problem<EigenMatrixDense>;
template class Problem<EigenMatrixSparse>;

With gcc we finally get:

$ nm -CD src/libroboptim-core.so | grep "W roboptim::Problem<roboptim::EigenMatrixDense>::initialize()"
00000000000f9e10 W roboptim::Problem<roboptim::EigenMatrixDense>::initialize()

EDIT: removed ROBOPTIM_CORE_DLLEXPORT from implementation.

Answer 30 · 2016-05-30T22:13:42.000Z

Experimental Windows support is currently available on the dev branch. The next step would be to validate the changes with a solver plugin.