Code fails to compile on Mac (issue with AVX intrinsics)
eikehmueller opened this issue · 6 comments
eikehmueller commented
Compilation on my MacBook fails with error messages of the type:
[…]
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccIqsgIb.s:9:no such instruction: `vxorpd %xmm5, %xmm5,%xmm5'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccIqsgIb.s:12:no such instruction: `vmovsd LC1(%rip), %xmm2’
[…]
This can be fixed by adding the compiler flag -Wa,-q, see following patch:
diff --git a/pyop2/compilation.py b/pyop2/compilation.py
index ec7d7af..2daf491 100644
--- a/pyop2/compilation.py
+++ b/pyop2/compilation.py
@@ -210,7 +210,7 @@ class MacCompiler(Compiler):
if cpp:
cc = "mpicxx"
stdargs = []
- cppargs = stdargs + ['-fPIC', '-Wall', '-framework', 'Accelerate'] + \
+ cppargs = stdargs + ['-Wa,-q','-fPIC', '-Wall', '-framework', 'Accelerate'] + \
opt_flags + cppargs
ldargs = ['-dynamiclib'] + ldargs
super(MacCompiler, self).__init__(cc,
I only tested this on my macbook, the used compiler is gcc:
eikemueller@Eikes-MBP $ mpicxx --version
g++ (Homebrew gcc48 4.8.3) 4.8.3
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
FabioLuporini commented
this honestly doesn't look like stuff that coffee spits out. Can you paste here the whole compilation line? What if you replace that option with just -O3 ?
eikehmueller commented
986dce2f61c72a82742bf5109164330b_p7138.txt
Compilation line is (so the -O3 flag is already included):
mpicc -std=c99 -fPIC -Wall -framework Accelerate -march=native -O3 -I/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/include -I/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/arch-darwin-c-opt/include -I/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2 -msse -o /var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T/pyop2-cache-uid501/986dce2f61c72a82742bf5109164330b_p7138.so.tmp /var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T/pyop2-cache-uid501/986dce2f61c72a82742bf5109164330b_p7138.c -dynamiclib -L/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/lib -L/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/arch-darwin-c-opt/lib -Wl,-rpath,/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/lib -Wl,-rpath,/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/arch-darwin-c-opt/lib -lpetsc -l
The full error message is this:
File "driver.py", line 652, in <module>
main(parameter_filename)
File "driver.py", line 569, in main
mesh_hierarchy = build_mesh_hierarchy(param_grid,param_orography)
File "driver.py", line 166, in build_mesh_hierarchy
layer_height= thickness/nlayer)
File "/Users/eikemueller/PostDocBath/EllipticSolvers/firedrake/firedrake/mg/mesh.py", line 121, in __init__
for lvl, m in enumerate(mesh_hierarchy)]
File "<string>", line 2, in __init__
File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/profiling.py", line 199, in wrapper
return f(*args, **kwargs)
File "/Users/eikemueller/PostDocBath/EllipticSolvers/firedrake/firedrake/mesh.py", line 901, in __init__
kernel=kernel)
File "/Users/eikemueller/PostDocBath/EllipticSolvers/firedrake/firedrake/extrusion_utils.py", line 236, in make_extruded_coords
height(op2.READ))
File "<string>", line 2, in par_loop
File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/versioning.py", line 154, in modifies_arguments
retval = func(*args, **kwargs)
File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/op2.py", line 328, in par_loop
return backends._BackendSelector._backend.par_loop(kernel, iterset, *args, **kwargs)
File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/base.py", line 4588, in par_loop
return _make_object('ParLoop', kernel, it_space, *args, **kwargs).enqueue()
File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/base.py", line 84, in enqueue
_trace.append(self)
File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/base.py", line 101, in append
computation._run()
File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/base.py", line 4045, in _run
return self.compute()
File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/base.py", line 4276, in compute
fun = self._jitmodule
File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/utils.py", line 64, in __get__
obj.__dict__[self.__name__] = result = self.fget(obj)
File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/sequential.py", line 147, in _jitmodule
direct=self.is_direct, iterate=self.iteration_region)
File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/caching.py", line 203, in __new__
obj = make_obj()
File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/caching.py", line 193, in make_obj
obj.__init__(*args, **kwargs)
File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/host.py", line 687, in __init__
self.compile()
File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/host.py", line 788, in compile
compiler=compiler.get('name'))
File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/compilation.py", line 298, in load
dll = compiler.get_so(src, extension)
File "/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2/compilation.py", line 144, in get_so
Original error: %s""" % (cc, logfile, errfile, e))
pyop2.exceptions.CompilationError: Command "['mpicc', '-std=c99', '-fPIC', '-Wall', '-framework', 'Accelerate', '-march=native', '-O3', '-I/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/include', '-I/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/arch-darwin-c-opt/include', '-I/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2', '-msse', '-o', '/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T/pyop2-cache-uid501/986dce2f61c72a82742bf5109164330b_p7138.so.tmp', '/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T/pyop2-cache-uid501/986dce2f61c72a82742bf5109164330b_p7138.c', '-dynamiclib', '-L/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/lib', '-L/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/arch-darwin-c-opt/lib', '-Wl,-rpath,/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/lib', '-Wl,-rpath,/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/arch-darwin-c-opt/lib', '-lpetsc', '-lm']" returned with error.
Unable to compile code
Compile log in /var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T/pyop2-cache-uid501/986dce2f61c72a82742bf5109164330b_p7138.log
Compile errors in /var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T/pyop2-cache-uid501/986dce2f61c72a82742bf5109164330b_p7138.err
Original error: status 1 invoking 'mpicc -std=c99 -fPIC -Wall -framework Accelerate -march=native -O3 -I/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/include -I/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/arch-darwin-c-opt/include -I/Users/eikemueller/PostDocBath/EllipticSolvers/PyOP2/pyop2 -msse -o /var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T/pyop2-cache-uid501/986dce2f61c72a82742bf5109164330b_p7138.so.tmp /var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T/pyop2-cache-uid501/986dce2f61c72a82742bf5109164330b_p7138.c -dynamiclib -L/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/lib -L/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/arch-darwin-c-opt/lib -Wl,-rpath,/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/lib -Wl,-rpath,/Users/eikemueller/PostDocBath/EllipticSolvers/petsc/arch-darwin-c-opt/lib -lpetsc -lm'
And the contents of the file /var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T/pyop2-cache-uid501/986dce2f61c72a82742bf5109164330b_p7138.err are:
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:9:no such instruction: `vxorpd %xmm5, %xmm5,%xmm5'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:12:no such instruction: `vmovsd LC1(%rip), %xmm2'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:15:no such instruction: `vmovsd (%rax), %xmm0'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:16:no such instruction: `vmulsd %xmm0, %xmm0,%xmm0'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:17:no such instruction: `vaddsd %xmm5, %xmm0,%xmm1'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:18:no such instruction: `vmovsd 8(%rax), %xmm0'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:19:no such instruction: `vmulsd %xmm0, %xmm0,%xmm0'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:20:no such instruction: `vaddsd %xmm0, %xmm1,%xmm1'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:21:no such instruction: `vmovsd 16(%rax), %xmm0'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:22:no such instruction: `vmulsd %xmm0, %xmm0,%xmm0'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:23:no such instruction: `vaddsd %xmm0, %xmm1,%xmm0'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:24:no such instruction: `vsqrtsd %xmm0, %xmm1,%xmm1'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:25:no such instruction: `vucomisd %xmm1, %xmm1'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:33:no such instruction: `vcvtsi2sd %r10d, %xmm3,%xmm3'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:36:no such instruction: `vmulsd (%rcx), %xmm3,%xmm0'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:37:no such instruction: `vdivsd %xmm1, %xmm0,%xmm0'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:38:no such instruction: `vaddsd %xmm2, %xmm0,%xmm0'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:39:no such instruction: `vmulsd (%rax), %xmm0,%xmm0'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:40:no such instruction: `vmovsd %xmm0, (%r9)'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:41:no such instruction: `vcvtsi2sd %r10d, %xmm0,%xmm0'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:42:no such instruction: `vmulsd (%rcx), %xmm0,%xmm4'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:43:no such instruction: `vdivsd %xmm1, %xmm4,%xmm4'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:44:no such instruction: `vaddsd %xmm2, %xmm4,%xmm4'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:45:no such instruction: `vmulsd (%rax), %xmm4,%xmm4'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:46:no such instruction: `vmovsd %xmm4, (%r8)'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:47:no such instruction: `vmulsd (%rcx), %xmm3,%xmm4'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:48:no such instruction: `vdivsd %xmm1, %xmm4,%xmm4'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:49:no such instruction: `vaddsd %xmm2, %xmm4,%xmm4'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:50:no such instruction: `vmulsd 8(%rax), %xmm4,%xmm4'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:51:no such instruction: `vmovsd %xmm4, 8(%r9)'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:52:no such instruction: `vmulsd (%rcx), %xmm0,%xmm4'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:53:no such instruction: `vdivsd %xmm1, %xmm4,%xmm4'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:54:no such instruction: `vaddsd %xmm2, %xmm4,%xmm4'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:55:no such instruction: `vmulsd 8(%rax), %xmm4,%xmm4'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:56:no such instruction: `vmovsd %xmm4, 8(%r8)'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:57:no such instruction: `vmulsd (%rcx), %xmm3,%xmm3'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:58:no such instruction: `vdivsd %xmm1, %xmm3,%xmm3'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:59:no such instruction: `vaddsd %xmm2, %xmm3,%xmm3'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:60:no such instruction: `vmulsd 16(%rax), %xmm3,%xmm3'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:61:no such instruction: `vmovsd %xmm3, 16(%r9)'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:62:no such instruction: `vmulsd (%rcx), %xmm0,%xmm0'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:63:no such instruction: `vdivsd %xmm1, %xmm0,%xmm1'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:64:no such instruction: `vaddsd %xmm2, %xmm1,%xmm1'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:65:no such instruction: `vmulsd 16(%rax), %xmm1,%xmm1'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:66:no such instruction: `vmovsd %xmm1, 16(%r8)'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:75:no such instruction: `vmovsd %xmm2, 40(%rsp)'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:76:no such instruction: `vmovsd %xmm5, 32(%rsp)'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:83:no such instruction: `vmovapd %xmm0, %xmm1'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:84:no such instruction: `vmovsd 40(%rsp), %xmm2'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//ccy7Axjb.s:85:no such instruction: `vmovsd 32(%rsp), %xmm5'
wence- commented
I think this occurs because the linker you use doesn't understand the avx instructions set. Passing -Wa,-q forces the compiler to always use the system linker which does understand things. Plausibly this is always safe on Mac.
eikehmueller commented
Thanks! I think there’s something not quite right with my mac, and the compiler toolchain is screwed up (gcc broke when I upgraded to Yosemite a year ago, so this might still be the aftermath...).
The error could be related to the issues I’m seeing with building PETSc (still not resolved - I’m in touch with Satish on petsc@maint). We located the error to this line now:
mpif90 -c -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -g -O sltimer.f
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//cccVkhkX.s:85:suffix or operands invalid for `movq'
/var/folders/yy/9wfgymp14_1b9zt2x0ycs64m0000gn/T//cccVkhkX.s:96:suffix or operands invalid for `movq’
This looks different since it is a compile error, not a linker error.
I will try to rebuild my entire homebrew tomorrow, maybe this helps.
On 17 Nov 2015, at 19:45, Lawrence Mitchell notifications@github.com wrote:
I think this occurs because the linker you use doesn't understand the avx instructions set. Passing -Wa,-q forces the compiler to always use the system linker which does understand things. Plausibly this is always safe on Mac.
—
Reply to this email directly or view it on GitHub #471 (comment).
wence- commented
I suspect it very probably has the same root cause.
eikehmueller commented
It was my compiler toolchain that was broken. I just upgraded my MacBook to OSX El Capitan, updated XCode and reinstalled all homebrew packages. After this I don't need the "-Wa,-q" flag any more (and PETSc compiles without problem as well).