Invalid manifest format after upgrading from 0.3 to 0.5

[unicolet]

I just upgraded our project from plugin version 0.3 to 0.5 to benefit from the .tar support.
After upgrading, some .war files fail to start, the exception is:

SEVERE: ContainerBase.addChild: start: 
org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/app-ws]]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:153)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:725)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:701)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:939)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1812)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.catalina.LifecycleException: Failed to start component [org.apache.catalina.webresources.StandardRoot@6ed89845]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:153)
        at org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:4928)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5058)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
        ... 10 more
Caused by: org.apache.catalina.LifecycleException: Failed to initialize component [org.apache.catalina.webresources.JarResourceSet@71310103]
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:136)
        at org.apache.catalina.webresources.StandardRoot.startInternal(StandardRoot.java:699)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
        ... 13 more
Caused by: java.lang.IllegalArgumentException: java.io.IOException: invalid manifest format
        at org.apache.catalina.webresources.JarResourceSet.initInternal(JarResourceSet.java:96)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
        ... 16 more
Caused by: java.io.IOException: invalid manifest format
        at java.util.jar.Manifest.read(Manifest.java:225)
        at java.util.jar.Manifest.<init>(Manifest.java:69)
        at java.util.jar.JarFile.getManifestFromReference(JarFile.java:194)
        at java.util.jar.JarFile.getManifest(JarFile.java:180)
        at org.apache.catalina.webresources.JarResourceSet.initInternal(JarResourceSet.java:94)
        ... 17 more

unfortunately the logs are not giving much details :-(

I noticed this commit, which could be probable cause for the above error, but I can't see anything wrong with it.

Also I tried inspecting the MANIFEST.MF file from the war META-INF dir, but that specific file does not appear to be the problem, so it must be somewhere else.

[unicolet]

It appears the plugin can generate invalid MANIFEST.MF when an entry continues on a newline:

Specification-Title: JasperReports Library
                           
              Font Extension

After applying version 0.5 of this plugin the above would become (I removed other lines for clarity):

              Font Extension
Specification-Title: JasperReports Library

For now I'll fix the offending line, but IMHO this looks like a bug in the plugin :-)

[Zlika]

Yes it is (a bug). The sorting of manifest entries does not work on multiline entries. :-( I have to rework this. Thanks for reporting this bug.

[lefou]

This bug also strikes when creating OSGi bundles, which have a lot of multi-line manifest entries.

[Zlika]

@unicolet , @lefou : can you please compile the HEAD version of the plugin and check it on your project to validate my fix?
Thank you.

[lefou]

I extended the test case (on the command line) to cover some common cases, and it looks like the HEAD version does not fix the issue.

Here's my branch: https://github.com/lefou/reproducible-build-maven-plugin/tree/manifest-ordering

[Zlika]

Hello @lefou,
Ok I see the problem. The fix I added was to support multi-line attributes. And now with your new test case I see that I also forgot to support Manifest sections. I seems that this manifest-sorting feature is a little more complex that I thought :-)

[Zlika]

The last commit should fix the problem. Can you check @lefou and @unicolet ?

[lefou]

@Zlika This version "looks" better. But I can't finally test it yet, because of new issue #16 .

[lefou]

After your fix to #16, I get a "invalid manifest format" error. The content of the manifest looks visually OK though.

[Zlika]

Can you show me your manifest?

[lefou]

Before I do that, let me just describe my findings from comparing the source-manifest and the stripped one.

The JAR spec says:

No line may be longer than 72 bytes (not characters), in its UTF8-encoded form. If a value would make the initial line longer than this, it should be continued on extra lines (each starting with a single SPACE).

But the produced manifest breaks the lines after the 70th character. (Since all characters are pure ASCII symbols, I didn't checked the bytes count.) One line of the original Manifest is exactly 71 characters long and unfortunately, the last two characters are spaces. As the manifest stripper breaks the line differently my expectation was, that the next space character ends on the next line (after the space for line continuation). But that is not the case. Instead, there is an empty line following, which spuriously creates a new section. But as the line after that does not start with Name:, the JAR manifest is invalid. That reasoning is of course wild guessing, I didn't traced it.

[Zlika]

Can you show me your manifest before/after stripping? Because this plugin is not supposed to change or truncate the lines in the manifest, only change their order.

[Zlika]

Maybe I found the problem: the "Name" attribute must be the first attribute of each section (except the main section), and it was not necessarily the case before. Could you check the last version?

[lefou]

The PR #18 I created is enough to reproduce the manifest handling error I described above (#15 (comment)).

Doesn't the test fail for you? At least on travis-ci the test ManifestStripperTest.testStripManifest fails and shows the issue I had. Beside this special case, which I worked around by removing the trailing spaces, the build succeeded for me.

[lefou]

I made a comment to PR #18. I think, I found the issue.

[Zlika]

@lefou, @unicolet: is the HEAD version now ok for you?

[lefou]

@Zlika Yes, thank you. A new release would be great!

[unicolet]

@Zlika apologies for not helping here, but AFK because I've taken a few days off :-)

[lefou]

After reading the original issue again, it appears that @unicolet's manifest entries contains a whole almost empty (only space) line. It looks like my fix from #18 should also fix this. To be sure, it would be good to add the offending line to the unit test.

[Zlika]

@unicolet no problem. If you could just test the HEAD version on your project to be sure it fixes your problem, so that I could release a new version.

[Zlika]

I released version 0.5.1. Tell me if you find other bugs. Thanks for your help.

[unicolet]

@Zlika I just tested on my project and it produces an invalid manifest 😢 Later today I'll try to understand what really is going on. Thanks for your time on this