jar2xml is a tool to extract API information from a java archive
and associated documentation. The information is stored in XML format.
* Prerequisites
- mono (to run build tools)
- xmllint (for doc scraping)
- xmlstarlet (ditto)
Also it uses ASM (included in the sources)
* Tools
- jar2xml : the main converter to generate the primary target xmls.
- scraper.exe : annotation doc scraper to generate annotations/*.xml.
scraper-*.sh : shells used for scraper input. They are piped.
* <del>Easy</del> comparison
There are two levels of comparisons.
DLL: gui-compare in mono-tools is your friend. Compare new one against
old assembly of the corresponding API profile.
XML: It is now implemented as "make test-8" as well as "make test-13"
See Makefile for details. Things are not really simple to explain shortly.
$ java -jar jar2xml.jar --jar=/home/atsushi/android-sdk-linux_x86/platforms/android-8/android.jar --out=_8.xml --docpath=/home/atsushi/android-sdk-linux_x86/docs/reference
$ mono-xmltool --prettyprint 8.xml > 8_.xml
$ xmlstarlet c14n 8_.xml > 8.xml
$ mono-xmltool --prettyprint _8.xml > _8_.xml
$ xmlstarlet c14n _8_.xml > _8.xml
$ diff -u 8.xml _8.xml
* Difference from AOSP API XML
- Type parameters on Class and Method are now explicitly written as:
<typeParameters> <!-- if any -->
<typeParameter name="...">
<genericConstraints> <!-- if any -->
<genericConstraint type="..."></genericConstraint>
</genericConstraints>
</typeParameter>
</typeParameters>
see e.g. android.accounts.AccountManagerCallback<V>
There are some cases that we dare output type parameters embedded
into type name (basically those in XML attributes):
- genericConstraint/@type (see above)
- exception/@type (it would hardly happen though)
These are temporarily disabled and will be back when we really move
to the new generic design.
- class/@extends
- implements/@name
- field/@type
* TODO
** Postponed
- java.security.Provider.put() : missing in jar2xml output.
Disappeared in the shade of Hashtable.put().
-> workarounded by explicitly excluding the case (hardcoded in jar2xml!)
This issue should rather get fixed by improving overrload resolution
in XML level i.e. change XML format to include *any* overloaded methods
that AOSP XML does not do.
- java.net.ssl.KeyStoreBuilderParameters.Parameters and .ctor() : types differ
- generic list
- Now there are some abstract methods that somehow lacked in AOSP XML
and it uncovered issues regarding overriden methods. While those overriden
methods are sometimes "final", which should be "sealed", they are not
marked as sealed. And we cannot mark them as sealed right now since
those methods are *overriden* in *Invoker classes.
e.g. java.nio.Buffer.arrayOffset() and those overriden methods in *Buffer.
- android.hardware.SensorManager.GRAVITY_DEATH_STAR_I always returns 0
- while reference and AOSP indicates value, it returns 0 on reflection,
asm or even on ASUS Transformer.
- android.test.MoreAsserts.assertEquals(Set<?>, Set<?>).
They are just XML difference between <?> (wrong) and <? extends Object>
(correct), which affected doc scraping for parameter name retrieval.
- java.util.Calendar AllStyles, Long and Short are now properties, but
they should be actually enums. The thing is, they are *missing* in AOSP
XML so they had weren't converted to enums by our generator stuff.
- android.util.Pair : ThresholdClass and ThreasholdType are extraneous.
They are not in jar2xml output either, maybe added by api fixup stage.
(First and Second are looking as extra, but they were rather missing)
** things changes are maybe preferred
*** int->enum
- Android.App.DownloadManager
- Android.Appwidget.AppWidgetManager
- Android.Bluetooth.BluetoothA2dp
- Android.Bluetooth.BluetoothHeadset
- Android.Content.PM.PackageManager
- Android.Content.Res.Configuration
- Android.Database.Sqlite.SQLiteDatabase
- Android.Graphics.PowerDuff.Mode
- Android.Media.AudioManager.ScoAudioState*
- Android.OS.BatteryManager
- Android.Views.InputDevice
- Android.Views.View.Measured*
- Android.Views.Window.Progress*
- Android.Views.Window.WindowManagerLayoutParams
- Android.Views.View.SystemUiVisibilityChangeEventArgs has int Visibility
which should be StatusBarVisibility.
- Android.Widget.OnScrollListenerConsts
*** methods->properties
- Android.App.Admin.DevicePolicyManager: there are lots of GetPassword*() and
SetPassword*() methods which we might be able to change to indexers.
* Notes (that you don't have to read)
** API difference between android.jar and J2SE
These API differences could be problematic when we reflect java system types (reflection limitation).
- javax.net.ssl.ServerSocketFactory.getDefault() is synchronized in android
and not in J2SE.
-> in jar2xml, it is manually fixed.
** AOSP XML issues
Lots of missing pieces:
http://code.google.com/p/android/issues/detail?id=19569
** Notes on Annotations
Annotation such as java.lang.Deprecated do not seem to exist
in android.jar. That includes, types themselves.
java.lang.annotation types suffer from this.
I find it almost desparate and the only solution I couold think of was
to build AOSP android.jar or some equivalence that does *not* strip
those annotations off, *for each platform version*.
That sounds so time-eater.
The AOSP API document is generated as out/target/common/obj/PACKAGING/android_jar_intermediaries/public_api.xml (at that state there still must be annotations, or this xml should not contain annotation information at all).
Also,
out/target/common/obj/JAVA_LIBRARIES/framework_intermediaries and
out/target/common/obj/JAVA_LIBRARIES/core_intermediaries
contains debug jars that still seem to hold annotations.
development/tools/apkcheck/README.txt might be of interest too.
This is from the latest AOSP build (as of Aug.17, 2011). I'm not sure
if this applies to older versions.
On the other hand, I noticed that Annotation types are very easy to fill;
they have almost fixed members (from java.lang.Object). So I rather
think they can be manually filled (or if we want to be puritan, scrape
"Known Indirect Subclasses" from java/lang/annotation/Annotation.html).
Annotations are:
android.test.FlakyTest
android.test.UiThreadTest
android.test.suitebuilder.annotation.LargeTest
android.test.suitebuilder.annotation.MediumTest
android.test.suitebuilder.annotation.SmallTest
android.test.suitebuilder.annotation.Smoke
android.test.suitebuilder.annotation.Suppress
android.view.ViewDebug$CapturedViewProperty
android.view.ViewDebug$ExportedProperty
android.view.ViewDebug$FlagToString
android.view.ViewDebug$IntToString
android.widget.RemoteViews$RemoteView
dalvik.annotation.TestTarget
dalvik.annotation.TestTargetClass
java.lang.Deprecated
java.lang.Override
java.lang.SuppressWarnings
java.lang.annotation.Documented
java.lang.annotation.Inherited
java.lang.annotation.Retention
java.lang.annotation.Target
java.lang.annotation.Documented
** Old Notes on Annotations
Current implementation uses java.lang.reflect API to get member
annotations, but it won't work as expected.
I found that .class files in android.jar are stripped off field
annotations. For example, java.net.HttpURLConnection#HTTP_SERVER_ERROR
should be annotated with @Deprecated, but it isn't.
On the other hand, class annotations are alive.
(e.g. java/security/Certificate.class)
I guess is that AOSP uses proguard which strips member annotations.
The reason why we can somehow retrieve @Deprecated on
HttpURLConnection#HTTP_SERVER_ERROR by jar2xml is likely that
URLClassLoader.getClass() actually does not load java.* classes from
the jar contents but just returns the runtime one (such as openjdk).
Javassist.Loader javadoc explains its limitation on loading java.*
/ javax.* classes. [*1]
(So, it is not even about "runtime visible/invisible annotations".
I tried to read android.jar classes using ASM ClassReader [*2] and
it couldn't find any annotations on FieldNode (and from what I read
from asm sources, I believe it is by no means bug in ASM).
I have added separate doc scraper, which uses xmllint (to xmlize html docs)
and xmlstarlet (for convenient XPath query). Run "make annotations/13.xml"
for instance (should be available to all versions, I'm just lazy to hack
makefiles before confirming that it's all good).
[*1] http://www.csg.is.titech.ac.jp/~chiba/javassist/html/javassist/Loader.html
[*2] http://asm.ow2.org/asm33/javadoc/user/org/objectweb/asm/ClassReader.html
** Notes on per-version doc scraping
We used to scrape only reference for the latest API which was problematic.
Now we have version-aware scraper and resolved the issue described below.
<historical>
Doc scraper still misses some parameter names
e.g. android.text.method.ScrollingMovementMethod.onKeyUp()
This matters when api-fixup is involved (it matches nodes by parameter
names and then it dismisses the target nodes!).
Actually this is not going to be very good; there are some methods
that are removed in the latest API (the example method above is exactly
the case). We'll have to sort out some alternative approach.
A possible approach is to checkout API docs for *every* version (from
AOSP).
</historical>
** Byte code engineering
Now we use ASM to solve the issues below:
<historical>
- Some fields were incorrectly marked as constant. For example,
android.os.Build.TIME is not a constant, but since the field.getLong()
returns 0, it is set as 0 (and it does not return null for get()).
java.io.File.pathSeparatorChar has the same problem, but it is worse;
it returns non-zero value so we cannot depend on the value.
Solution ideas: scrape constant values, or use ASM.
- Constant values for "protected" fields need to be retrieved, but Java reflection API throws IllegalAccessException for such attempt.
This likely has to be resolved by bytecode engineering (such as ASM).
</historical>
** Issues regarding stripped non-public class (resolved)
<maybe-historical>
- java.lang.AbstractStringBuilder exposes an issue that
interface members on non-public types are returned by
class#getDeclaredMethods() while they are *not* declared.
Those methods (or those AbstractStringBuilder methods) are excluded by
checking special modifier 0x1000 (4096).
Also this involves check for derived methods (many overriden methods are
excluded, but this brings another check to *not* exclude methods when
the corresponding base methods are excluded).
</maybe-historical>
** Method override resolution
<maybe-historical>
- Do not skip certain overriden methods. This in JavaClass.java gives
equivalent output to AOSP xml, but it's not good:
if (base_method != null) {
int base_mods = base_method.getModifiers ();
if (!Modifier.isAbstract (base_mods) && (Modifier.isPublic (mmods) == Modifier.isPublic (base_mods)))
continue;
}
This causes GridView.setAdapter() ignored and thus fails to generate
valid property. It hides overriden methods.
I simply commented out, then a bunch of hidden overriden methods
appeared. And since we don't handle int->enum conversions on those
methods, they simply fail to compile. Maybe fixing above to match
existing format is a good-enough compromization.
Another notable thing is that AbsListView.setAdapter() does not
exist in api-4.xml. It is likely because abstract methods are
ignored (it is abstract, though that fact does not appear on
android javadoc).
android.view.MotionEvent.getSource() is also missing.
</maybe-historical>