Error on flamegraph output
Closed this issue · 13 comments
danosipov commented
I ran the profiler, and see the CPU trace metrics in graphite. However, when I ran graphite_dump.py, it fails with this error:
$ /usr/local/graphite_dump.py -o 127.0.0.1 -s 20:25_20150312 -e 20:50_20150312 -p stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace
Traceback (most recent call last):
File "/usr/local/graphite_dump.py", line 82, in <module>
format_output(args.prefix, results)
File "/usr/local/graphite_dump.py", line 67, in format_output
print '%s %d' % (format_metric(metric, prefix), value)
TypeError: %d format: a number is required, not NoneType
Adding a null check to value on line 67 produces no output. Let me know if the error is in my StatsD/Graphite setup (I've used a default configuration)
ajsquared commented
I will take a look at this and let you know what I find.
ajsquared commented
Could you give me a list of the metrics that appear under stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace?
danosipov commented
Here is the output from http://graphite.host/metrics/expand?query=stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.*
Let me know if it contains what you're looking for
{"results": ["stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.10", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.149", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.cascading-flow-planner-BaseFlowStep-getPathsBetween", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.cascading-pipe-Operator-equals", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.cascading-scheme-hadoop-TextLine-sourceHandleInput", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.cascading-tuple-Tuple-size", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-annotation-JsonAutoDetectVisibility-isVisible", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-core-json-JsonReadContext-init", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-core-json-ReaderBasedJsonParser-_finishString2", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-core-sym-CharsToNameCanonicalizer-hashSeed", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-BeanDescription-getBeanClass", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-DeserializationConfig-introspectForCreation", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-ObjectMapper-clinit", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-ObjectMapper-init", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-cfg-DeserializerFactoryConfig-hasDeserializerModifiers", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-cfg-MapperConfig-constructSpecializedType", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-deser-BasicDeserializerFactory-createCollectionDeserializer", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-deser-BasicDeserializerFactory-findDefaultDeserializer", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-deser-DeserializerCache-_createAndCache2", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-deser-DeserializerCache-findKeyDeserializer", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-deser-DeserializerCache-modifyTypeByAnnotation", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-deser-std-MapDeserializer-_readAndBindStringMap", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-deser-std-MapDeserializer-withResolved", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-introspect-Annotated-hasAnnotation", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-introspect-AnnotatedClass-_collectRelevantAnnotations", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-introspect-AnnotatedClass-construct", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-introspect-AnnotatedClass-fields", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-introspect-AnnotatedClass-memberMethods", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-introspect-AnnotatedClass-resolveFields", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-introspect-AnnotatedClass-resolveMemberMethods", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-introspect-AnnotatedMethodMap-iterator", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-introspect-AnnotationMap-addIfNotPresent", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-introspect-AnnotationMap-get", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-introspect-BasicClassIntrospector-clinit", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-introspect-JacksonAnnotationIntrospector-findNameForDeserialization", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-introspect-JacksonAnnotationIntrospector-findNamingStrategy", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-introspect-JacksonAnnotationIntrospector-findPOJOBuilder", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-introspect-JacksonAnnotationIntrospector-hasAnyGetterAnnotation", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-introspect-JacksonAnnotationIntrospector-isAnnotationBundle", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-introspect-POJOPropertiesCollector-_addGetterMethod", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-introspect-POJOPropertiesCollector-_addInjectables", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-introspect-POJOPropertiesCollector-_renameProperties", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-introspect-POJOPropertyBuilder-_anyIgnorals", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-ser-BasicSerializerFactory-clinit", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-ser-std-StdJdkSerializers-all", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-type-SimpleType-init", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-databind-util-LRUMap-get", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-module-scala-deser-SeqDeserializerResolver-SEQ", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-module-scala-deser-UnsortedMapDeserializer-builderFor", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-module-scala-deser-UnsortedSetDeserializerResolver-findCollectionLikeDeserializer", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-module-scala-introspect-ScalaClassIntrospectoranonfunisScalaPackage1-init", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-module-scala-modifiers-CollectionLikeTypeModifierclass-modifyType", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-module-scala-modifiers-GenTypeModifierclass-classObjectFor", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-fasterxml-jackson-module-scala-util-ClassWclass-hasSigHelper1", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-google-protobuf-GeneratedMessageBuilder-onBuilt", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-shazam-JacksonWrapperanon2-getRawType", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-shazam-TitleExtractoranonfun3-apply", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-shazam-TitleExtractoranonfun7-apply", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-twitter-chill-Externalizer-get", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-twitter-chill-ScalaCollectionsRegistrar-apply", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-twitter-chill-ScalaTupleSerializationanon1-apply", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-twitter-scalding-typed-MapFn-apply", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-io-ByteArrayOutputStream-size", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-io-InputStream-init", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-io-ObjectStreamClass-hasStaticInitializer", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-io-UnixFileSystem-getBooleanAttributes", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-io-UnixFileSystem-getBooleanAttributes0", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-Class-forName0", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-Class-getAnnotation", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-Class-getClassLoader0", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-Class-getConstantPool", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-Class-getDeclaredConstructors0", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-Class-getDeclaredFields0", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-Class-getDeclaredMethods", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-Class-getDeclaringClass", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-Class-getEnclosingMethod0", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-Class-getGenericSignature", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-Class-getInterfaces", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-Class-getModifiers", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-Class-getName0", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-Class-initAnnotationsIfNecessary", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-Class-isAssignableFrom", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-Class-isPrimitive", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-ClassLoader-checkCerts", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-ClassLoader-checkName", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-ClassLoader-defineClass1", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-ClassLoader-findBootstrapClass", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-ClassLoader-findLoadedClass0", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-ClassLoaderNativeLibrary-load", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-Object-clone", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-Object-getClass", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-Object-hashCode", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-String-checkBounds", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-StringCoding-deref", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-StringCoding-encode", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-StringCodingStringEncoder-encode", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-System-arraycopy", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-System-currentTimeMillis", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-System-identityHashCode", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-System-nanoTime", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-Throwable-fillInStackTrace", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-reflect-AccessibleObject-setAccessible0", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-reflect-Constructor-getParameterAnnotations", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-reflect-Method-getDeclaredAnnotations", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-reflect-Modifier-isTransient", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-net-Inet4AddressImpl-lookupAllHostAddr", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-net-PlainSocketImpl-socketConnect", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-net-PlainSocketImpl-socketSetOption", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-net-SocketInputStream-socketRead0", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-net-URL-getUserInfo", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-net-URLClassLoader-findClass", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-net-URLStreamHandler-parseURL", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-net-URLStreamHandler-toExternalForm", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-nio-DirectLongBufferU-get", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-nio-DirectLongBufferU-put", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-nio-charset-CharsetEncoder-init", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-security-AccessController-doPrivileged", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-util-AbstractCollection-toArray", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-util-ArrayList-contains", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-util-ArrayList-grow", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-util-Collections-emptyIterator", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-util-HashMap-addEntry", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-util-HashMap-getEntry", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-util-HashMap-hash", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-util-HashMap-inflateTable", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-util-HashMap-resize", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-util-HashMapValues-iterator", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-util-Vector-init", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-util-WeakHashMap-maskNull", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-util-WeakHashMap-put", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-util-concurrent-ConcurrentHashMap-hash", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-util-concurrent-ConcurrentHashMap-putIfAbsent", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-util-zip-Inflater-inflateBytes", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-util-zip-Inflater-reset", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-util-zip-Inflater-setInput", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-util-zip-InflaterInputStream-init", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-util-zip-ZipCoder-getBytes", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-util-zip-ZipFile-getEntry", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-util-zip-ZipFile-read", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.javax-crypto-Cipher-tokenizeTransformation", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.org-apache-hadoop-fs-FSDataOutputStreamPositionCache-write", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.scala-collection-AbstractSet-init", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.scala-collection-IndexedSeqLikeElements-next", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.scala-collection-TraversableLikeclass-builder1", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.scala-collection-TraversableOnce-MonadOps", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.scala-collection-TraversableOnceMonadOps-init", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.scala-collection-immutable-List-length", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.scala-collection-immutable-List-newBuilder", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.scala-collection-immutable-MapEmptyMap-updated", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.scala-collection-immutable-MapMap4-get", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.scala-collection-immutable-MapMap4-init", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.scala-collection-immutable-MapMap4-updated", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.scala-collection-immutable-SetEmptySet-plus", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.scala-collection-immutable-StringOps-drop", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.scala-collection-immutable-StringOps-slice", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.scala-collection-mutable-BitSet-empty", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.scala-collection-mutable-Bufferclass-init", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.scala-collection-mutable-DefaultEntry-init", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.scala-collection-mutable-ListBuffer-exported_eq", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.scala-collection-mutable-ListBuffer-pluseq", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.scala-collection-mutable-ListBuffer-result", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.scala-collection-mutable-ListBuffer-toList", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.scala-collection-mutable-WrappedArrayofRef-length", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.scala-math-Ordering-init", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.scala-package-init", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.sun-misc-Unsafe-compareAndSwapLong", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.sun-misc-Unsafe-getObject", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.sun-misc-Unsafe-getObjectVolatile", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.sun-misc-Unsafe-putInt", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.sun-misc-Unsafe-putLong", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.sun-misc-Unsafe-putOrderedObject", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.sun-net-www-ParseUtil-canonizeString", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.sun-net-www-ParseUtil-init", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.sun-net-www-protocol-jar-Handler-parseURL", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.sun-reflect-ClassFileAssembler-append", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.sun-reflect-Reflection-getCallerClass", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.sun-reflect-generics-factory-CoreReflectionFactory-getScope", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.sun-reflect-generics-parser-SignatureParser-parseTypeVariableSignature", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.sun-reflect-generics-parser-SignatureParser-parseZeroOrMoreThrowsSignatures", "stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.sun-security-x509-X509AttributeName-init"]}
ajsquared commented
That looks pretty reasonable. Could you apply this patch https://gist.github.com/ajsquared/9fb38229132d19e750f5 and run the script again? It will produce some debugging output that may help track down the issue.
danosipov commented
Lots o' output, but looks like json_results is empty
Truncated:
Bounds: 10 149
[u'stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.cascading-scheme-hadoop-TextLine-sourceHandleInput.cascading-scheme-hadoop-TextLine-source.cascading-tuple-TupleEntrySchemeIterator-getNext.cascading-tuple-TupleEntrySchemeIterator-hasNext.cascading-flow-stream-SourceStage-map.cascading-flow-stream-SourceStage-run.cascading-flow-hadoop-FlowMapper-run.org-apache-hadoop-mapred-MapTask-runOldMapper.org-apache-hadoop-mapred-MapTask-run.org-apache-hadoop-mapred-Child4-run.java-security-AccessController-doPrivileged.javax-security-auth-Subject-doAs.org-apache-hadoop-security-UserGroupInformation-doAs.org-apache-hadoop-mapred-Child-main', u'stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-twitter-scalding-typed-MapFn-apply.com-twitter-scalding-FlatMapFunction-operate.cascading-flow-stream-FunctionEachStage-receive.cascading-flow-stream-FunctionEachStage-receive.cascading-flow-stream-SourceStage-map.cascading-flow-stream-SourceStage-run.cascading-flow-hadoop-FlowMapper-run.org-apache-hadoop-mapred-MapTask-runOldMapper.org-apache-hadoop-mapred-MapTask-run.org-apache-hadoop-mapred-Child4-run.java-security-AccessController-doPrivileged.javax-security-auth-Subject-doAs.org-apache-hadoop-security-UserGroupInformation-doAs.org-apache-hadoop-mapred-Child-main', u'stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-System-currentTimeMillis.cascading-tap-hadoop-util-MeasuredRecordReader-next.cascading-scheme-hadoop-TextLine-sourceReadInput.cascading-scheme-hadoop-TextLine-source.cascading-tuple-TupleEntrySchemeIterator-getNext.cascading-tuple-TupleEntrySchemeIterator-hasNext.cascading-flow-stream-SourceStage-map.cascading-flow-stream-SourceStage-run.cascading-flow-hadoop-FlowMapper-run.org-apache-hadoop-mapred-MapTask-runOldMapper.org-apache-hadoop-mapred-MapTask-run.org-apache-hadoop-mapred-Child4-run.java-security-AccessController-doPrivileged.javax-security-auth-Subject-doAs.org-apache-hadoop-security-UserGroupInformation-doAs.org-apache-hadoop-mapred-Child-main', u'stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.com-shazam-TitleExtractoranonfun3-apply.scala-collection-Iteratoranon11-next.scala-collection-Iteratorclass-foreach.scala-collection-AbstractIterator-foreach.com-twitter-scalding-FlatMapFunction-operate.cascading-flow-stream-FunctionEachStage-receive.cascading-flow-stream-FunctionEachStage-receive.cascading-flow-stream-SourceStage-map.cascading-flow-stream-SourceStage-run.cascading-flow-hadoop-FlowMapper-run.org-apache-hadoop-mapred-MapTask-runOldMapper.org-apache-hadoop-mapred-MapTask-run.org-apache-hadoop-mapred-Child4-run.java-security-AccessController-doPrivileged.javax-security-auth-Subject-doAs.org-apache-hadoop-security-UserGroupInformation-doAs.org-apache-hadoop-mapred-Child-main', u'stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.java-lang-String-checkBounds.java-lang-String-init.cascading-scheme-hadoop-TextLine-makeEncodedString.cascading-scheme-hadoop-TextLine-sourceHandleInput.cascading-scheme-hadoop-TextLine-source.cascading-tuple-TupleEntrySchemeIterator-getNext.cascading-tuple-TupleEntrySchemeIterator-hasNext.cascading-flow-stream-SourceStage-map.cascading-flow-stream-SourceStage-run.cascading-flow-hadoop-FlowMapper-run.org-apache-hadoop-mapred-MapTask-runOldMapper.org-apache-hadoop-mapred-MapTask-run.org-apache-hadoop-mapred-Child4-run.java-security-AccessController-doPrivileged.javax-security-auth-Subject-doAs.org-apache-hadoop-security-UserGroupInformation-doAs.org-apache-hadoop-mapred-Child-main', ...]
[]
ajsquared commented
Hmm that is pretty odd. If you pick one of the leaf metrics (e.g stats.gauges.bigdata.profiler.hadoop.com.shazam.TitleExtractor.C83612C60A954CB4A442C56597331C5E.1.cpu.trace.cascading-scheme-hadoop-TextLine-sourceHandleInput.cascading-scheme-hadoop-TextLine-source.cascading-tuple-TupleEntrySchemeIterator-getNext.cascading-tuple-TupleEntrySchemeIterator-hasNext.cascading-flow-stream-SourceStage-map.cascading-flow-stream-SourceStage-run.cascading-flow-hadoop-FlowMapper-run.org-apache-hadoop-mapred-MapTask-runOldMapper.org-apache-hadoop-mapred-MapTask-run.org-apache-hadoop-mapred-Child4-run.java-security-AccessController-doPrivileged.javax-security-auth-Subject-doAs.org-apache-hadoop-security-UserGroupInformation-doAs.org-apache-hadoop-mapred-Child-main), do you see any values in Graphite for it?
Also, what version of Graphite are you using?
danosipov commented
Hmm, they appear empty. That might be a problem
I'm using Graphite 0.9.13, with StatsD running on the same server.
ajsquared commented
I'm on 0.9.12, but I'm not aware of any changes in 0.9.13 that would cause issues like this. The metrics definitely seem to be making it to Graphite or else you wouldn't even see the list of leaves. Are they entry for all time or just the time range you'd expect values there? I am wondering if the values were record from a different time zone or something to that effect.
danosipov commented
I'm attempting to rerun the profiler, but now my bound metrics are missing. Do they appear after some time?
The metrics are empty for all time. I was running StatsD with debug mode turned on to see that connections were working, I've restarted it in non-debug mode.
The heap/gc metrics are in place and contain data.
ajsquared commented
The bounds would appear at the end of profiling. StatsD might be dropping packets.
ajsquared commented
Did you have any luck with StatsD in non-debug mode?
danosipov commented
Unfortunately no, but I'm investigating if StatsD or Graphite are dropping stats. I'm following the advice given here: http://stackoverflow.com/a/18491559/680399
Let me know if you know any other places to check.
Also, I guess this issue can be closed, as the flamegraph generation is probably fine.
ajsquared commented
That does look like a good place to start. I will close for now, but if you run into other problems with it let me know!