Cluster documentation is generally an area that's not always given much attention in many corporate environments. Both generation of the cluster documentation and maintaining it as and when changes go in into the cluster configurations is takes a significant amount of time for the cluster administration teams. In most cases, logging in to Ambari is the most followed route for finding out certain FAQ.
- Eg
- What's our HDFS blocksize?
- Is LLAP enabled in this cluster ?
- What's the default replication for Kafka ?
- What's the NN URL ?
- What's the Hive JDBC URL ?
- Which nodes do we have these components installed ?
- What components are installed on this particular node ?
This project is an attempt to crack this problem using automation of generating the pages with the information available via Ambari REST API and other REST APIs in the cluster components.
The nested JSON structures emitted by Ambari REST responses are parsed and condensed into HTML output with only the most needed configurations displayed in the output.
The result is an HTML document that can be inserted into Confluence pages. At a later stage, I plan to add direct publishing of contents to Confluence via it's REST APIs. There's some fancy graph representations doable for host component representation as a graph somewhere along the lines of https://martin.atlassian.net/wiki/spaces/lestermartin/pages/1019871233/Hadoop+Component+Dependency+Graph