cerndb/hdfs-metadata

It would be great to add an option to dump raw aggregated data

Closed this issue · 6 comments

Tagar commented

Great tool. It would be very helpful to add an option to dump raw aggregated data.

For example, following output

DiskId 0 0 0 0 0 0 0 0 0 0 1 1
0 1 2 3 4 5 6 7 8 9 0 1 Unknown Count Average
Host
pc1udatahad11.xyz...= = = = = = = = = = 0 1470 147
pc1udatahad08.xyz...= = = = = = = = = = 0 773 77
pc1udatahad06.xyz...- + = - + - 0 331 55
skip..
pc1udatahad07.xyz...= = = = = = = = = = 0 783 78
pc1udatahad04.xyz...+ + = - = = 0 306 51
pc1udatahad10.xyz...= = = = = = = = = = = = 0 1158 96
pc1udatahad09.xyz...= = = = = = = = = = = = 0 1302 108

would be dumped just as a csv table with raw aggregated data.
We're planning to do some visualizations based on that.

Thanks.

Nice suggestion @Tagar , we will implement it soon

The feature has been implemented, argument that needs to be specified has been documented in the README.

Data is dumped in CSV format in order to make the reading easier.

In case you find any issue or would like a modification, do not hesitate to share it.

Tagar commented

Thanks a lot!!

Tagar commented

I ran this new version to dump data.
Here's what I got :

Host,0,1,2,3,4,5,6,7,8,9,10,11,Unknown,Count,Average
pc1udatahad05...,=,=,=,=,=,=,,,,,,,0,2454,409
pc1udatahad11...,=,=,=,=,=,=,=,=,=,=,,,0,9267,926
pc1udatahad06...,=,=,=,=,=,=,,,,,,,0,2459,409
pc1udatahad08...,=,=,=,=,=,=,=,=,=,=,,,0,7646,764
pc1udatahad12...,=,=,=,=,=,=,=,=,=,=,,,0,11280,1128
pc1udatahad07...,=,=,=,=,=,=,=,=,=,=,,,0,7637,763
pc1udatahad04...,=,+,=,=,=,=,,,,,,,0,2391,398
pc1udatahad01...,=,=,=,=,=,=,,,,,,,0,2290,381
pc1udatahad02...,=,=,=,=,=,=,,,,,,,0,2386,397
pc1udatahad10...,=,=,=,=,=,=,=,=,=,=,=,=,0,9725,810
pc1udatahad03...,=,=,=,=,=,=,,,,,,,0,2482,413
pc1udatahad13...,=,=,=,=,=,=,=,=,=,=,,,0,10383,1038
pc1udatahad09...,=,=,=,=,=,=,=,=,=,=,=,=,0,9644,803

So it dumped results (+,-,+ signs) and not actual numbers.
We would like to see raw number that you use to display those signs (+,-,+ ) - we want to embed
it to a system that has its own graphical visualization, so using just
+,-,+ signs wouldn't do much.

Thanks again.

Hi @Tagar ,

I have just done the modification.

Please feel free to do any modification in the code and create a merge request for future changes.

Cheers!

Tagar commented

Thanks a lot @dlanza1 !