Did not pull in Entire spreadsheet
Brendinho106 opened this issue · 5 comments
I tried to upload a large spreadsheet, and after a few minutes of loading excelastic returned the following:
Imported 44139 rows from 'pitching.xlsx' into october-2017 successfully.
However, when I go into Kibana/ES to do any searching I only have about 131 rows worth of data. Thoughts?
Hello! Are there any error messages in the log? The success message indicates that the rows are successfully received by ES. There could be type inconsistensies in the spreadsheet, or that types used in the spreadsheet are not mapped in excelastic.
Some things to try
- reformat the columns to the expected format
- check the data types
- are there any obvious differences between indexed and not indexed values?
- try indexing a subset of the spreadsheet
Could you upload the file? Would like to reproduce this issue.
Which version of ES are you using? Which version of the importer tool?
Hey Thanks for the response. I found what looks like one error in the log, which relates to one of the columns.
"Failed to execute bulk item bulkshardrequest"
That column is probably one of the less crucial ones so if that's a problem I can drop it.
I'm using ES 5.6.2, using Excelastic 1.2.2b
I have attached the document. Please let me know what your thoughts are.
Thanks,
-B
AMSReport.xlsx
I am also facing similar issue. data is showing added to index(yuge1 here). Total of 5032 records. But the docs count are 3308 and few columns are not showing up in Grafana/Kibana. Below is the log and have attached the excel sheet for you reference. :
Oct 04, 2017 4:53:29 PM com.codingchili.Model.ElasticWriter lambda$null$3
INFO: Submitted items [0 -> 254] of 5032 with result [200] OK into 'yuge1' [5.1%]
Oct 04, 2017 4:53:33 PM com.codingchili.Model.ElasticWriter lambda$null$3
INFO: Submitted items [255 -> 509] of 5032 with result [200] OK into 'yuge1' [10.1%]
Oct 04, 2017 4:53:36 PM com.codingchili.Model.ElasticWriter lambda$null$3
INFO: Submitted items [510 -> 764] of 5032 with result [200] OK into 'yuge1' [15.2%]
Oct 04, 2017 4:53:38 PM com.codingchili.Model.ElasticWriter lambda$null$3
INFO: Submitted items [765 -> 1019] of 5032 with result [200] OK into 'yuge1' [20.3%]
Oct 04, 2017 4:53:44 PM com.codingchili.Model.ElasticWriter lambda$null$3
INFO: Submitted items [1020 -> 1274] of 5032 with result [200] OK into 'yuge1' [25.3%]
Oct 04, 2017 4:53:49 PM com.codingchili.Model.ElasticWriter lambda$null$3
INFO: Submitted items [1275 -> 1529] of 5032 with result [200] OK into 'yuge1' [30.4%]
Oct 04, 2017 4:53:58 PM com.codingchili.Model.ElasticWriter lambda$null$3
INFO: Submitted items [1530 -> 1784] of 5032 with result [200] OK into 'yuge1' [35.5%]
Oct 04, 2017 4:54:03 PM com.codingchili.Model.ElasticWriter lambda$null$3
INFO: Submitted items [1785 -> 2039] of 5032 with result [200] OK into 'yuge1' [40.5%]
Oct 04, 2017 4:54:07 PM com.codingchili.Model.ElasticWriter lambda$null$3
INFO: Submitted items [2040 -> 2294] of 5032 with result [200] OK into 'yuge1' [45.6%]
Oct 04, 2017 4:54:14 PM com.codingchili.Model.ElasticWriter lambda$null$3
INFO: Submitted items [2295 -> 2549] of 5032 with result [200] OK into 'yuge1' [50.7%]
Oct 04, 2017 4:54:19 PM com.codingchili.Model.ElasticWriter lambda$null$3
INFO: Submitted items [2550 -> 2804] of 5032 with result [200] OK into 'yuge1' [55.7%]
Oct 04, 2017 4:54:23 PM com.codingchili.Model.ElasticWriter lambda$null$3
INFO: Submitted items [2805 -> 3059] of 5032 with result [200] OK into 'yuge1' [60.8%]
Oct 04, 2017 4:54:29 PM com.codingchili.Model.ElasticWriter lambda$null$3
INFO: Submitted items [3060 -> 3314] of 5032 with result [200] OK into 'yuge1' [65.9%]
Oct 04, 2017 4:54:32 PM com.codingchili.Model.ElasticWriter lambda$null$3
INFO: Submitted items [3315 -> 3569] of 5032 with result [200] OK into 'yuge1' [70.9%]
Oct 04, 2017 4:54:34 PM com.codingchili.Model.ElasticWriter lambda$null$3
INFO: Submitted items [3570 -> 3824] of 5032 with result [200] OK into 'yuge1' [76.0%]
Oct 04, 2017 4:54:36 PM com.codingchili.Model.ElasticWriter lambda$null$3
INFO: Submitted items [3825 -> 4079] of 5032 with result [200] OK into 'yuge1' [81.1%]
Oct 04, 2017 4:54:43 PM com.codingchili.Model.ElasticWriter lambda$null$3
INFO: Submitted items [4080 -> 4334] of 5032 with result [200] OK into 'yuge1' [86.1%]
Oct 04, 2017 4:54:45 PM com.codingchili.Model.ElasticWriter lambda$null$3
INFO: Submitted items [4335 -> 4589] of 5032 with result [200] OK into 'yuge1' [91.2%]
Oct 04, 2017 4:54:48 PM com.codingchili.Model.ElasticWriter lambda$null$3
INFO: Submitted items [4590 -> 4844] of 5032 with result [200] OK into 'yuge1' [96.3%]
Oct 04, 2017 4:54:51 PM com.codingchili.Model.ElasticWriter lambda$null$3
INFO: Submitted items [4845 -> 5031] of 5032 with result [200] OK into 'yuge1' [100.0%]
Thanks alot for providing more test data. Have not had time to look into this issue yet. Expect to look into this on friday.
The issue appears to be fixed in 1.2.3, I have tested both spreadsheets and all rows are imported successfully when checking with kibana.
Tested with Kibana 5.6.2 and ElasticSearch 5.6.2.
The issue was caused by two errors, one being that empty cells was not accounted for when counting the number of cells in a row. This lead to the data types being misaligned with the titles/data types. The second issue was that fields mapped as date types failed to import a row when the date cell was empty. Solved by not importing empty cells with value null or empty string.
Let me know if you have any further issues.