Plan to replace `embulk-output-bigquery` with trocco-io's `embulk-output-bigquery_java`
dmikurube opened this issue · 4 comments
embulk-output-bigquery has a long history. It started as a File Input plugin written in Java, and then it was totally rewritten as an Input Plugin written in Ruby.
Now, Embulk is going to make JRuby optional through v0.10. Maintaining Ruby plugins is less prioritized.
In addition, we are seeing several problems around embulk-output-bigquery -- although it does not look like a problem of the plugin, but a kind of JRuby and JRE.
https://github.com/trocco-io has developed and is maintaining embulk-output-bigquery_java, which is a Java-based plugin aimed to be compatible with existing Ruby-based embulk-output-bigquery.
We've discussed with https://github.com/trocco-io, and we agreed to replace the existing embulk-output-bigquery with their embulk-output-bigquery_java after it achieves sufficient compatibility.
@potato2003 Thanks for joining the maintainance of embulk-output-bigquery recently. As described above, we have a plan to migrate it to a Java version. Let me describe some more details for you.
- The Trocco team is going to talk about the migration plan in the meetup this week.
- After the migration happens, the Trocco team will stay as maintainers of
embulk-output-bigquery. You can choose either:- to stay as a maintainer even in the new Java version, or
- to leave after the new Java version.
- We are discussing the plan in the developer Slack workspace. Join us if you're interested!
@dmikurube Thank you for your description of the migration plan.
After the migration happens, the Trocco team will stay as maintainers of embulk-output-bigquery. You can choose either:
I will leave maintainers.
Sure. Thank you for your contribution!
From a view of an author of JRuby version of embulk-output-bigquery.
The JRuby version of embulk-output-bigquery has had an issue that formatter performance is worse than Java version.
As I described at https://github.com/embulk/embulk-output-bigquery#formatter-performance-issue, we could avoid the problem using https://github.com/civitaspo/embulk-filter-to_json. But, I admit it is tricky.
I wanted to resolve the issue by replacing the embulk-output-bigquery by rewriting with Java, but I could not have time to do it. But, trocco-io did it instead of me. Thanks!
The Java version should work better than JRuby version in terms of performance. Stay tuned.