is there any solution to get last record from stdout?
Opened this issue · 5 comments
hi i am jeff
i am now using embulk with little weird way.
- a batch python script generates some dynamic yaml files every some minutes.
- reads diff.yaml file which is generated by last batch job.
- applies last record into dynamic yaml file of 1.
- executes embulk several times with dynamic yaml files through python bash interface.
- drops diff.yaml file. within it, there is last record.
is there any better way to get last record?
or to call embulk in code way? not in bash way?
thanks
Hello, @makkaba
I think that I don't understand your use case completely yet.
If you execute Embulk from Python,
what do you think to use a Liquid template engine?
You just set an environment variable from Python.
Another idea.
embulk_test=# select * from incremental_test;
id | name
----+------
1 | var1
2 | var2
3 | var3
(3 rows)
in:
type: postgresql
host: localhost
port: 5432
user: user
password: ****
database: embulk_test
table: incremental_test
incremental: true
incremental_columns:
- id
out:
type: stdout
embulk run test.yml -c diff.yml
generate the following file
It output stdout like the following and create diff.yml file.
1,var1
2,var2
3,var3
diff.yml
in:
last_record: [3]
out: {}
The number 3
is the last record.
It is a YAML file. so I think you just create it with Python.
thank you for your reply.
i want to make sure my purpose:
last record = 300000
for 4
input => db
output => db [1,2,3,4]
i have already used diff.yaml file and dynamic yaml file.
like you said. (python template way)
every time yaml file is generated like this.
in:
type: sqlserver
driver_path: ~
host: ~
user: ~
password: ~
query: "SELECT ~ FROM WHERE ~ AND [idx] > :idx"
use_raw_query_with_incremental: true
incremental_columns:
- idx
incremental: true
last_record:
- 111111111
out:
type: mysql
host: ~
user: ~
password: ~
database: ~
table: ~
mode: merge
options: {useUnicode: true, characterEncoding: UTF-8}
but now, i want to get returned value by stdout or programatic return for some reason. (but output must be mysql.)
when i deal output with mysql, stdout will be kind of this..
********************************** INFORMATION **********************************
Join us! Embulk-announce mailing list is up for IMPORTANT announcement such as
compatibility-breaking changes and key feature updates.
https://groups.google.com/forum/#!forum/embulk-announce
java.lang.RuntimeException: java.nio.file.NoSuchFileException: sample.yaml
at org.embulk.EmbulkRunner.run(EmbulkRunner.java:152)
at org.embulk.cli.EmbulkRun.runSubcommand(EmbulkRun.java:437)
...
... 3 more
i just want to share use case.
thanks!
i want to get returned value by stdout or programatic return for some reason.
Embulk provides EmbulkEmbed that allows us to execute Embulk from Java program.
https://github.com/embulk/embulk/blob/master/embulk-core/src/main/java/org/embulk/EmbulkEmbed.java
We (Arm Treasure Data) are using this mechanism in our platform to execute Embulk from other codes.
Unfortunately, this mechanism is intended to be executed from Java, not Python.
EmbulkEmbed would be helpful.
thank you all !