zerodha/dungbeetle

Improve the read/insert process in some way...

Closed this issue · 0 comments

The code inside cmd/jobber.go

func writeResults(jobID string, task *Task, ttl time.Duration, rows *sql.Rows, jobber *Jobber) (int64, error) {

for rows.Next() {
	if err := rows.Scan(resPointers...); err != nil {
		return numRows, err
	}
	if err := w.WriteRow(resCols); err != nil {
		return numRows, fmt.Errorf("error writing row to result backend: %v", err)
	}

	numRows++
	fmt.Println("Writing row to result backend: %d", numRows)
}

it's generic for all drivers thus database/sql dont support some kind of bulks... but if target backend support multiples rows in insert could be a simple aproach to improve velocity and could be configurable... #23