hrpc.NumberOfRows is not working
LG991103 opened this issue · 11 comments
am i using the scan methond wrong way?
// NumberOfRows is an option for scan requests.
// Specifies how many rows are fetched with each request to regionserver.
// Should be > 0, avoid extremely low values such as 1 because a request
// to regionserver will be made for every row.
NumberOfRows doesn't limit how many rows are returned by the scan in total. It limit how many rows are returned by each scans request send to HBase.
A scan make multiples requests to regionservers. NumberOfRows only limit the number of rows returned PER requests, not for the totality of the scan.
This will do what you want, get 2 results and stop:
hrpc.NewScanRangeStr(ctx, table, startRow, stopRow, hrpc.NumberOfRows(2), hrpc.CloseScanner())CloseScanner will close the scanner immediately after the first results are returned.
Another option, especially if you want to follow the same pattern for a larger number of rows or if the rows are spread on multiple regionservers, is to exit the for loop when you have reached the size you wanted and make sure to call scan.Close() to free the scanner on the regionservers.
Hello, may I ask how can I pull x number of rows from a start key? I am trying to achieve paginated scans. I tried to leave the end key empty, but it still always fetches more than the specified limit.
Hmmm, yeah I get that part. But when I did this,
req, err := hrpc.NewScanRangeStr(ctx, tableName, cursor, "", hrpc.NumberOfRows(uint32(limit)+1), hrpc.CloseScanner())
It still returns more rows that limit + 1.
I made it work like this:
// scanLimit reads at most n number of cells from scanner and returns the result.
// It is the caller's responsibility to close the scanner once all reading has been finished.
func scanLimit(scanner hrpc.Scanner, n int) ([]*hrpc.Cell, error) {
var cells []*hrpc.Cell
for len(cells) < n {
result, err := scanner.Next()
if err == io.EOF {
break
} else if err != nil {
return nil, err
}
for _, cell := range result.Cells {
// result.Cells can contain anywhere from 1 to many cells
if len(cells) < n {
cells = append(cells, cell)
}
}
}
return cells, nil
}
Caller:
scan, err := hrpc.NewScanRangeStr(ctx, table, k, "",
hrpc.NumberOfRows(10),
)
if err != nil {
return err
}
scanner := runner.client.Scan(scan)
cells, err := scanLimit(scanner, 10)
if err != nil {
return err
}
scanner.Close()
Note that despite setting hrpc.NumberOfRows(10), scanner.Next() can return anywhere from 1 cell at a time to over a million cells at once, which is quite strange.
Note that despite setting hrpc.NumberOfRows(10), scanner.Next() can return anywhere from 1 cell at a time to over a million cells at once, which is quite strange.
1 Cell = 1 Column. If you ask for 1 row but that row has 10000000 columns, you will get 10000000 cells. You can change that by asking only for a specific columns using the hrpc.Families(...) option