Not all rows returned

[ebsanford]

My project contains the following snippet of code. For the most part it performs exactly as design. However, I have one table that contains in excess of 400,000 rows. No matter what I do, this function will return is maximum of 57,071 rows, even when doing a simple "SELECT * FROM ServerLog". No errors are thrown and all appears normal. If I didn't already know better, I'd think the ServerLog table contained exactly 57,071 rows. This table has only 10 columns, 8 are varchar, 1 is a date, and 1 is an integer. The varchar columns are less than 10 characters. So I can't hardly believe it's a memory exhaustion related issue.

Am I missing something a "fetch more data" operation? Any guidance would be greatly appreciated.

BTW, I think PostgresClientKit is excellent, and it should become a standard for Swift. I've tried all the NIO flavors (Kitura, Promise, and Vapor), and PostgresClientKit is, by far, the best.

`public func Query( _ select: String, parms: [PostgresValueConvertible?] = [] ) -> [Cursor.Element]? {

  guard (self.isConnected), let hndl = self.Handle
     else {
        self.Message_ = "Attempted SELECT query while not connected"
        return nil
        }

  do {
     let stmt = try hndl.prepareStatement(text: select)
     let curs = try stmt.execute(parameterValues: parms, retrieveColumnMetadata: true)

     var ROWS = [Cursor.Element]()

     for row in curs {
        ROWS.append(row)
        }

     curs.close()
     stmt.close()

     self.Message_ = ""
     return ROWS
     }

  catch let error as PostgresError {
     self.Message_ = error.localizedDescription
     return nil
     }

  catch {
     self.Message_ = error.localizedDescription
     return nil
     }

  }`

[pitfield]

Your code snippet looks fine. A few suggestions on troubleshooting this:

What does SELECT count(*) FROM ServerLog; return when executed through this Swift method?

There is a unit test in PostgresClientKit that creates a table, inserts some rows, and selects them. It's located in Tests/PostgresClientKitTests/SQLStatementTest.swift and is named testCRUD(). In my environment, I increased the number of rows from 1,000 to 1,000,000 and ran the test. It returned all 1,000,000 rows:

Test Suite 'Selected tests' started at 2021-08-22 12:07:58.672
Test Suite 'PostgresClientKitTests.xctest' started at 2021-08-22 12:07:58.673
Test Suite 'SQLStatementTest' started at 2021-08-22 12:07:58.674
Test Case '-[PostgresClientKitTests.SQLStatementTest testCRUD]' started.
[2021-08-22T19:13:41.659Z info] INSERT 1000000 rows: elapsed time 314904.33502197266 ms
[2021-08-22T19:14:45.138Z info] SELECT 1000000 rows: elapsed time 63478.48296165466 ms

What happens if you run this test in your environment? (See Setting up a Postgres database for testing.)

If these suggestions don't help you track down the problem, feel free to send me a standalone test case that reproduces the problem (e.g. creating the table, inserting test data, and the SELECT that fails to return all the rows) -- and I will investigate this further in my environment. Thanks!

[ebsanford]

Thanks for suggesting the "SELECT count(*) ... ", because that correctly returned a single, 1-column row with the correct row count. Then I likewise hard-coded a "SELECT * ..." and it did indeed return all rows!

Just chalk it up to me having a senior moment. While I thought "SELECT ... FROM ServerLog" was being sent to my Query function, the calling code was adding a "WHERE (CURRENT_TIMESTAMP-stamp <= '1 week' )" clause on the fly based on an "out of sight out of mind" parameter setting.

My apologies for interrupting your day with this.
Once again, the PostgresClientKit is the best!!

[pitfield]

Thanks! Glad you got it sorted out.