typedb/typedb-driver-python

Client hangs when iterating through QueryManager answers

TheDr1ver opened this issue · 1 comments

TheDr1ver commented

Description

When running through a script that does many READ and WRITE transactions throughout its entirety, I'm finding myself running into an odd issue where sometimes, at seemingly random times, the script can hang for as long as 10 seconds when iterating through the answers provided by the QueryManager.

I've seen this happen on both match operations and insert operations, again, seemingly at random.

Environment

  1. OS (where TypeDB server runs): TypeDB server runs on Ubuntu 20 desktop (VMWare client). Python client runs on Windows 10 host.
  2. TypeDB version (and platform): 2.10.0
  3. TypeDB client-python version: 2.9.0
  4. Python version: 3.9.12

Reproducible Steps

Steps to create the smallest reproducible scenario:

  1. queue up a bunch of transactions to run - either of match or insert types

  2. execute all transactions under the same session until hang occurs (write transactions are committed/closed and a new transaction is opened for subsequent actions in this particular scenario). When last tested this happened about 4 seconds after starting the script, and after a little less than 30 read and write (match & insert) transactions had been executed.

pseduocode

answers = tx.<whatever_action>(<query_string>)

_log.info(f"processing answers")
_log.info(f"answers: {answers}")

for answer in answers: # <------------- This is where it hangs
    _log.info(f"answer: {type(answer)} {answer}")
    if isinstance(answer, ConceptMap):
        _log.info(f"Getting concepts")
        concepts = answer.concepts()
        _log.info(f"Concepts retrieved!")

Expected Output

Iterator doesn't hang when accessed (works as intended for almost 30 iterations before freezing up)

Actual Output

...everything runs fine for the first ~30 or so transactions....

2022-05-27 12:05:41 [INFO] ledhntr > typedb_client.py > (process_query_answers) [1715] >  processing answers
2022-05-27 12:05:41 [INFO] ledhntr > typedb_client.py > (process_query_answers) [1716] >  answers: <generator object _QueryManager.match.<locals>.<genexpr> at 0x0000024CE735F740>
#### HANGS 2 SECONDS ####
2022-05-27 12:05:43 [INFO] ledhntr > typedb_client.py > (process_query_answers) [1719] >  answer: <class 'typedb.concept.answer.concept_map._ConceptMap'> [running_2/_Relation[running:0x847080168000000000000004]]
2022-05-27 12:05:43 [INFO] ledhntr > typedb_client.py > (process_query_answers) [1721] >  Getting concepts
2022-05-27 12:05:43 [INFO] ledhntr > typedb_client.py > (process_query_answers) [1723] >  Concepts retrieved!

...everything runs fine again for a bunch of additional transactions....

2022-05-27 12:05:46 [INFO] ledhntr > typedb_client.py > (process_query_answers) [1715] >  processing answers
2022-05-27 12:05:46 [INFO] ledhntr > typedb_client.py > (process_query_answers) [1716] >  answers: <generator object _QueryManager.match.<locals>.<genexpr> at 0x0000024CE70AD190>
#### HANGS 7 SECONDS ####
2022-05-27 12:05:53 [INFO] ledhntr > typedb_client.py > (process_query_answers) [1719] >  answer: <class 'typedb.concept.answer.concept_map._ConceptMap'> [running_2/_Relation[running:0x847080168000000000000005]]
2022-05-27 12:05:53 [INFO] ledhntr > typedb_client.py > (process_query_answers) [1721] >  Getting concepts
2022-05-27 12:05:53 [INFO] ledhntr > typedb_client.py > (process_query_answers) [1723] >  Concepts retrieved!

...everything runs fine again for a bunch of additional transactions....

2022-05-27 12:05:55 [INFO] ledhntr > typedb_client.py > (process_query_answers) [1715] >  processing answers
2022-05-27 12:05:55 [INFO] ledhntr > typedb_client.py > (process_query_answers) [1716] >  answers: <generator object _QueryManager.insert.<locals>.<genexpr> at 0x0000024CE732CC10>
#### HANGS 2 SECONDS ####
2022-05-27 12:05:57 [INFO] ledhntr > typedb_client.py > (process_query_answers) [1719] >  answer: <class 'typedb.concept.answer.concept_map._ConceptMap'> [http/_Entity[http:0x826e800a8000000000000002]][rel/_Relation[running:0x847080168000000000000005]]
2022-05-27 12:05:57 [INFO] ledhntr > typedb_client.py > (process_query_answers) [1721] >  Getting concepts
2022-05-27 12:05:57 [INFO] ledhntr > typedb_client.py > (process_query_answers) [1723] >  Concepts retrieved!
TheDr1ver commented

I tried recreating this using randomly-generated dummy data and a sample schema, but was unable to get it to hang again with any sort of significance.

Additionally, since posting this issue, I've started batching my inserts and matches separately... which has significantly improved performance, and essentially makes this a non-issue now.

All the same, thanks for taking a look - I'm closing this for now, but will reopen it if I somehow manage to better isolate the problem by accident.

  • 👍1