Bug in PRAW fallback in retrieve_submission_comments
Closed this issue · 1 comments
AADeLucia commented
There's a bug in retrieve_submission_comments
. The general except Exception
hid it. Don't do that. The PRAW fallback block didn't turn the list into a DataFrame, which threw an error when it got to df.sort_values
, and the program thought it was a request error instead of a code error so it keeps making requests and failing.
Here's the modified code:
## Fall Back to PRAW
if hasattr(self, "_init_praw") and self._init_praw and len(df) == 0:
temp_dfs = []
for s in submissions_clean:
temp_dfs.append(self._retrieve_submission_comments_praw(submission_id=s))
df = pd.concat(temp_dfs)
## Sort
if len(df) > 0:
df = df.sort_values("created_utc", ascending=True)
df = df.reset_index(drop=True)
return df
except Exception as e:
self.logger.warning(f"{e}")
sleep(backoff)
backoff = 2 ** backoff
kharrigian commented
Closed by 86cde22.