kharrigian/retriever

Bug in PRAW fallback in retrieve_submission_comments

Closed this issue · 1 comments

There's a bug in retrieve_submission_comments. The general except Exception hid it. Don't do that. The PRAW fallback block didn't turn the list into a DataFrame, which threw an error when it got to df.sort_values, and the program thought it was a request error instead of a code error so it keeps making requests and failing.

Here's the modified code:

                ## Fall Back to PRAW
                if hasattr(self, "_init_praw") and self._init_praw and len(df) == 0:
                    temp_dfs = []
                    for s in submissions_clean:
                        temp_dfs.append(self._retrieve_submission_comments_praw(submission_id=s))
                    df = pd.concat(temp_dfs)
                ## Sort
                if len(df) > 0:
                    df = df.sort_values("created_utc", ascending=True)
                    df = df.reset_index(drop=True)
                return df
            except Exception as e:
                self.logger.warning(f"{e}")
                sleep(backoff)
                backoff = 2 ** backoff

Closed by 86cde22.