decrypto-org/spider

Infinite loop - detected but exception gets consumed

Closed this issue · 2 comments

Cheerio and Winston are unfortunately not working well together. Logging in cheerios $.each looper should not be done for now, since this leads to an infinite loop. We can probably circumvent the issue if we can override Winston in the errors case. If this is not possible, I have to identify, whether the issue actually lies within Winston or Cheerio and create a pull request for this

Failing example (within async function, without callback or await):

/**
 * @param {html} body - contains the html body of a webpage
 */
async extraction(body){
const $ = cheerio.load(body);
    $("a").each((i, elem) =>{
        logger.info("currently looking at element: ", elem);
    });
}

// call of the function
extraction(body);

I'll check for a small working example of the bug. Since we switched to regex url matching to capture more (also not specially marked in the dom) urls, I'll close this issue for now.