resuming old thread

Question

resuming old thread

d4op opened this issue 7 years ago · 7 comments

theres a bug its adding the old urls found before and goes on without problems.
when i check after stopping dcrawl the file, it didnt add anything new.

Answer 1 · 2017-08-18T12:06:27.000Z

Can you see in console output that it is adding new domains?

Answer 2 · 2017-08-19T07:27:29.000Z

Yes it adds the old ones in console. But Dosent saves the new ones Am 18.08.2017 um 14:06 schrieb Kuba Gretzky <notifications@github.com<mailto:notifications@github.com>>: Can you see in console output that it is adding new domains? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#2 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ALLcyhjy0veaqDR0sIfSKdHFMYmguYzEks5sZX5DgaJpZM4O7Y_p>.

Answer 3 · 2017-08-19T07:46:27.000Z

I will need to see some console output to be able to assist.

…

On 19 Aug 2017 09:27, "d4op" ***@***.***> wrote: Yes it adds the old ones in console. But Dosent saves the new ones Am 18.08.2017 um 14:06 schrieb Kuba Gretzky ***@***.***< ***@***.***>>: Can you see in console output that it is adding new domains? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<https://github.com/ kgretzky/dcrawl#2#issuecomment-323336284>, or mute the thread< https://github.com/notifications/unsubscribe-auth/ ALLcyhjy0veaqDR0sIfSKdHFMYmguYzEks5sZX5DgaJpZM4O7Y_p>. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ASXmJ5APpJRGOyEpbry-IOvEgKYhxzlGks5sZo5igaJpZM4O7Y_p> .

Answer 4 · 2017-08-23T14:45:25.000Z

there is no output, it says added xxxx.com
etc
but the new ones arent written to the old file.

Answer 5 · 2017-10-02T18:41:04.000Z

@kgretzky Thank you for building this, saves me bunch of time.

I can confirm this happening. There is no additional details in the console but this is how you can replicate this (have tried both on OS X 10.11.6 and Debian Stretch):

Started crawling, output to file
Output saved as supposed to
Stop crawling
Start crawling again. Shows progress as supposed to.
Nothing added to the file

Answer 6 · 2019-07-14T09:43:27.000Z

Issue fixed , you can use https://github.com/ubogdan/dcrawl until the owner will have some time to merge it into master branch.

Answer 7 · 2019-08-12T22:11:26.000Z

the issue will be fixed by changing the file open line to this

fo, err := os.OpenFile(*output_file, os.O_RDWR|os.O_APPEND|os.O_CREATE, 0664)