Cloudflare and FanFiction.net doing its thing again

Question

Cloudflare and FanFiction.net doing its thing again

chocolatechipcats opened this issue 3 years ago · 102 comments

FanFicFare 4.3.0, Calibre running from source.

Attempting to update from FFNet gives out a Cloudflare 2 error. Several other users have reported this on MobileRead. Cache works as expected. Checked Twitter and there's no indication that FictionPress is working on an update, but I will keep monitoring.

Not much to be done about it (there doesn't seem to be any recent updates to the scraper), but at least here's a report to monitor the issue.

Answer 1 · 2021-06-02T14:48:10.000Z

I don't know if it is the same thing but I get the following error when using CLI:

cloudscraper.exceptions.CloudflareCaptchaProvider: Cloudflare Captcha detected, unfortunately you haven't loaded an anti Captcha provider correctly via the 'captcha' parameter.

During handling of the above exception, another exception occurred:

fanficfare.exceptions.FailedToDownload: cloudscraper reports: "Cloudflare Captcha detected, unfortunately you haven't loaded an anti Captcha provider correctly via the 'captcha' parameter."

It looks like there is a change in cloudscraper's use and it needs a new option to be set.
Or maybe it's an old option and now it raises an error... I don't know

Answer 2 · 2021-06-02T14:56:09.000Z

It looks like ffnet has increased their Cloudflare blocking level again.

As far as I know, there isn't anything more we can do in FFF than there was the last time this happened.

I suggest reviewing these resources:

FAQ:
https://github.com/JimmXinu/FanFicFare/wiki/FAQs#why-am-i-having-errors-downloading-from-fanfictionnet--why-am-i-getting-cloudflare-errors-downloading-from-fanfictionnet

Browser Cache Feature:
https://github.com/JimmXinu/FanFicFare/wiki/BrowserCacheFeature

Browser Proxy (third party):
https://github.com/nsapa/fanfictionnet_ff_proxy

Answer 3 · 2021-06-02T15:01:29.000Z

@JimmXinu We can use puppeteer though? I just tested my https://github.com/NightMachinary/.shells/blob/master/scripts/javascript/curlfull.js , and it worked fine. Using the browser cache is not a very good solution (I use fanficfare on a server).

Answer 4 · 2021-06-02T15:03:19.000Z

@JimmXinu A general workaround is to let the user supply a downloader, like youtube-dl's --external-downloader, if you don't like bundling puppeteer.

Answer 5 · 2021-06-02T15:27:02.000Z

First off, I'm not doing anything about this in a hurry. More than once, we've seen ffnet's CF level go up for a while and then go back down after a while--think days, not hours.

Second, Cloudflare is a service specifically for detecting and blocking automated access. I assume (and past evidence agrees) that they can already block well known automated browser tools. If I recall correctly, earlier in the year, we experimented with headless proxy browsers and all of them were blocked after a few requests.

That's why our current generation of work arounds are based on using a live browser.

Answer 6 · 2021-06-02T15:27:10.000Z

I was already using Cache... but since I reread the wiki I tried and I downloaded the entire story using WebToEpub and it downloaded the whole story.
But when I immediately afterwards used FFF is didn't work

Answer 7 · 2021-06-02T15:35:43.000Z

I was already using Cache... but since I reread the wiki I tried and I downloaded the entire story using WebToEpub and it downloaded the whole story.
But when I immediately afterwards used FFF is didn't work

Try use_browser_cache_only:true to prevent it falling back to the site. I'm assuming it's some user error on my end, but for whatever reason I have trouble getting it to use the cache if I only set use_browser_cache:true.

Answer 8 · 2021-06-02T23:21:57.000Z

Is it worth - if ffn keep this up - investigating how their app bypasses these checks?

Answer 9 · 2021-06-03T12:10:56.000Z

Try use_browser_cache_only:true to prevent it falling back to the site. I'm assuming it's some user error on my end, but for whatever reason I have trouble getting it to use the cache if I only set use_browser_cache:true.

This is what I've tried (using FanFicFare 4.3.0 and Firefox 88):

Download https://www.fanfiction.net/s/13747780/1/ with WebToEpub
Add browser_cache_path:/home/matej/.cache/mozilla/firefox/uxi2c7cz.default/cache2 to [defaults] in personal.ini
Run:

tmp@stitny$ fanficfare -o use_browser_cache_only=true https://www.fanfiction.net/s/13747780/1/
cloudscraper.exceptions.CloudflareChallengeError: Detected a Cloudflare version 2 Captcha challenge, This feature is not available in the opensource (free) version.

During handling of the above exception, another exception occurred:

fanficfare.exceptions.FailedToDownload: cloudscraper reports: "Detected a Cloudflare version 2 Captcha challenge, This feature is not available...."

Something’s wrong. Why even with use_browser_cache_only=true I get Internet connection?

Answer 10 · 2021-06-03T16:29:07.000Z

Do you also have use_browser_cache:true under [www.fanfiction.net]? use_browser_cache_only:true doesn't do anything without use_browser_cache:true

Answer 11 · 2021-06-03T23:39:54.000Z

Do you also have use_browser_cache:true under [www.fanfiction.net]? use_browser_cache_only:true doesn't do anything without use_browser_cache:true

Which is a bit weird, isn't it? Anyway, this works.

fanficfare -o use_browser_cache=true -o use_browser_cache_only=true -o use_cloudscraper=false \
    https://www.fanfiction.net/s/13747780/1/

Answer 12 · 2021-06-07T10:17:20.000Z

I'm assuming it's some user error on my end, but for whatever reason I have trouble getting it to use the cache if I only set use_browser_cache:true.

Just for posterity, check_next_chapter:true seemed to be the cause of this.

Answer 13 · 2021-06-09T14:54:58.000Z

I have also recieved the following errors in the last 24 hours.

Status
Title
Author
Comment
URL
Error
Unknown
Unknown
cloudscraper reports: "Detected a Cloudflare version 2 Captcha challenge, This feature is not available...."
https://www.fanfiction.net/s/5598642/1/
Error
Unknown
Unknown
cloudscraper reports: "Detected a Cloudflare version 2 Captcha challenge, This feature is not available...."
https://www.fanfiction.net/s/5244813/1/
Error
Unknown
Unknown
cloudscraper reports: "Detected a Cloudflare version 2 Captcha challenge, This feature is not available...."
https://www.fanfiction.net/s/5012016/1/Partners

Answer 14 · 2021-06-09T15:11:34.000Z

I have also recieved the following errors in the last 24 hours.

Yes, Cloudflare is acting again, and there is nothing we can do about it. Workaround with WebToEpub is the only thing I know to work.

Answer 15 · 2021-06-14T20:19:53.000Z

Is it worth - if ffn keep this up - investigating how their app bypasses these checks?

The author of at least one other 3rd party reader app I know of has already looked at it and states that the code at issue is a totally obfuscated mess that he has no intention of trying to dig into. If you want to try, feel free. But, keep in mind, even if you manage to figure something out, as soon as more than a few people try to use it they'll just update the app and invalidate everything you did.

The owner of FFN is focused on getting his ad revenue, and going to war with a company like Cloudflare is a zero sum fight you aren't going to win in the end. The only thing you can try to do is work around them like Jimm is with the FFF browser cache feature. I just hope FFN doesn't try to implement some form of cache busting technique.

Answer 16 · 2021-06-14T20:25:21.000Z

Is it worth - if ffn keep this up - investigating how their app bypasses these checks?

The author of at least one other 3rd party reader app I know of has already looked at it and states that the code at issue is a totally obfuscated mess that he has no intention of trying to dig into. If you want to try, feel free. But, keep in mind, even if you manage to figure something out, as soon as more than a few people try to use it they'll just update the app and invalidate everything you did.

The owner of FFN is focused on getting his ad revenue, and going to war with a company like Cloudflare is a zero sum fight you aren't going to win in the end. The only thing you can try to do is work around them like Jimm is with the FFF browser cache feature. I just hope FFN doesn't try to implement some form of cache busting technique.

The official FFN app doesn't have ads, so whatever reason they have for this can't be ad revenue.

Answer 17 · 2021-06-14T20:46:53.000Z

Rather than believing that ffnet has deliberately set out to ruin our collective day, I suspect that they are using Cloudflare for protection from aggressive, automated systems attacking their site.

Unfortunately, individual users with automated download software (IE, us) are extremely similar in appearance from their POV.

I'll add what I said over on MR recently:

I have lost basically all interest in fighting with ffnet about this. Browser cache works well enough for me for the few active authors I still read on ffnet.

Unless someone comes up with a radical new idea, I'm not thinking about it anymore right now.

Many thanks to the members here are willing and able to help each other.

Answer 18 · 2021-06-14T21:10:58.000Z

I have lost basically all interest in fighting with ffnet about this. Browser cache works well enough for me for the few active authors I still read on ffnet.

I basically agree with what you said, with one exception (and no I have no idea how to do it): at least checking that the work has new update. I have still 776 FFN stories (yes, trying to move to AO3 as much as possible, but this is the best I have managed so far) and I have no clue how to check that I am not missing on new chapters. Update itself is done easily with browser cache.

Answer 19 · 2021-06-14T21:13:50.000Z

If you have an ffnet account, you can go to Account Page > Alerts > Story Alerts and compare the dates of followed stories to your local copies. Setting up Calibre with a last-updated column will help too.

Answer 20 · 2021-06-14T22:22:19.000Z

I have a discord bot which has been able to access FFN most of the time recently. Is fanficfare using the latest version of cloudscraper?

…

On Jun 14, 2021, at 5:14 PM, chocolatechipcats ***@***.***> wrote: If you have an ffnet account, you can go to Account Page > Alerts > Story Alerts and compare the dates of followed stories to your local copies. Setting up Calibre with a last-updated column will help too. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

Answer 21 · 2021-06-14T22:25:25.000Z

Cloudscraper's last update was on April 9, and I believe it's already been incorporated into FFF. Also, the error specifically mentions the version 2 CAPTCHA, which the Cloudscraper developer has behind a paywall (this was discussed in the last thread).

Answer 22 · 2021-06-20T21:07:47.000Z

Has anyone even gotten the paywall-d version to work? At this point I'd be willing to shell out a few bucks, but not if it doesn't work with Fanficfare.

Answer 23 · 2021-06-20T21:09:39.000Z

Has anyone even gotten the paywall-d version to work? At this point I'd be willing to shell out a few bucks, but not if it doesn't work with Fanficfare.

I haven’t found a paid version. Is there a one?

Answer 24 · 2021-06-20T21:12:31.000Z

It was discussed in the last thread: #622 (comment)

Answer 25 · 2021-07-01T18:58:07.000Z

The Google Chrome extension: WebToEpub still can download from FF.net.

https://github.com/dteviot/WebToEpub/tree/ExperimentalTabMode

Answer 26 · 2021-07-01T19:00:25.000Z

The Google Chrome extension: WebToEpub still can download from FF.net.

Works in Firefox as well. But the problem is that some more sophisticated things FFF can do, this extension cannot (namely, --update-epub).

Answer 27 · 2021-07-01T20:38:43.000Z

@mcepl Better than nothing, especially if one can't use that Windows program to download stuff from FFN, but has a Firefox, Chrome or Chrome-based browser.

Answer 28 · 2021-07-22T03:28:23.000Z

I run FanFicFare on a headless Raspberry Pi and submit URLs from my phone, so using any cache isn't viable unfortunately. For now, I've just stopped it checking for updates and I'm seriously considering whether it's worth decompiling the app and seeing if the same method it uses is viable (or if using the app cache is viable)...

Answer 29 · 2021-07-31T20:07:38.000Z

does any one know if there is a fanfiction.net app that let us read like a book instead of scrooling if so please let me know so ican read my books intill this problem is solved beacse the scrolling on fanfiction.net makes me sick

Answer 30 · 2021-07-31T20:09:06.000Z

FanFiction.net has an official app. From what I understand all other ones are blocked.

Answer 31 · 2021-07-31T20:12:38.000Z

[s there a there away around it

Answer 32 · 2021-07-31T20:12:42.000Z

does any one know if there is a fanfiction.net app that let us read like a book instead of scrooling if so please let me know so ican read my books intill this problem is solved beacse the scrolling on fanfiction.net makes me sick

In the official app you can "turn page" with the volume keys, it's an option in the settings.

Not the best place to ask this though..

Answer 33 · 2021-07-31T20:13:51.000Z

where would i ask i am new here

Answer 34 · 2021-07-31T20:40:18.000Z

Google for fanfiction.net app. This is off-topic for both this project and this issue.

Answer 35 · 2021-08-01T01:34:25.000Z

It looks like ffnet has increased their Cloudflare blocking level again.

As far as I know, there isn't anything more we can do in FFF than there was the last time this happened.

I suggest reviewing these resources:

FAQ:
https://github.com/JimmXinu/FanFicFare/wiki/FAQs#why-am-i-having-errors-downloading-from-fanfictionnet--why-am-i-getting-cloudflare-errors-downloading-from-fanfictionnet

Browser Cache Feature:
https://github.com/JimmXinu/FanFicFare/wiki/BrowserCacheFeature

Browser Proxy (third party):
https://github.com/nsapa/fanfictionnet_ff_proxy

Hey @JimmXinu, I've read your BrowserCacheFeature page several times and nowhere in that does it say where this personal.ini file I'm supposed to edit is located. I'm getting more and more frustrated the more I try to implement this solution. Can you please update the wiki page or at least inform me where this file is located?

Thanks in advance!

Answer 36 · 2021-08-01T01:40:53.000Z

There's a separate wiki page about INI and the first thing in it is INI File Location.

But I will add a link from the BrowserCacheFeature page.

Answer 37 · 2021-08-01T01:42:34.000Z

There's a separate wiki page about INI and the first thing in it is INI File Location.

But I will add a link from the BrowserCacheFeature page.

Thank you sir! Much appreciated!

Answer 38 · 2021-08-01T02:29:18.000Z

There's a separate wiki page about INI and the first thing in it is INI File Location.
But I will add a link from the BrowserCacheFeature page.

Thank you sir! Much appreciated!

Click the down arrow by the FFF button in Calibre, and select the option to configure FFF. On the configuration page that pops up, click the Personal.ini tab. There is an "Edit Personal.ini" button there, and once it is open for editing there is a save button at the bottom of the edit window.

One piece of advice I learned the hard way - be very careful using when using copy/paste into Personal.ini. Somehow (as near as I can tell, anyway) I managed to paste some type of invisible embedded character in there. It caused me all sorts of random misery with FFF for weeks that I just couldn't fix before ownedbycats on the MobileRead site pointed out to me how to reset Personal.ini to defaults.

Answer 39 · 2021-09-12T00:02:46.000Z

has any one figured how to get around this i do not care about downloading but i want to update my stories any help would be welcome thank you and god bless

Answer 40 · 2021-09-12T00:17:07.000Z

The only new information is that the Browser Cache work around can also use the cache created by FanFictionDownloader.

Answer 41 · 2021-09-12T02:54:17.000Z

That's terrific news, since it should allow more batching - I think FFD will download whatever I feed it if I give it a list in a file, but can just delete those and re-feed the list into FFF. Which version of FFF do I need to enable the FFD cache? I'm currently using 4.3.0.

So what I would do is have use FFF to create the list (read it from a web page or drag/drop the desired items from the page onto its download list), save the resulting list to a .txt file, which I then feed to FFD; let it do its thing to populate the cache, then run the FFF download-from-url action, and finally delete the redundant FFD files. Make sense?

Answer 42 · 2021-09-12T04:36:18.000Z

would any one have direct instructions for a total noob with pictures if possible i am autistic

Answer 43 · 2021-09-12T05:56:10.000Z

@CodeyAce95, are you looking for instructions regarding using FFD with FFF, along the lines I described above, just specific parts of the process, or something else? For example, do you need help with finding and setting up FFD (looks like you already have and use FFF), or just the bits I described that suggest what might work by using them together? There is a download link to FFD in JimmXinu's post above if you don't already have it.

I haven't yet tried what I described, but once I do (and confirm it works), I could probably put together a Word document with screen-grabs, if that would be helpful - and if this site will let me attach the doc to a message here. It seems like it allows attachments, I just don't what restrictions, if any, there might be to the types or sizes of the attachments. It may take me a few days to get to it, since I'm behind on some stuff at the moment :)

Answer 44 · 2021-09-12T11:26:32.000Z

Alternatively, you can download ffnet fics using a different tool, like ff2ebook or fichub, personally I'm currently using FicHub-cli and it works fine.

This tools get around cloudfare using FlareSolverr, but I don't know if this would work with FFF. In the past ff2ebook just used cookies and it worked too, so that's something that someone could try.

Answer 45 · 2021-09-12T13:30:49.000Z

@CodeyAce95, are you looking for instructions regarding using FFD with FFF, along the lines I described above, just specific parts of the process, or something else? For example, do you need help with finding and setting up FFD (looks like you already have and use FFF), or just the bits I described that suggest what might work by using them together? There is a download link to FFD in JimmXinu's post above if you don't already have it.

I haven't yet tried what I described, but once I do (and confirm it works), I could probably put together a Word document with screen-grabs, if that would be helpful - and if this site will let me attach the doc to a message here. It seems like it allows attachments, I just don't what restrictions, if any, there might be to the types or sizes of the attachments. It may take me a few days to get to it, since I'm behind on some stuff at the moment :)

yes i think that might be just what i need thank you and god bless

Answer 46 · 2021-09-12T15:37:41.000Z

We've discussed FlareSolverr before. It starts a proxy process that starts a puppeteer headless browser process. nsapa's Browser Proxy is effectively similar and already working.

However, since FlareSolverr offers binary downloads (for win and linux), I may attempt using it. But I would point out the following from FlareSolverr's home page about captcha solvers:

⚠️ At this time none of the captcha solvers work. You can check the status in the open issues. Any help is welcome.

The fichub projects listed are frontends to an AJAX API server. I don't see a public repository for the actual download code. By running all downloads through their own server, all issues with running headless browsers are at least concentrated in one place.

FFF used to have a web version which operated under somewhat similar basic architecture, but it's been retired:

I continued to support the web service in recent years as a legacy for the users who can't run the CLI or Calibre versions. But I'm not interested in spending my money on it, or dealing with the accounting and possible tax implications of collecting donations to run it.

@LoisGNS - If you are willing to put together words and pictures for a user guide on using FFDL and FFF together, I would be willing to convert it to markdown and add it to the project as a wiki page--properly credited, of course.

Answer 47 · 2021-09-13T02:23:19.000Z

I started such a document, doing screen-grabs as I went. But it didn't work. I am using ver 4.5.0 of FFF and 0.9.2 of FFD. Do I need a different version of one or both?

Answer 48 · 2021-09-13T02:28:02.000Z

You probably need the latest FanFictionDownloader, v0.9.4.

Answer 49 · 2021-09-13T03:10:32.000Z

I can confirm it works with 0.9.4.

Since I made the same mistake, check that you're using the right cache folder. There's \cache\ but you actually want \cache\QtWebEngine\Default\Cache\. Mine has several data_# files and an index.

Answer 50 · 2021-09-13T03:51:47.000Z

I just tried after updating FFD to 0.9.4, and it still isn't working for me.

@chocolatechipcats, are you saying I need to change the browser_cache_path in personal.ini from the one I use when just clicking through in FireFox? That one is C:\Users[myusername]\AppData\Local\Mozilla\Firefox\Profiles[myfolder].default\cache2, which has two subfolders: "doomed" and "entries" and a file called "Index" with no extension. There's lots of stuff under Entries. This works with the browser-cache method, and is the one that was described in the instructions for it.

I don't find any folder like the one you describe. I looked in c:\users (etc.), as well as the FFD and Calibre folders. I also asked Windows to search for it, and it didn't find it either.

There is also a Chrome browser cache shown as an option in the instructions, browser_cache_path:C:\Users\YourUser\AppData\Local\Google\Chrome\User Data\Default\Cache or trying another profile name in place of "default". I don't have a Chrome profile name other than "default," so tried that, which didn't work either.

I'm stumped now.

Answer 51 · 2021-09-13T04:12:19.000Z

Yes, you must change browser_cache_path.

FFDL uses it's own browser instance, not your usual browser. As I documented, for me the FFDL cache dir was C:\Users\YourUser\AppData\Local\fanficdownloader\cache\QtWebEngine\Default\Cache

Answer 52 · 2021-09-13T06:48:01.000Z

Ok, that worked. I hadn't seen the additional documentation. I attached illustrated documentation of the process. Feel free to process as appropriate and add to the wiki.

Batch downloading with FFF and FFD.docx

Answer 53 · 2021-09-13T16:55:43.000Z

Thanks! I've created a new wiki page: FanFictionDownloader-FanFicFare-Batch

Answer 54 · 2021-09-13T17:52:05.000Z

i can;t find cache in my personal.ini am i do somthing wrong

Answer 55 · 2021-09-13T19:14:11.000Z

i can;t find cache in my personal.ini am i do somthing wrong

Something like this line in [defaults] section ~/.fanficfare/personal.ini should help:

browser_cache_path:/home/matej/.cache/mozilla/firefox/uxi2c7cz.default/cache2

Answer 56 · 2021-09-16T18:56:11.000Z

I've posted test versions with an experiment feature to support using FlareSolverr.

If you are not able to obtain, install and run FlareSolverr on your own from the FlareSolverr documentation, I politely suggest that you are not a good candidate for evaluating this experimental feature at this time.

After you have FlareSolverr running, to enable it in FFF use:

[www.fanfiction.net]
use_flaresolverr_proxy:true
# optional settings with default values shown:
#flaresolverr_proxy_address:localhost
#flaresolverr_proxy_port:8191

Obviously, this is mostly for ffnet, but it does appear to work for many other sites. Notably, it does not work for SpaceBattles, SufficientVelocity, QQ and I assume the other XenForoForum sites. FlareSolverr fails with a 'Page Crashed' error.

I've tested the FFF code in Win10, Mac and Ubuntu 21.04 using Calibre 5.x official releases and Win10 CLI. I've only run FlareSolverr on Windows and used the flaresolverr_proxy_address setting to point the different Calibre to the same FlareSolverr proxy server.

Answer 57 · 2021-09-17T18:17:00.000Z

Sadly if there is a way to make this work on android, I don't know it. It does work fine on a desktop, even if I can't find a reasonable excuse to keep the desktop on just so my phone can update all its stories with a single command.

Answer 58 · 2021-09-22T17:13:20.000Z

FYI, I believe the flaresolverr proxy solution is solid enough to try for anyone who's comfortable downloading and running flaresolverr.

Be warned, however, that I don't intend to offer support for flaresolverr itself.

Answer 59 · 2021-09-22T17:24:01.000Z

Is there some comparison between advantages and disadvantages of flaresolverr and FanFictionDownloader? Are they essentially the same or is there some difference?

Answer 60 · 2021-09-22T17:24:59.000Z

Is there some comparison between advantages and disadvantages of flaresolverr and FanFictionDownloader? Are they essentially the same or is there some difference?

Something that came up on the MR thread is that FFDL's cache doesn't get the cover image, if that's important to you.

Answer 61 · 2021-09-22T17:27:30.000Z

Something that came up on the MR thread is that FFDL's cache doesn't get the cover image, if that's important to you.

OK, but if I use FFDL via use_nsapa_proxy:true then there is no difference?

Answer 62 · 2021-09-22T17:45:46.000Z

Is there some comparison between advantages and disadvantages of flaresolverr and FanFictionDownloader? Are they essentially the same or is there some difference?

I don't know how FFDL's solution work, but with FS you just set it up once and forget, after that FFF will just download any fics without you having to do anything, as if this whole issue didn't exist.

I've been downloading a couple dozen ffnet fics everyday since I setup FS and I've yet to have any issue with cloudfare.

Answer 63 · 2021-09-22T17:52:01.000Z

use_nsapa_proxy:true and use_flaresolverr_proxy:true are roughly equivalent. use_browser_cache:true works differently.

In the first case, you need to have downloaded and started fanfictionnet_ff_proxy and in the second case you need to have downloaded and started flaresolverr. For both, FFF talks to a proxy server which is running a full fledged (albeit headless) browser instance to do the actual page requests and deal with Cloudflare.

The browser cache feature is different, it looks for pages that have already been downloaded in a supported browser cache.

Using FFDL with use_browser_cache:true is equivalent to using a normal browser and manually viewing the pages (or using WebToEpub).

Answer 64 · 2021-09-22T17:58:45.000Z

I tested Flaresolverr and it worked perfectly. :) Does Calibre allow for launching of external applications? If so, it would be nice to input the executable path and have FFF automatically launch that.

Answer 65 · 2021-09-22T18:14:56.000Z

Does Calibre allow for launching of external applications? If so, it would be nice to input the executable path and have FFF automatically launch that.

You see that right there? That's why I dragged my feet on supporting this type of proxy.

I have no interest in trying to figure out how to launch a proxy from inside Calibre. And even less interest in supporting it across all the different py2/py3/mac/linux/win combinations. And the inevitable firewall/antivirus/antimalware issues.

Plus there's the fact that at any moment ffnet could ratchet up to requiring captchas, negating all progress gained.

Answer 66 · 2021-09-23T09:29:35.000Z

Where can I get 4.5.10 of the plugin? After various attempts at testing, I conclude that's what I probably need, but all I can find is some files on the Code page, which don't seem to be the right collection of files for the plugin. The more ususal sources only go up to 4.5.0,

Answer 67 · 2021-09-23T09:44:16.000Z

Go here:
https://www.mobileread.com/forums/showthread.php?t=259221

The second post has the latest test version (anything that doesn't end in .0)

Answer 68 · 2021-09-23T10:03:30.000Z

Aha! I was on that page, but didn't scroll down to the second post - doh! Got it set up and it's working as described.

Answer 69 · 2021-09-23T15:32:37.000Z

Sorry, I assume everybody knows about the 'usual places'. :-) I've added a section to the project home page.

Answer 70 · 2021-09-24T01:34:31.000Z

Funny thing is, I found the MobileRead page and just didn't scroll down far enough to find the link to the test version. Having as many places as possible to find the 'usual places' is definitely helpful.

Answer 71 · 2021-10-14T05:05:37.000Z

The fichub projects listed are frontends to an AJAX API server. I don't see a public repository for the actual download code. By running all downloads through their own server, all issues with running headless browsers are at least concentrated in one place.

Looks like fichub is a front end for hermes, which is a convoluted mess with nested hard-coded if statements for specific fandoms on different sites. The actual scraping on it just... goes through the normal requests library. As best I can tell, here's the code that does all the requests for every single site they manage:

r = requests.get(
        apiUrl,
	self.headers,
	params,
	self.extraData,
	self.auth,
	self.timeout
)

And yet, it's working? I just checked with a work on fanfiction.net that was published a week ago or so.

It looks like the request parameters are filled by a bunch of configuration variables. They do mention that these are required to be filled for FFN to work. So they somehow have a working FFN scraper via setting a baseUrl, apiKey, uname, and/or upass and just using requests.

I'm not sure what's going on here. Did they somehow get access to the API beta FFN mentioned on twitter in January and never since?

Answer 72 · 2021-10-14T07:05:37.000Z

apiUrl

What is it?

Answer 73 · 2021-10-14T16:32:23.000Z

Looking at the code, docs and other repos by FanFicDev, hermes is a surprisingly inclusive program for downloading, storing, managing and reading web fics (defined as web published fanfic and original fiction). It requires a PostGreSQL DB available to store everything in.

For downloading from ffnet, hermes depends on having a skitter_api instance available, which is itself a wrapper for... flaresolverr.

skitter_api is a small wrapper around a set of FlareSolverr instances for load
balancing and redundancy purposes. Essentially a series of docker containers
comprised of gluetun providing a VPN connection and FlareSolverr providing CF
workarounds are kept alive and proxied to.

So I don't think there's any new magic here.

Answer 74 · 2021-10-14T22:00:40.000Z

Looking at the code, docs and other repos by FanFicDev, hermes is a surprisingly inclusive program for downloading, storing, managing and reading web fics (defined as web published fanfic and original fiction). It requires a PostGreSQL DB available to store everything in.

For downloading from ffnet, hermes depends on having a skitter_api instance available, which is itself a wrapper for... flaresolverr.

skitter_api is a small wrapper around a set of FlareSolverr instances for load
balancing and redundancy purposes. Essentially a series of docker containers
comprised of gluetun providing a VPN connection and FlareSolverr providing CF
workarounds are kept alive and proxied to.

So I don't think there's any new magic here.

Ah, not sure how I missed that skitter had a separate repo. I was looking at the skipper_client file here which seemed to be making the requests directly. I suppose the configuration variables are designed to point at the skitter_api repo not the sites themselves 🤦

Answer 75 · 2021-10-15T08:14:16.000Z

Hermes isn't worth the effort it takes to install it, tbh (it's a pain to install and configure that postgresql stuff it needs).

Answer 76 · 2021-10-17T19:31:42.000Z

Hermes isn't worth the effort it takes to install it, tbh (it's a pain to install and configure that postgresql stuff it needs).

I don't know, it doesn't look so bad. Wouldn't be a difficult Dockerfile + Docker Compose setup. Would be nice if it had one provided though

Answer 77 · 2021-10-22T21:55:20.000Z

FYI, Flaresolverr was bumped to v2.0.0. That new version work flawlessly (successfully bypassed all Cloudflare protection testing with cURL) but as of now, FFF is unable to communicate with it. Seems extremely promising though

Answer 78 · 2021-10-22T21:58:01.000Z

Thanks for the notification.

Answer 79 · 2021-10-22T22:37:30.000Z

FYI, at this time, only Flaresolverr v1.2.9 is known to work with FFF.

Flaresolverr v2.0.0 removes the download option that FFF uses to do its own decoding and allow for cover image downloads.

Answer 80 · 2021-10-23T07:55:54.000Z

FYI, at this time, only Flaresolverr v1.2.9 is known to work with FFF.

??? https://github.com/nsapa/fanfictionnet_ff_proxy.git seems to work for me just fine. ???

Answer 81 · 2021-10-23T08:01:00.000Z

FYI, at this time, only Flaresolverr v1.2.9 is known to work with FFF.

??? https://github.com/nsapa/fanfictionnet_ff_proxy.git seems to work for me just fine. ???

I'm pretty sure that JimmXinu was referring to Flaresolver versions, not to other solutions.

Answer 82 · 2021-10-23T08:29:19.000Z

I'm pretty sure that JimmXinu was referring to Flaresolver versions, not to other solutions.

I see.

Answer 83 · 2021-10-23T13:31:22.000Z

That's correct; I should have been more specific:

At this time, v1.2.9 is the only version of Flaresolverr known to work with FFF.

To the best of my knowledge, nsapa's fanfictionnet_ff_proxy still works fine.

Answer 84 · 2021-10-26T19:43:34.000Z

Note that FanFIction.net and seems to have put the site back into under-attack mode. Flaresolverr 1.2.9 isn't working for me at the moment - lots of timeout errors. FFDN still does, so I can use its cache. If FFDN breaks, there's still the browser.

EDIT: If you saw my initial comments about server migration, disregard - the Twitter embed on their front page lists a tweet from January as "34 minutes ago"??

Answer 85 · 2021-10-26T22:59:39.000Z

I think ultimately the problem is that nothing short of a full deep-archive of the entire FFN site (which they'd probably view as a DDoS) and convincing all authors to migrate all new/ongoing work to AO3 is going to cut the FFN admins out of the hot path for bugs in FFF. Since that is unlikely to happen, this cat and mouse game will continue to get more absurd.

Answer 86 · 2021-10-27T08:50:29.000Z

Here is the latest set of errors I received:

Status
Title
Author
Comment
URL
Error
Unknown
Unknown
HTTP Error in FFF '500 Server Error: Internal Server Error for url: http://localhost:8191/v1'(500)
https://www.fanfiction.net/s/13973217/1/Deception
Error
Unknown
Unknown
HTTP Error in FFF '

<title>Just a moment...</title> <style type="text/css"> html, body {width: 100%; height: 100%; margin: 0; padding: 0;} body {background-color: #ffffff; color: #000000; font-family:-apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen, Ubuntu, "Helvetica Neue",Arial, sans-serif; font-size: 16px; line-height: 1.7em;-webkit-font-smoothing: antialiased;} h1 { text-align: center; font-weight:700; margin: 16px 0; font-size: 32px; color:#000000; line-height: 1.25;} p {font-size: 20px; font-weight: 400; margin: 8px 0;} p, .attribution, {text-align: center;} #spinner {margin: 0 auto 30px auto; display: block;} .attribution {margin-top: 32px;} @Keyframes fader { 0% {opacity: 0.2;} 50% {opacity: 1.0;} 100% {opacity: 0.2;} } @-webkit-keyframes fader { 0% {opacity: 0.2;} 50% {opacity: 1.0;} 100% {opacity: 0.2;} } #cf-bubbles > .bubbles { animation: fader 1.6s infinite;} #cf-bubbles > .bubbles:nth-child(2) { animation-delay: .2s;} #cf-bubbles > .bubbles:nth-child(3) { animation-delay: .4s;} .bubbles { background-color: #f58220; width:20px; height: 20px; margin:2px; border-radius:100%; display:inline-block; } a { color: #2c7cb0; text-decoration: none; -moz-transition: color 0.15s ease; -o-transition: color 0.15s ease; -webkit-transition: color 0.15s ease; transition: color 0.15s ease; } a:hover{color: #f4a15d} .attribution{font-size: 16px; line-height: 1.5;} .ray_id{display: block; margin-top: 8px;} #cf-wrapper #challenge-form { padding-top:25px; padding-bottom:25px; } #cf-hcaptcha-container { text-align:center;} #cf-hcaptcha-container iframe { display: inline-block;} </style>

  <meta http-equiv="refresh" content="35">

Please turn JavaScript on and reload the page.

<div id="cf-bubbles">
  <div class="bubbles"></div>
  <div class="bubbles"></div>
  <div class="bubbles"></div>
</div>
<h1><span data-translate="checking_browser">Checking your browser before accessing</span> fanfiction.net.</h1>

<div id="no-cookie-warning" class="cookie-warning" data-translate="turn_on_cookies" style="display:none">
  <p data-translate="turn_on_cookies" style="color:#bd2426;">Please enable Cookies and reload the page.</p>
</div>
<p data-translate="process_is_automatic">This process is automatic. Your browser will redirect to your requested content shortly.</p>
<p data-translate="allow_5_secs" id="cf-spinner-allow-5-secs" >Please allow up to 5 seconds&hellip;</p>
<p data-translate="redirecting" id="cf-spinner-redirecting" style="display:none">Redirecting&hellip;</p>

<script type="text/javascript">
  //<![CDATA[
  (function(){
      var a = document.getElementById('cf-content');
      a.style.display = 'block';
      var isIE = /(MSIE|Trident\/|Edge\/)/i.test(window.navigator.userAgent);
      var trkjs = isIE ? new Image() : document.createElement('img');
      trkjs.setAttribute("src", "/cdn-cgi/images/trace/jschal/js/transparent.gif?ray=6a4a4f5ace9d32dc");
      trkjs.id = "trk_jschal_js";
      trkjs.setAttribute("alt", "");
      document.body.appendChild(trkjs);
      var cpo=document.createElement('script');
      cpo.type='text/javascript';
      cpo.src="/cdn-cgi/challenge-platform/h/g/orchestrate/jsch/v1?ray=6a4a4f5ace9d32dc";
      document.getElementsByTagName('head')[0].appendChild(cpo);
    }());
  //]]>
</script>

      <div class="attribution">
        DDoS protection by <a rel="noopener noreferrer" href="https://www.cloudflare.com/5xx-error-landing/" target="_blank">Cloudflare</a>
        <br />
        <span class="ray_id">Ray ID: <code>6a4a4f5ace9d32dc</code></span>
      </div>
  </td>
 
</tr>

'(503) URL:'https://www.fanfiction.net/s/13905221/1/Interior-of-a-Family' https://www.fanfiction.net/s/13905221/1/Interior-of-a-Family Error Unknown Unknown HTTP Error in FFF '500 Server Error: Internal Server Error for url: http://localhost:8191/v1'(500) https://www.fanfiction.net/s/13974185/1/Elizabeth-is-Missing Error Unknown Unknown HTTP Error in FFF '500 Server Error: Internal Server Error for url: http://localhost:8191/v1'(500) https://www.fanfiction.net/s/13965687/1/A-Test-of-Fire-A-Pride-and-Prejudice-Variation-Novella Error Unknown Unknown HTTP Error in FFF ' <title>Just a moment...</title> <style type="text/css"> html, body {width: 100%; height: 100%; margin: 0; padding: 0;} body {background-color: #ffffff; color: #000000; font-family:-apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen, Ubuntu, "Helvetica Neue",Arial, sans-serif; font-size: 16px; line-height: 1.7em;-webkit-font-smoothing: antialiased;} h1 { text-align: center; font-weight:700; margin: 16px 0; font-size: 32px; color:#000000; line-height: 1.25;} p {font-size: 20px; font-weight: 400; margin: 8px 0;} p, .attribution, {text-align: center;} #spinner {margin: 0 auto 30px auto; display: block;} .attribution {margin-top: 32px;} @Keyframes fader { 0% {opacity: 0.2;} 50% {opacity: 1.0;} 100% {opacity: 0.2;} } @-webkit-keyframes fader { 0% {opacity: 0.2;} 50% {opacity: 1.0;} 100% {opacity: 0.2;} } #cf-bubbles > .bubbles { animation: fader 1.6s infinite;} #cf-bubbles > .bubbles:nth-child(2) { animation-delay: .2s;} #cf-bubbles > .bubbles:nth-child(3) { animation-delay: .4s;} .bubbles { background-color: #f58220; width:20px; height: 20px; margin:2px; border-radius:100%; display:inline-block; } a { color: #2c7cb0; text-decoration: none; -moz-transition: color 0.15s ease; -o-transition: color 0.15s ease; -webkit-transition: color 0.15s ease; transition: color 0.15s ease; } a:hover{color: #f4a15d} .attribution{font-size: 16px; line-height: 1.5;} .ray_id{display: block; margin-top: 8px;} #cf-wrapper #challenge-form { padding-top:25px; padding-bottom:25px; } #cf-hcaptcha-container { text-align:center;} #cf-hcaptcha-container iframe { display: inline-block;} </style>

  <meta http-equiv="refresh" content="35">

Please turn JavaScript on and reload the page.

<div id="cf-bubbles">
  <div class="bubbles"></div>
  <div class="bubbles"></div>
  <div class="bubbles"></div>
</div>
<h1><span data-translate="checking_browser">Checking your browser before accessing</span> fanfiction.net.</h1>

<div id="no-cookie-warning" class="cookie-warning" data-translate="turn_on_cookies" style="display:none">
  <p data-translate="turn_on_cookies" style="color:#bd2426;">Please enable Cookies and reload the page.</p>
</div>
<p data-translate="process_is_automatic">This process is automatic. Your browser will redirect to your requested content shortly.</p>
<p data-translate="allow_5_secs" id="cf-spinner-allow-5-secs" >Please allow up to 5 seconds&hellip;</p>
<p data-translate="redirecting" id="cf-spinner-redirecting" style="display:none">Redirecting&hellip;</p>

<script type="text/javascript">
  //<![CDATA[
  (function(){
      var a = document.getElementById('cf-content');
      a.style.display = 'block';
      var isIE = /(MSIE|Trident\/|Edge\/)/i.test(window.navigator.userAgent);
      var trkjs = isIE ? new Image() : document.createElement('img');
      trkjs.setAttribute("src", "/cdn-cgi/images/trace/jschal/js/transparent.gif?ray=6a4a5e4dad7af051");
      trkjs.id = "trk_jschal_js";
      trkjs.setAttribute("alt", "");
      document.body.appendChild(trkjs);
      var cpo=document.createElement('script');
      cpo.type='text/javascript';
      cpo.src="/cdn-cgi/challenge-platform/h/g/orchestrate/jsch/v1?ray=6a4a5e4dad7af051";
      document.getElementsByTagName('head')[0].appendChild(cpo);
    }());
  //]]>
</script>

      <div class="attribution">
        DDoS protection by <a rel="noopener noreferrer" href="https://www.cloudflare.com/5xx-error-landing/" target="_blank">Cloudflare</a>
        <br />
        <span class="ray_id">Ray ID: <code>6a4a5e4dad7af051</code></span>
      </div>
  </td>
 
</tr>

'(503) URL:'https://www.fanfiction.net/s/13919640/1/The-Winter-Moon' https://www.fanfiction.net/s/13919640/1/The-Winter-Moon Error Unknown Unknown HTTP Error in FFF ' <title>Just a moment...</title> <style type="text/css"> html, body {width: 100%; height: 100%; margin: 0; padding: 0;} body {background-color: #ffffff; color: #000000; font-family:-apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen, Ubuntu, "Helvetica Neue",Arial, sans-serif; font-size: 16px; line-height: 1.7em;-webkit-font-smoothing: antialiased;} h1 { text-align: center; font-weight:700; margin: 16px 0; font-size: 32px; color:#000000; line-height: 1.25;} p {font-size: 20px; font-weight: 400; margin: 8px 0;} p, .attribution, {text-align: center;} #spinner {margin: 0 auto 30px auto; display: block;} .attribution {margin-top: 32px;} @Keyframes fader { 0% {opacity: 0.2;} 50% {opacity: 1.0;} 100% {opacity: 0.2;} } @-webkit-keyframes fader { 0% {opacity: 0.2;} 50% {opacity: 1.0;} 100% {opacity: 0.2;} } #cf-bubbles > .bubbles { animation: fader 1.6s infinite;} #cf-bubbles > .bubbles:nth-child(2) { animation-delay: .2s;} #cf-bubbles > .bubbles:nth-child(3) { animation-delay: .4s;} .bubbles { background-color: #f58220; width:20px; height: 20px; margin:2px; border-radius:100%; display:inline-block; } a { color: #2c7cb0; text-decoration: none; -moz-transition: color 0.15s ease; -o-transition: color 0.15s ease; -webkit-transition: color 0.15s ease; transition: color 0.15s ease; } a:hover{color: #f4a15d} .attribution{font-size: 16px; line-height: 1.5;} .ray_id{display: block; margin-top: 8px;} #cf-wrapper #challenge-form { padding-top:25px; padding-bottom:25px; } #cf-hcaptcha-container { text-align:center;} #cf-hcaptcha-container iframe { display: inline-block;} </style>

  <meta http-equiv="refresh" content="35">

Please turn JavaScript on and reload the page.

<div id="cf-bubbles">
  <div class="bubbles"></div>
  <div class="bubbles"></div>
  <div class="bubbles"></div>
</div>
<h1><span data-translate="checking_browser">Checking your browser before accessing</span> fanfiction.net.</h1>

<div id="no-cookie-warning" class="cookie-warning" data-translate="turn_on_cookies" style="display:none">
  <p data-translate="turn_on_cookies" style="color:#bd2426;">Please enable Cookies and reload the page.</p>
</div>
<p data-translate="process_is_automatic">This process is automatic. Your browser will redirect to your requested content shortly.</p>
<p data-translate="allow_5_secs" id="cf-spinner-allow-5-secs" >Please allow up to 5 seconds&hellip;</p>
<p data-translate="redirecting" id="cf-spinner-redirecting" style="display:none">Redirecting&hellip;</p>

<script type="text/javascript">
  //<![CDATA[
  (function(){
      var a = document.getElementById('cf-content');
      a.style.display = 'block';
      var isIE = /(MSIE|Trident\/|Edge\/)/i.test(window.navigator.userAgent);
      var trkjs = isIE ? new Image() : document.createElement('img');
      trkjs.setAttribute("src", "/cdn-cgi/images/trace/jschal/js/transparent.gif?ray=6a4a63a29aa61a44");
      trkjs.id = "trk_jschal_js";
      trkjs.setAttribute("alt", "");
      document.body.appendChild(trkjs);
      var cpo=document.createElement('script');
      cpo.type='text/javascript';
      cpo.src="/cdn-cgi/challenge-platform/h/g/orchestrate/jsch/v1?ray=6a4a63a29aa61a44";
      document.getElementsByTagName('head')[0].appendChild(cpo);
    }());
  //]]>
</script>

      <div class="attribution">
        DDoS protection by <a rel="noopener noreferrer" href="https://www.cloudflare.com/5xx-error-landing/" target="_blank">Cloudflare</a>
        <br />
        <span class="ray_id">Ray ID: <code>6a4a63a29aa61a44</code></span>
      </div>
  </td>
 
</tr>

'(503) URL:'https://www.fanfiction.net/s/13946832/1/A-P-P-Christmas-Carol' https://www.fanfiction.net/s/13946832/1/A-P-P-Christmas-Carol Error Unknown Unknown HTTP Error in FFF '500 Server Error: Internal Server Error for url: http://localhost:8191/v1'(500) https://www.fanfiction.net/s/13977419/1/Superstitions

I'm also seeing "maximum timeout" errors from Flaresolverr. Here's a portion:

2021-10-27T04:02:48-04:00 INFO REQ-35 Cloudflare detected
2021-10-27T04:03:18-04:00 ERROR REQ-35 Error: Maximum timeout reached. maxTimeou
t=30000 (ms)
at C:\snapshot\FlareSolverr\node_modules\await-timeout\src\utils.js:20:38
at Timeout. (C:\snapshot\FlareSolverr\node_modules\await-timeout
src\index.js:32:46)
at listOnTimeout (internal/timers.js:549:17)
at processTimers (internal/timers.js:492:7)
2021-10-27T04:03:18-04:00 ERROR REQ-35 Unable to process browser request. Error:
Error: Maximum timeout reached. maxTimeout=30000 (ms)

I have gone back to using FFD, which is still working, fortunately!

Answer 87 · 2021-10-28T16:02:33.000Z

Note that FanFIction.net and seems to have put the site back into under-attack mode. Flaresolverr 1.2.9 isn't working for me at the moment - lots of timeout errors. FFDN still does, so I can use its cache. If FFDN breaks, there's still the browser.

Neither fanfictionnet_ff_proxy nor even WebToEpub Firefox extension work at the moment and I get periodically CloudFlare browser checks. DDoS?

Answer 88 · 2021-10-29T06:16:29.000Z

That's correct; I should have been more specific:

At this time, v1.2.9 is the only version of Flaresolverr known to work with FFF.

To the best of my knowledge, nsapa's fanfictionnet_ff_proxy still works fine.

I couldn't get the proxy to even load for me, so I am back to FFD until one of the other solutions starts working again.

Answer 89 · 2021-11-08T02:10:06.000Z

Even after the version of 4.7 release, I think that Flaresolverr v2 still doesn't work

2021-11-08T02:08:16+00:00 INFO REQ-188 Cloudflare detected

2021-11-08T02:08:18+00:00 INFO REQ-188 Challenge solved

2021-11-08T02:08:18+00:00 INFO REQ-188 Response in 5.024 s

2021-11-08T02:08:30+00:00 INFO REQ-189 Incoming request => POST /v1 body: {"cmd":"request.get","url":"https://www.fanfiction.net/s/13789183/45/The-Self-Made-Man","maxTimeout":30000,"download":true,"cookies":[{"name":"__cf_bm","value":"soXtaXigV8Go1GAbIVqGLurvyoWjNEpFLc6uFwjG0VA-1636337295-0-AW9hG5VIeLm8cTWv4yCzTTr14IMFXSgs4FYK5lrAEC1dE/7J3wTjQ9rbUlXtcSosDlLy8vqD7CJIsPy4V3p+TN4=","domain":".fanfiction.net","path":"/","expires":1636339095},{"name":"cf_clearance","value":"pu0iL7beRJU.QxZMKuvuW5QkmwrNexR5exqHWTBVoMg-1636337297-0-150","domain":".fanfiction.net","path":"/","expires":1667876897},{"name":"cf_chl_2","value":"3194f4f2b4fd7a8","domain":"www.fanfiction.net","path":"/","expires":1636340896},{"name":"cf_chl_prog","value":"x9","domain":"www.fanfiction.net","path":"/","expires":1636340898}],"postData":null}

2021-11-08T02:08:30+00:00 WARN REQ-189 Request parameter 'download' was removed in FlareSolverr v2.

from the Flaresolverr output, so it seems like the v2 code isn't triggering.

Answer 90 · 2021-11-08T02:14:28.000Z

At about 21:15 GMT on November 7 (if I did the timezone calculations correctly), I was able to succesfully update a ffnet fic with flaresolverr. I got similar log output to you.

Answer 91 · 2021-11-08T02:15:56.000Z

At about 21:15 GMT on November 7 (if I did the timezone calculations correctly), I was able to succesfully update a ffnet fic with flaresolverr. I got similar log output to you.

I'm trying it live right now, with Flaresolverr 2.0.0 and FFF 4.7.0 cli, and that's still what I'm getting. Trying Flaresolverr 1.2.9 kept timing out.

Answer 92 · 2021-11-08T02:17:49.000Z

Are you using Calibre? A few days ago, I seem to recall having a Flaresolverr download seem to stall up on me - the Calibre progress dialog was at something like 96% close to an hour. I eventually cancelled it and used FFDL cache for that one fic, but haven't had the issue recur again.

Answer 93 · 2021-11-08T02:25:36.000Z

Using the CLI. When it times out, it's on the first request with 1.2.9, and the request for v2 is incorrectly formatted.

Answer 94 · 2021-11-08T03:10:42.000Z

Ah, this was my error, sorry for the spam. It seems as though it is working, albeit slowly, but that the download param is always added - FlareSolverr just ignores the param, but still throws the warning.

Answer 95 · 2021-11-08T03:25:53.000Z

Correct. Since Flaresolverr warns, but continues to work, I've left the download flag in for 1.2.9 and in the hope future versions will honor it again.

Answer 96 · 2021-11-08T09:03:58.000Z

And just for those who think too slowly like myself. Just browsing through the story in the browser (to fill the cache) and using

fanficfare -o use_browser_cache=true -o use_browser_cache_only=true URL

still works (make an alias for it). Only you have to use full URL (e.g., https://www.fanfiction.net/s/13979293/1/Revelations-Regarding-Instructional-Services, not just https://www.fanfiction.net/s/13979293).

Answer 97 · 2022-09-12T01:37:57.000Z

Would it be possible to update this line to either have a 60sec timeout or to have it be a config option? Flaresolverr recently had an update which adds a 30sec delay as a temp fix to get around a CloudFlare update, but since FFF uses a hard limit of 30sec it will always time out.

Answer 98 · 2022-09-12T02:00:15.000Z

Sure, why not.

Answer 99 · 2022-10-06T00:06:09.000Z

I am struggling as I have never struggled before in my life. No matter what I do I cannot get the ini to save correctly. If I only change "YourUser" to my user "beebutler" I get 'line outside section' error. If I put the full address I get from "about:cache" I get 'line outside section' again. What am I missing? Also I commented out "browser_cache_age_limit:4.0" cause that errors also. Please help me.

Answer 100 · 2022-10-06T02:40:13.000Z

That error suggests that you haven't included appropriate section lines, don't have your settings under that section lines, or have typo'ed the section lines some how. Without knowing what's in your personal.ini, all I can do is point you to the wiki pages:

https://github.com/JimmXinu/FanFicFare/wiki/BrowserCacheFeature
https://github.com/JimmXinu/FanFicFare/wiki/INI-File

Reviewing what's in the BrowserCacheFeature page, I see that the example ini text doesn't include the [defaults] line; it is mentioned in the preceding text. I'll add it.