Index generation for clojurians-log app?

[dotemacs]

Looking at the index of shadow-cljs logs, the latest log I see right now is for 1st March.
https://clojurians-log.clojureverse.org/shadow-cljs

But looking at some other index, say for announcements, I can see the log from yesterday, 8th March.

Just to be clear, once you go into the page for 1st March:
https://clojurians-log.clojureverse.org/shadow-cljs/2020-03-01
and click on the "next" arrow, to go to the next date, you'll to the logs of any day where the logs exist.

It's just the indexing that is a bit off.

Looked at the indexing:
https://github.com/clojureverse/clojurians-log-app/blob/1eeab540241ad0743d24d461fca40d51016cb782/src/clojurians_log/components/indexer.clj

Which from my understanding should generate this index every 6000000 milliseconds.

So something must not be working as expected (or at least, the way I understand it), which prevents the indexes from being generated.

I did try to access the shadow-cljs index page in different browsers and they all show the same issue, as described above.

[plexus]

Thanks for reminding me to look into this. My suspicion is it's actually CloudFlare that's caching things too aggressively. Because of past performance issues we've cached as much as possible at multiple levels (nginx, cloudflare), but these days we're fairly healthy so that's probably not necessary any more.

You can bypass cloudflare by going to clojurians-log-internal.clojureverse.org , that way you're connecting directly to the VPS.

[dotemacs]

My suspicion is it's actually CloudFlare that's caching things too aggressively.

And bingo, going to: https://clojurians-log-internal.clojureverse.org/shadow-cljs I can see the logs for days after the 1st March.

Thanks for the quick response.

What needs to be tweaked here to get CloudFlare to not be so aggressive?
Quick search of this repo & nebula don't offer any pointers.

[plexus]

Good question... looking at CloudFlare I don't see any relevant settings, which may mean that it's really up to the app to set appropriate expiration times in headers.

Checking with curl, these are the headers I get back, which don't contain any expiration time. We do have them on the pages for individual dates it seems, so maybe we just need to set them on the index pages as well.

 % curl -I -s --insecure https://clojurians-log-internal.clojureverse.org/shadow-cljs
HTTP/2 200 
server: nginx/1.14.0 (Ubuntu)
date: Mon, 09 Mar 2020 16:13:53 GMT
content-type: text/html;charset=UTF-8
content-length: 52234
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-cache-status: HIT

 % curl -I -s --insecure https://clojurians-log-internal.clojureverse.org/shadow-cljs/2020-03-01
HTTP/2 200 
server: nginx/1.14.0 (Ubuntu)
date: Mon, 09 Mar 2020 16:13:57 GMT
content-type: text/html;charset=UTF-8
content-length: 25820
cache-control: public, max-age: 2678400
x-xss-protection: 1; mode=block
x-frame-options: SAMEORIGIN
last-modified: Mon, 9 Mar 2020 16:12:58 GMT
x-content-type-options: nosniff
x-cache-status: HIT

Would you have time to and be interested to look into that?

CloudFlare settings, nothing to see here:

[dotemacs]

I would be interested in looking into this, but not today.

So if you don't mind delegating it to me... :)

[plexus]

Not at all!

[plexus]

I purged the CloudFlare cache so the problem is over for now :)