Pagination for the regular fetch() on feeds.
stevenlafl opened this issue · 15 comments
Right now here is the WIP:
https://github.com/stevenlafl/threads-web-client/blob/master/src/pages/api/feed.ts
and associated Feed component: https://github.com/stevenlafl/threads-web-client/blob/master/src/app/Feed.tsx
I need to get it to display more. It looks like I'm limited to 8, and refreshing may or may not get me a similar result set. Is there pagination available?
yeah I saw pagination in there but wanted to get the base stuff in before I worried about that stuff. there should be a token in there that you can use but idk how you use it. like I said, working on getting my setup back up
fwiw i'm also working on a web client so the stuff that you're bringing up is stuff i'm experiencing
@stevenlafl the endpoint for the timeline is gzipped for some reason but look at line 7 of this file:
https://github.com/threadsjs/threads.js/blob/main/src/managers/FeedManager.js
you need to pass the max_id you get from the response of this endpoint to the body of the next one.
Like this?
FeedManager.js
const RESTManager = require('./RESTManager');
class FeedManager extends RESTManager {
async fetch(max_id) {
return await this.request('/api/v1/feed/text_post_app_timeline/', {
method: 'POST',
body: 'pagination_source=text_post_feed_threads' + (max_id ? '&max_id=' + max_id : ''),
})
}
...
}
module.exports = FeedManager;
index.d.ts
...
declare module '@threadsjs/threads.js/src/managers/FeedManager.js' {
import RESTManager from "@threadsjs/threads.js/src/managers/RESTManager.js";
export default class FeedManager extends RESTManager {
fetch(max_id?: number): Promise<any>;
fetchThreads(user: string): Promise<any>;
fetchReplies(user: string): Promise<any>;
recommended(): Promise<any>;
}
}
...
I went ahead and used the RESTManager in my implementation for now for infinite scroll. I am not sure if it works for certain, but it does get me a new result set without erroring.
I might be wrong but I don’t think that’s paginating. I think your fetching a new feed on each call which might eventually flag as it doesn’t follow normal usage. The app paginates as you scroll taking a gzipbody with values such as the below.
{"media id*: "3143463776240999995 180569377", "version": 24 , "media pet": 1.0, "time info: {"10': 8920, "25*: 8920, "50": 8920, "75": 8716}, 'was_share tapped": false), {"media_id": "3143527582710965035_403375573"
"version": 24, "media pct": 0.9230769, "time info': {10: 865, "25*: 865, "50: 865, "75": 254}, "was_share tapped" :false},
{"media_id": "3143533596829412672 40730850"
*version": 24, "media pct":0.9594017, "time info": (*10": 347, "25":347, "50": 347, "75" :3473, "was_share_tapped" false),
{"media id': "3143519892437762445 403375573"
"version"
24, "media pet" :1.0, "time info" (*10" 1689, "25 1689, "50' 1079, "75":6197, "was_share_tapped" false),
{"media_id": "3143528201681201769 403375573"
"'version":24, "media_pct":1.0, "time_ info" ('10" :8920, 25":8514, "50": 8514, "75": 8514}, "was_share_tapped": false),
('media id":"3143523828357551367 180569377"
"*version": 24, "media_pct' :1.0, "time info": (™10": 1476, "25":1476, "50" :1476, "75" :1476), "was_share_tapped" false),
{"media id': "3143528196759601588 27288745"
"version": 24, "media pct": 1.0, "time info': (™10": 1274, "25":1274, "50": 1274, "75" :1274}, "was_share_tapped": false),
"media id": "3143503512472939477 4053226" , "version": 24, "media pct: 1.0,
"time info": (*10" 206, "25": 206, "50" 206, *75":2067, "was_share_tapped": false),
{"media id': "3143486442730059357 51091500"
"version": 24, "media pct: 1.0, "time_info': {10":205, "25" :205, "50': 205, "75": 205}, "was_share_tapped": false}.
{"media id": "3143468471916764278 395761959"
"media pet":1.0, "time_ info": ("10":205, "25":205, "50": 205, "75": 205}, "was_share_tapped": false).
'media id" "3143530367872313738 4094723", "Version'" 24, 'media pet" 0.93615873, 'time info" ("10" 205, "25 - 205, 50": 205, "75" 143),
"was_share_tapped" :false),
("media id': "3143459173832072096_3225079059
"version" :24, "media pet': 1.0, "time_ info': (*10": 8716, "25" :8716, "50":8716, "75" :8716), "was share_tapped":false)
]
I'll take a look at this in the morning
what thisisdice shared is the analytics that Meta is getting from your scrolling. so if you looked at a post for longer, it notes that down. idk what that stuff means but that's just it. there's a lot of analytics to threads lol, in that timeline endpoint Meta even gets your battery level
when you do a POST request to the timeline endpoint, what changes in the body when you pull to refresh and when you paginate is the reason (which changes to "pagination") and the max_id param, which you get from next_max_id from the previous response.
I'll see what I can do today about this
One more thing to be aware of is that in implementing viewing replies with posts.fetch() we also need pagination. I see:
@elijah-wright @sooluh @stevenlafl I’m able to pull endpoints, requests and responses but I’m not the best at deciphering them into reusable components. Any suggestions for a collaborative effort to get as many of them in welcome! Should I start dumping them here for others to have a go?
@stevenlafl did you figure out how to get the proper feed? I am not even sure the meta engineers know. Every time I open the app I see something new 🤣
@thisisdice sure
@stevenlafl did you figure out how to get the proper feed? I am not even sure the meta engineers know. Every time I open the app I see something new 🤣
Hah, I don't see duplicates as I scroll down. Doesn't mean what I did was correct, I just have to trust @elijah-wright. But I have it to where I can refresh and see new stuff if I want. Otherwise just keep scrolling. Even if there are duplicates, I've got it to where it removes them before trying to display.
I managed to get it to display things like this (quote posts only showing links with no text in body of post), or even quotes of replies.
When I reply to a post, I get back the full renderable post, so that feeds right back into my component structure. A few quirks but it works well. @elijah-wright is a wizard for making any of this even possible.
@birobirobiro has been assisting me with some of the styling too.
thank you lol!
https://i.instagram.com/api/v1/feed/text_post_app_timeline/
[decoded gzip] URLEncoded form
has_camera_permission: 0
feed_view_info: [{"media_id":"3143468282492089976_702310471","version":24,"media_pct":1.0,"time_info":{"10":37820642,"25":37820642,"50":37820440,"75":37820239},"was_share_tapped":false},{"media_id":"3143480192010578160_1626596167","version":24,"media_pct":0.37179488,"time_info":{"10":2481,"25":2073,"50":0,"75":0},"was_share_tapped":false}]
phone_id: d9f9f6a3-****-4ac3-b4b1-0e6004702f82
reason: cold_start_fetch **<---- this changes to pagination for subsequent** requests
battery_level: 75
timezone_offset: 3600
pagination_source: text_post_feed_threads
device_id: ****4c42-6663-****-884b-66418b4bb843
request_id: ****8f5e-dea0-****-a3d6-68484e6faed1
is_pull_to_refresh: 0 **<---- this changes 1 for homepage refresh**
_uuid: ****4c42-6663-****-884b-66418b4bb843
is_charging: 0
is_dark_mode: 0
will_sound_on: 0
session_id: ****8bd-4699-****-a42d-cf267108094b
bloks_versioning_id: 5f56efad68e1edec7801f630b5c122704ec5378adbee6609a448f105f34a9c73
https://i.instagram.com/api/v1/notifications/badge/
content-type: application/x-www-form-urlencoded; charset=UTF-8
phone_id: d9f9f6a3-****-4ac3-b4b1-0e6004702f82
user_ids: 35048****
device_id: ****4c42-6663-****-884b-66418b4bb843
_uuid: ****4c42-6663-****-884b-66418b4bb843
returns
json
{
"badge_payload": {
"35048****": {
"badge_count_map": {
"ac": 1
},
"total_count": 1
}
},
"status": "ok"
}
Confirmed that this is right, however the next_max_id ends with an equal sign which needs to be urlencoded (%3D)
So it becomes:
const RESTManager = require('./RESTManager');
class FeedManager extends RESTManager {
async fetch(max_id) {
return await this.request('/api/v1/feed/text_post_app_timeline/', {
method: 'POST',
body: 'pagination_source=text_post_feed_threads' + (max_id ? '&max_id=' + encodeURIComponent(max_id) : ''),
})
}
...
}
module.exports = FeedManager;
should be fixed