Handel pagination / better abstraction in general

[andynd]

Right now I have to do stuff like that to get all settings in a dict:

def settings_dump(api):
        req = api.settings()
        if not req.ok:
                raise Exception("Could not read settings. Error {}".format(req.text))
        data = req.json()
        settings = {sett_.get('_id'): sett_.get('value') for sett_ in data.get('settings', [])}
        api_default_count = settings.get('API_Default_Count', 50)
        for call in range(1, int(data.get('total', 1) / api_default_count)+1):
                req = api.settings(count=api_default_count, offset=call*api_default_count)
                if not req.ok:
                        raise Exception("Could not read settings. Error {}".format(req.text))
                data = req.json()
                for setting in data.get('settings', []):
                        settings[setting.get('_id')] =  setting.get('value')
        if not settings:
                raise Exception("Could not read settings for unknown reason.")
        return settings

Instead I'd realy prefere to do something like that:

for key, value in api.settings():
        ...

Same for user/room/channel/...-lists

Meaning it would be great if the API wrapper would handel pagination and checking of return values. Right now the wrapper does little more than URL generation for the python requests module. It would be great to have an actual wrapper which results in a pythonic feeling of the API.

Such functionality could very easily be implemented using iterators and wrappers, for example like that:

def api_request_wrapper(api_function, *args, **kwargs):
        def inner(**inner_kwargs):
                tmp = dict(kwargs)
                tmp.update(inner_kwargs)
                req = api_function(*args, **tmp)
                if not req.ok:
                        raise Exception("Could not retrive data from Rocket.Chat API. Error: {}".format(req.text))
                return req.json()
        return inner

def api_pagination_wrapper(api_function, *args, **kwargs):
        def inner(**inner_kwargs):
                tmp = dict(kwargs)
                tmp.update(inner_kwargs)
                current_page = 0
                data_per_page = 50
                total_entries = 0
                func = api_request_wrapper(api_function, *args, **tmp)
                while total_entries >= (current_page + 1) * data_per_page or current_page == 0:
                        data = func(count=data_per_page, offset=current_page*data_per_page)
                        total_entries = data.get('total', 1)
                        attrname = set(data.keys()) - {"total", "count", "offset", "success"}
                        assert len(attrname) == 1
                        yield from data[attrname.pop()]
                        current_page += 1
        return inner

def settings_dump(api):
        return {setting.get('_id'): setting.get('value') for setting in api_pagination_wrapper(api.settings)()}

My reason to return callables and not just call the functions directly is to enable usecases like that:

def api_multitry_wrapper(api_functions, *args, **kwargs):
        def inner(**inner_kwargs):
                tmp = dict(kwargs)
                tmp.update(inner_kwargs)
                error = None
                for api_function in api_functions:
                        try:
                                return api_function(*args, **tmp)
                        except Exception as e:
                                error = e
                raise error
        return inner
        
def channel_info(api, name=None, id=None):
        data = api_multitry_wrapper([
                api_request_wrapper(api.channels_info),
                api_request_wrapper(api.groups_info),
                api_request_wrapper(api.rooms_info)], room_name=name, room_id=id)()
        return data.get("room", data.get("group", data.get("channel")))

This makes it possible to ignore the strange api design to use different APIs for private and public rooms.

[jadolg]

I understand the problem, but I'd like to maintain parity with the API regarding this. I would not like to break projects already using these methods the way they are. Maybe we could add extra methods with this behavior for every method that supports pagination?

[github-actions]

Stale issue