access python generator in q

[michaelsimonelli]

Having issues iterating over a python generator.

Python function is mapped to a q, function returns a python generator "", and this object is inaccessible in native q;

q)qfunc[`:returnPyGenerator;<];
q)qvar:qfunc[pargs];
q)qvar
foreign

qvar` errors with rank
no success in wrapping with .p.wrap
no success trying with .p.closure

I see documentation on creating a generator in q, and then passing it to python, and iterating in the python space, but is there anyway to do the same natively

Thanks

[Dianeod]

Hi,

Would it be possible to send on a snippet of the code that you are referring to in order clarify the problem? Also a python generator is used to produce for-loops in python, could you explain what you need the generator for?

[michaelsimonelli]

Note I've find a workaround and have included at bottom of comment.

Sure, here's the python function:

def get_account_history(self, account_id, **kwargs):
    """ List account activity. Account activity either increases or
    decreases your account balance.
    Entry type indicates the reason for the account change.
    * transfer: Funds moved to/from Coinbase to cbpro
    * match:  Funds moved as a result of a trade
    * fee:      Fee as a result of a trade
    * rebate:   Fee rebate as per our fee schedule
    If an entry is the result of a trade (match, fee), the details
    field will contain additional information about the trade.
    Args:
        account_id (str): Account id to get history of.
        kwargs (dict): Additional HTTP request parameters.
    Returns:
        list: History information for the account. Example::
            [
                {
                    "id": "100",
                    "created_at": "2014-11-07T08:19:27.028459Z",
                    "amount": "0.001",
                    "balance": "239.669",
                    "type": "fee",
                    "details": {
                        "order_id": "d50ec984-77a8-460a-b958-66f114b0de9b",
                        "trade_id": "74",
                        "product_id": "BTC-USD"
                    }
                },
                {
                    ...
                }
            ]
    """
    endpoint = '/accounts/{}/ledger'.format(account_id)
    return self._send_paginated_message(endpoint, params=kwargs)

It's from this repo: https://github.com/danpaquin/coinbasepro-python
It's a python client for interacting with the coinbase exchange api.

Many of the client calls return a paginated message, which could then just be cycled through.
However, I couldn't not find a way to successfully loop over the generator if it was created in python.
I saw examples in the reference docs on how to create a generator in kdb and then loop, but nothing on how to loop a python generator passed to the kdb.

Here is how the kdb function was called

acctHist:.cbpro.client.get_account_history[accountId];
/ acctHist returns foreign
acctHist
foreign
/ attempt to return q object results in rank error
acctHist`
rank

.p.wrap was unsuccessful
importing the function with or without '<' was also unsuccessful

workaround:
created a helper python function and wrapped the original kdb function

def pq_list(x):
    return list(x)

.py.list:.p.get[`pq_list;<];
.cbpro.getAccountHistory:{[accountId]
  accPage:.cbpro.client.get_account_history[accountId];
  accHist:"ZjFFS*"$/:`time`id`amount`balance`typ`details xcol .py.list[accPage];
  accHist};

This just iterates the entire paginated results and returns it as a table.
There's definitely a more elegant way to handle, even give the user an option to iterate through number of pages at a time - but this works for now.
If there is a way to access natively, or within embedPy, let me know!

Thanks!

[Dianeod]

Hi,

I was not able to replicate your problem as I didn't have access to the data from the github repo mentioned but below I have written an example of how a general generator from python works in a q:

p)def double(n):
    num = 0
    while num < n:
        new = num *2
        yield new
        num += 1

q)double:.p.get[`double]

q)d100:double 100

q)nxt:.p.import[`builtins]`:next

q)nxt[<][d100]
0
q)nxt[<][d100]
2
q)nxt[<][d100]
4
q)nxt[<][d100]
6
q)nxt[<][d100]
8

Thanks,
Diane

[elopezaguilera]

Closing this issue. Diane's solution allows to iterate the generator without need of converting the whole object although your solution is good too. Feel free to reopen if you still have issues with it.