Make `to_dict` recursive
tko22 opened this issue · 5 comments
the to_dict function in the class Mixin is used as a helper function in SQLAlchemy ORM models for converting the objects into python data structures for easy json serialization. However, it isn't recursive.
For example, let's have a Person and Email table. Person has name and emails as fields while Email has total_mail, email_addr, person. The emails field is a one-to-many relationship to entries in Email table. The following happens when I run to_dict() function:
person = Person.query.all()[0] # query and get the first person
person.to_dict()
# This outputs
{
name: "tim"
}
# but it doesn't show `emails`, which should be an array of emails even if it exists
# desired output...
{
name: "tim",
emails: [
{
"total_mail": 100,
"email_addr": "tim@email.com"
},
{
"total_mail": 20,
"email_addr": "tim2@gmail.com"
}
]
}Hi, I like you project and the way you organize your code !
Since few weeks I'm learning back-end. And few days ago I have done a recursive SQL alchemy to_dict.
But i did this directly by creating my own model instead of creating a Mixin class, so my function it might require modifications.
So i have init my flask app like this :
from flask_sqlalchemy import SQLAlchemy
from backend.MyModel import MyModel
app = Flask( #...)
#...
db = SQLAlchemy(app, model_class=MyModel)For the next days i ll try to handle your boilerplate, start to code REST api, then should have time to adapt my recursive function (unless you have already done it :)
class MyModel(object):
__abstract__ = True
query_class = None
query = None
max_depth_recursion = 2
DONOTSEND_MODEL = {'_sa_instance_state'}
DONOTSEND = []
_jsonified = None
modified = False
def __repr__(self):
return '<{}>'.format(self.__class__.__name__)
@property
def jsonify(self):
# if self._jsonified is None: # need using events
self._jsonified = self._to_dict_recursive(max_depth=1, depth=0, starter_obj=self)
return self._jsonified
@property
def to_dict_recursive(self, max_depth=None):
if max_depth is None:
max_depth = self.max_depth_recursion
return self._to_dict_recursive(max_depth=max_depth, depth=0, starter_obj=self)
def _to_dict_recursive(self, max_depth, depth, starter_obj):
depth += 1 # recursive depth
# functions
def anti_circular_recursion(attr):
if isinstance(starter_obj, attr.__class__) or depth >= max_depth:
# both choice possible
# return '<'+attr.__class__.__name__+'>'
return str(attr)
else:
return attr._to_dict_recursive(max_depth, depth, starter_obj)
result = {}
# __mapper__ is equivalent to db.inspect(self)
# but db (database) is not created yet cause we send this model to the constructor
for key in self.__mapper__.attrs.keys():
if key not in self.DONOTSEND:
attr = getattr(self, key)
if issubclass(type(attr), MyModel):
result[key] = anti_circular_recursion(attr)
elif isinstance(attr, list):
value = []
for obj in attr:
value.append(anti_circular_recursion(obj))
result[key] = value
# else : attr is not an instance of relationship (int, str..)
else:
result[key] = attr
return resultYeah that's kind of messy ^^"
edit : then how i use my model (by the way, you should have notice that i used max deepth against circular relationships) :
from backend import db
class User(db.Model):
__tablename__ = 'user'
id = db.Column(db.Integer, primary_key=True)
email = db.Column(db.String(254), nullable=False, unique=True)
# pseudo = db.Column(db.String(50), nullable=False, unique=True)
prenom = db.Column(db.String(50))
nom = db.Column(db.String(50))
likes = db.Column(db.Integer, default=0)
password_hash = db.Column(db.String(350))
DONOTSEND = ['password_hash']edit 2 : This the result of the recursive with deepth=5, here you can see the circurlar issue (only handled for the object which call the recursive function)
that's very cool! I haven't done this so go for it. For _init_.py, I originally had initialized the SQLAlchemy instance there but realized there could be circular imports and thus I moved it in the models module.
For the to_dict() function, let's have a parameter depth and have a default depth of 2 or 1 (I'm not sure which one would be better). This should help with the performance, especially since it is indeed recursive.
Yeah I like the place where you initialize the db instance, but i think we should create the model/mixin in the same file (base.py) or at least the same package (directory).
For the depth, i had a way to stop circular recursive relationship, but i was worried about performances. An other possibility would be to have 2 methods :
- one with a complex and gluttonous algorithm, that stop every circular relationship recursion
- one with a simple and light algorithm, that need a depth as parameter or default value
But the problem with depth parameter (as you can see on the screenshot i have shot) is that you may send too much data to a client.
That's why I'm actually learning about the cleanest way to create API. And with a REST API, we would have no need to send a recursive block of data, but only first depth with id-s of children. So a recursive block should be used only for back-end treatment. This point could seem obvious for you, but i think it is import to have in focus the final goal.
As soon I'll have time to spend on it, I'll insert the function to your project. I think I'll create the different options to facilitate the choice.
Unfortunately, I have no experience on GitHub (and branch management), I ll need your advices to not screw up everything or even put a mess there ><.
Finally I did a lot of modifications on my fork. I invite you to see my abstraction, you might find few interesting concepts.
At least I would like to have your opinion on it !