ContextLab/quail

combining meta data when using quail.stack_eggs

KirstensGitHub opened this issue · 6 comments

KirstensGitHub commented

currently, if you stack eggs, the metadata is lost!

we propose that the meta data should be combined in the same order/ fashion that the eggs are combined

for example, at present 'ids' is a key in the egg.meta dictionary, with a list of subject IDs. when eggs are stacked, the meta dict returns empty after stacking. instead, perhaps, it could return the same key 'ids', as the concatenated list of subIDs from both eggs that were stacked (ie subject IDs matching the order of the newly-stacked data)

andrewheusser commented

definitely agree that stacked eggs preserve meta data somehow..but what about cases where each egg contains non-overlapping meta data? concatenating the key values may be sorta weird in that case. another possibility is that its a user choice. for example, meta='concatenate' would try to concatenate keys that are shared between the two eggs, and just copy non-overlapping keys. meta='separate' would keep a separate meta dict for each egg, perhaps in a list of dicts. what do you think?

paxtonfitzpatrick commented

I like that approach. I think in the majority of situations, the user would want to concatenate shared keys, but having the option to keep and display them separately as lists of dictionaries would be useful in the case of within-subjects experiments, where there would be the same list of subjects in the meta data for each egg, and each egg being stacked is a different condition.

andrewheusser commented

cool - @paxtonfitzpatrick and/or @KirstensGitHub are you up for the challenge of implementing this? I think this would be a great first feature to implement.

here's the breakdown of steps:

  • create new branch named "enh-stack-eggs"
  • become familiar with how stack_eggs works: https://github.com/ContextLab/quail/blob/master/quail/helpers.py#L113-L142
  • add a new keyword argument to the stack_eggs function called meta with some default parameter (maybe meta='concatenate')
  • add the key word argument to the docstring describing what it is and what it can be
  • write the logic of concatenating the dictionaries or keeping them separate in a list based on what the user passes to the function
  • pass meta to the Egg function when creating the new_egg
  • test it out, perhaps with one of the example scripts, or in your notebooks to see if it does what you think it does
  • push the new branch and your changes to github
  • submit a pull request

If you complete these steps, you will be well on your way to being a python ninja :)

KirstensGitHub commented

agreed, the option to keep them separate makes sense.. I wasn't interested at all until I saw the word "ninja". Now, however.... ;)

I think @paxtonfitzpatrick and I could easily tag team it. I'm trying to put most of my energy into getting autoFR together, but this seems like a simple, useful addition to implement (famous last words)

jeremymanning commented

that's great @KirstensGitHub and @paxtonfitzpatrick! the key thing to avoid getting stuck for too long (i.e. ninja'ing your way through the code) is to continually break the problems down into smaller and smaller pieces, and then solve any of the pieces that you can. any pieces that are difficult to solve, break down further. and then when something isn't clear or doesn't work right, post issues or questions here or to slack.