feature: metadata extraction - [merged]
appledora opened this issue · 19 comments
Merges 41-metadata-extraction -> main
Extracts all the metadata from the dump json, except for ["article_body", "url", "namespace", "name", "in_language"] keys.
Closes #41
requested review from @martingerlach
added 4 commits
- d7a7f0e...1dd9f0e - 2 commits from branch
main - e6a3551 - update: dump.py remove irrelevant customization
- 5fb1f87 - update: html string in context.py replaced with json object
added 1 commit
- eee9c11 - update: html string in context.py replaced with json object
added 1 commit
- 602c5bce - update: html string in context.py replaced with json object
In GitLab by @geohci on Aug 23, 2022, 21:09
Commented on src/parse/article.py line 26
I do think we'll want to remove this attribute. Right now I think it's just misleading ('en' is the language, which we already have, not the page_namespace). What's the reasonining for keeping this in? also, the page_namespace_id implementation looks good to me -- thanks!
In GitLab by @geohci on Aug 23, 2022, 21:10
Commented on src/parse/utils.py line 248
let's add a quick comment explaining why we skip these
Well, this attribute is actually used in the wiki link namespace extraction task. The NAMESPACE dictionary is nested. The first key is a namespace(I used to think it's only language acronyms, but it also contains simple) , inside this are the actual wiki namespaces like article and talks. To get to the actual namespace we have to do something like NAMESPACE[primary_namespace][secondary_namespace]. The primary namespace for the wikilinks come from the page_namespace. I actually should call it something other than page_namespace. SUggestions?
donezo
I just checked and found that the page_namespace and the page_namespace_id do not correspond to each other, so we DEFINITELY have to change it. Good grief =_=
changed this line in version 8 of the diff
In GitLab by @geohci on Aug 23, 2022, 21:27
Commented on src/parse/article.py line 26
ahhh I think I understand now. I had forgotten about our prior conversation about this (https://gitlab.wikimedia.org/repos/research/html-dumps/-/merge_requests/12#note_9977). sorry, let me try to clarify:
- what you are calling
secondary_namespaceis what I mean bynamespace. This can have two forms -- the numeric identifier (0) or the prefix/name (Main). I prefer that we record the numeric identifier, which is what you have underpage_namespace_id. no change needed there. - what you are calling
primary_namespaceis the database name for the wiki. This is what you are currently extracting asself.page_namespace. You don't need to change the extraction but to avoid confusion, you actually want to call thisself.wiki_dbto match how it's usually referred to (i earlier suggestedself.wikibutself.wiki_dbis even clearer). - we also have self.language, which is not relevant for namespaces but we should continue to extract. It's usually the same as
self.wiki_dbbut there are some crucial differences -- e.g.,simpleis the wiki_db butenis the language for Simple English Wikipedia. again, no changes needed there.
made the renaming changes!
In GitLab by @geohci on Aug 23, 2022, 21:58
Commented on src/parse/article.py line 26
perfect, thanks!