weibo-public-opinion-datasets

Continuously updated Sina Weibo Public Opinion Datasets (only for research use)

Introduction

Sina Weibo is Chinese largest public social media platform. The latest and most popular social events will be disclosed and discussed on Weibo as soon as possible. Therefore, it is of great significance to build a real-time and full-scale Weibo public opinion dataset. At present, there are two methods for constructing Weibo public opinion datasets:

Searching public opinion keywords, but due to the limitation of the Weibo search interface, the specified keywords can only obtain up to 1000 data on the specified date. The overall number of datasets is too low.
Traversing all Weibo users and crawl their tweets containing keywords in a specified period. However, traversing billions of Weibo users requires much resources and time! The overall construction efficiency of datasets is too low.

We propose a new way to construct Weibo public opinion datasets, which can significantly improve the crawling efficiency while maintaining a relatively full amount of data. Specifically, we dynamically maintain a weibo active user pool, and we will only traverses these users to crawl tweets that include keywords and in a specific period.

Compared with the above method 1, the amount of data for constructing the dataset is greatly increased, and especially for popular public opinion keywords, the amount of data is increased by more than 1000 times. Compared with the method 2, the data completeness is maintained above 90%, and the construction efficiency is increased by 20 times!

Weibo Active User Pool

Based on init seed users and continuous expansion through social relationships, we have built a Weibo user pool including more than 300 million users. The active weibo user pool is constructed based on some filtering rules, and the rules are shown in the following table.

Item	Rule	Item	Rule
Follows number	> 50	Tweets number	> 50
Fans number	> 50	Recent post	< 30 days

After filtering, the number of Weibo active user pools accounts for about 5% of all Weibo users.

Download link	Active users	Update time
https://pan.baidu.com/s/1EnfGYWro-28eZAsTjlzWXw Extract Code:eolz	20,000,000	2020-04-19

Data Format

_id,nick_name,gender,province,city,brief_introduction,birthday,tweets_num,follows_num,fans_num,sex_orientation,sentiment,vip_level,authentication,labels,crawl_time
1642909335,微博小秘书,女,北京,,对微博的任何问题请私信@微博客服，或访问客服中心→http://kefu.weibo.com,0001-00-00,4923,2927,199182410,,,6级,微博官方帐号,,1576187979
2671109275,新手指南,女,北京,海淀区,教授新手使用微博方法专业户，每天推荐方法介绍、微博热点...有问题请call我，@、私信都可以，欢迎*扰~,0001-00-00,17306,2986,185217690,,,7级,新浪微博新手指南官方微博,"新手小技巧,新手大课堂,新手幼儿园",1576148765
2016713117,微博客服,女,北京,海淀区,客服中心→http://kefu.weibo.com；,01-01,14526,155,183843761,,,6级,客服中心首页即可快捷反馈问题；私信咨询请先按照提示选择对应问题分类,"微博帮助,玩转微博,微博客服",1576085666
1934183965,微博管理员,女,北京,,大家好，我的职责是和大家一起维护微博社区秩序。针对微博中的不实信息、抄袭信息、人身攻击等行为请直接投诉，我们会第一时间处理。,1984-10-15,1887,176,159145854,,,6级,新浪微博社区管理官方微博,"举报,求辟谣,不实信息",1575999468
1192329374,谢娜,女,北京,海淀区,太阳女神（也可称为喜神）的光芒照四方呀嘛照四方：）工作邮箱：w761324@qq.com（仅限工作）感谢！,0001-00-00,10183,712,125274267,,,7级,知名女主持人,"社会闲杂人等,主持人",1576189968
1195230310,何炅,男,北京,朝阳区,,0001-00-00,9334,859,118220708,,,7级,湖南卫视知名主持人,主持人,1581017948
2803301701,人民日报,男,北京,,人民日报法人微博。参与、沟通、记录时代。,1948-06-15,112982,3048,113769758,,,6级,《人民日报》法人微博,人民日报,1576189425
5878659096,超话社区,男,北京,,,2016-03-09,5616,1633,113033090,,,6级,超话社区官方微博,北京生活,1576128879
2656274875,央视新闻,男,北京,,“央视新闻”微博是**电视台新闻中心官方微博，是央视重大新闻、突发事件、重点报道的首发平台。,2012-11-01,124048,2699,107442067,,,6级,**电视台新闻中心官方微博,,1576059017
1195242865,杨幂,女,北京,,这里有一只狐狸，幸福，感恩，知足，爱~＞＜,0001-09-12,4067,613,107157676,,,7级,演员，代表作《宫》《仙剑奇侠传三》《我是证人》等,"平凡小演员,80后,一只狐狸",1581017987
....

The description of fields