[Article Proposal] Creating a dataset of tweets

[fernandojvdasilva]

My resource

Twitter is a great resource to retrieve a vast amount of text to be used in datasets for many applications, such as sentiment analysis. In this article, we present the main APIs with some examples with code in Python, as well as restrictions and access policies.

Topic:

• Search API (and example)
• Streaming API (and example)
• Account Activity API
• Standard API restrictions

Outline:

My content is

• A Kili Tutorial / Guide / How to article
• [ x ] An Article

[theodu]

Hi @fernandojvdasilva
I think that this is a great topic and that it could help several people!
Can you provide us with a more detailed outline please ?

[fernandojvdasilva]

Hi @fernandojvdasilva I think that this is a great topic and that it could help several people! Can you provide us with a more detailed outline please ?

Hi @theodu!

Sure! Here are more details. Actually, I divided the previous "Standard API restrictions" section into the other sections and I added a section to talk about implementation issues related to the dataset collection.

• Search API (and an example): If we want to create a dataset of tweets related to certain terms, we can use this API to search for them. In this section, we will describe how the search API can be used to retrieve tweets that mention certain terms and the API limitations and some tips to overcome them when collecting a dataset. We will also discuss other data that can be retrieved with this API, such as users, friends, followers, and trending topics.

• Streaming API (and an example): If we only want to collect tweets posted from now on, we can use the streaming API to capture them. In this section, we will discuss how to monitor these tweets and capture them.

• Account Activity API: We will briefly present this API and the possibilities to information such as follows, unfollows, deleted tweets, etc.

• Collecting Dataset: In this section, we will discuss implementation issues when creating a dataset collection script. We will talk about creating a cron job for scheduling a script using the Search API and a background process to run the dataset collection script using the Streaming API. In the end, we will compare the computational resources needed in each case.

[theodu]

Hi @fernandojvdasilva
I really like this idea. It could be of great help for people doing NLP.
We currently have a lot of articles in the pipeline. I put this one on hold and I come back to you when we are ready to launch the writing of this article!

[theodu]

Hi @fernandojvdasilva!
I come back to you, the outline that you wrote is very interesting and it could help a lot of people to create a Twitter dataset.
Are you still up to writing such content?

[fernandojvdasilva]

Hi @theodu !

Sure, I can still write this article! Should I proceed to write the article straight away, or is there any other previous step?

[theodu]

Very nice !
You can begin to write it straight away!
Just make sure to read the writing guidelines in the FAQ before and follow them. You will find a template for your article in there too.

[fernandojvdasilva]

That's great @theodu! I'll take a look at the guidelines for sure and I'll start writing!

[JustBrn]

Hi @fernandojvdasilva, checking in! How is the writing going?

[fernandojvdasilva]

Hi Justine! I'm sorry, I had some personal issues and I couldn't move forward with the article... But I estimate I can finish it until the middle of April. Best regards, Fernando Vieira da Silva Em qua., 16 de mar. de 2022 às 12:12, Justine ***@***.***> escreveu:

…

Hi @fernandojvdasilva <https://github.com/fernandojvdasilva>, checking in! How is the writing going? — Reply to this email directly, view it on GitHub <#61 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKECB2BLHL4J25LDQYW3CDVAH26PANCNFSM5LGTKW6A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[JustBrn]

Hi @fernandojvdasilva no worries! Mid April is fine. Thanks :)

[JustBrn]

Hi @fernandojvdasilva ! Hope you're doing well. Have you been able to work on this article so far?

[fernandojvdasilva]

Hi Justine! I'm sorry for the delay on replying. I've just started working on the article to be honest. Could we reschedule it for May/31? Em seg, 25 de abr de 2022 13:20, Justine ***@***.***> escreveu:

…

Hi @fernandojvdasilva <https://github.com/fernandojvdasilva> ! Hope you're doing well. Have you been able to work on this article so far? — Reply to this email directly, view it on GitHub <#61 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKECB43JYKTZUIGJ3FEKHTVG3A6BANCNFSM5LGTKW6A> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[JustBrn]

Hi @fernandojvdasilva, no worries! May 31st works :)

[JustBrn]

Hi @fernandojvdasilva! How is the writing going?
Happy to get some news

[fernandojvdasilva]

Hi @JustBrn! Yes, I've been working on the article and it's on track now!

[JustBrn]

Hi @fernandojvdasilva ! Hope you're doing good. How is the writing going? Let me know if you need anything.

[fernandojvdasilva]

Hi @JustBrn ! I'm so sorry for missing the deadline again, but I'm almost finish with the article, I just need to write the last section and review it. May I we rescheduled it for June/12, please?

[JustBrn]

Hi @fernandojvdasilva of course! Let us know when you are done and we can review it.

[fernandojvdasilva]

https://drive.google.com/drive/folders/17LIG1q9c2rTR-TP8IDQpWMzWKYkw00f8?usp=sharing

[fernandojvdasilva]

Hi @JustBrn ! Just finished the article for review and shared the Google Drive's link above.

[theodu]

Hi @fernandojvdasilva
I just reviewed your article. I found it very cool. Concise and easy to reproduce.
I validate it! @JustBrn
Our team will carry the writing on the website and we will warn you as soon as it is published.
You will then be able to send the invoice when you'll have the article's url once published.
Thanks for your work!

Best wishes
Théo

[mtakili]

Hi @fernandojvdasilva

Thanks for this cool article!

Can you please provide us send some examples of tweets you've collected with your code?
We'd need it for illustration so that we can move on to the publication.

Best,
Marianne

[fernandojvdasilva]

Hi Marianne! Sure! I actually have not saved them, but I'll run the script again and send you some sample tweets later today! Sorry for not adding them to the article by the way... I should have thought about it... :) Best regards, Fernando Vieira da Silva Em qua., 29 de jun. de 2022 às 11:39, mtakili ***@***.***> escreveu:

…

Hi @fernandojvdasilva <https://github.com/fernandojvdasilva> Thanks for this cool article! Can you please provide us send some examples of tweets you've collected with your code? We'd need it for illustration so that we can move on to the publication. Best, Marianne — Reply to this email directly, view it on GitHub <#61 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKECBYQO3YDGCHHFGZLCO3VRRN2ZANCNFSM5LGTKW6A> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[fernandojvdasilva]

Hi @mtakili ! Here are some sample tweets:

Tweets collected with the Stream API:

"Covid-19: NZ to stay in orange traffic light setting https://t.co/HTmYxmRxu9"
"RT @MissMia1988: Big thanks to all the people who took the Covid vaccine without a second
thought. Now the FDA is deciding we no longer nee…"
"Our new data shows that 70% of 10-year-olds in developing countries cannot read a basic text, worsened due to COVID-re…"

Tweets collected with Search API:
"Anthony Fauci says that he's experienced a rebound in Covid symptoms
after taking a course of Pfizer Inc.'s antiviral…"
"COVID isn’t over and no amount of mass gaslighting will change that"
"Even those who think they’ve experienced ‘Covid lite’ are in denial. So many I talk to say ‘It was…"

[mtakili]

Thank you @fernandojvdasilva :) I'll come back to you once the article is published.

Have a nice one!

@theodu, here are the tweets. :)

[mtakili]

Hey, hello @fernandojvdasilva. :)

Hope you've been doing good.

We're moving on to the publication process but got stuck for the visuals because we don't have enough tweets. Could you collect more of them with the Stream API and send them to us, please?

Best,
Marianne

[fernandojvdasilva]

Hi Marianne! Sure! I'll collect some more tweets and I'll send to you until tomorrow! Best regards, Fernando. Em ter, 12 de jul de 2022 06:00, mtakili ***@***.***> escreveu:

…

Hey, hello @fernandojvdasilva <https://github.com/fernandojvdasilva>. :) Hope you've been doing good. We're moving on to the publication process but got stuck for the visuals because we don't have enough tweets. Could you collect more of them with the Stream API and send them to us, please? Best, Marianne — Reply to this email directly, view it on GitHub <#61 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKECB2V2UUFWSMOAIZLLLTVTUX4HANCNFSM5LGTKW6A> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[fernandojvdasilva]

Hi Marianne! Here are some more tweets collected right now with the Streaming API: You can find new active cases in your region here: https://t.co/L5ni2qJkXB We are also sadly reporting 16 COVID-19 related deaths today. Out of respect for the friends and whānau, we will no… https://t.co/lszc5Gj70W RT @MartyMakary: The Paxlovid rebound phenomenon wasn't disclosed in Pfizer's peer-reviewed paper on the trial in the New England Journal o… RT @brittlestar: A few people still downplaying Covid like all the dead people are just faking it and this is just the longest, worst but m… RT @RepThomasMassie: Federal judge has certified a national class action lawsuit against the entire U.S. Air Force &amp; issued a temporary res… RT @IrishEcocritic: The HIV virus takes 2-8 years to develop into AIDS. The HPV virus takes even longer to cause cervical cancer. We don't… RT @Amalgamquietude: Those of you still wearing a mask: as COVID cases go up and no mandates are put in place, you are the visible encourag… ⏺️ Wider availability of antiviral treatments From Monday 18 July 2022, the access criteria for three antiviral tr… https://t.co/UVotyCEtUe RT @geoff_buxcey: Tamara didn’t commit any act of violence, she didn’t use any weapon, she didn’t utter any threats, she didn’t cause any d… This includes all people aged 75 years and over and those who have been admitted previously to an Intensive Care Un… https://t.co/Cz6ZhvjbQU Best regards, Fernando Vieira da Silva Em qua., 13 de jul. de 2022 às 10:21, Fernando J V da Silva < ***@***.***> escreveu:

…

Hi Marianne! Sure! I'll collect some more tweets and I'll send to you until tomorrow! Best regards, Fernando. Em ter, 12 de jul de 2022 06:00, mtakili ***@***.***> escreveu: > Hey, hello @fernandojvdasilva <https://github.com/fernandojvdasilva>. :) > > Hope you've been doing good. > > We're moving on to the publication process but got stuck for the visuals > because we don't have enough tweets. Could you collect more of them with > the Stream API and send them to us, please? > > Best, > Marianne > > — > Reply to this email directly, view it on GitHub > <#61 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAKECB2V2UUFWSMOAIZLLLTVTUX4HANCNFSM5LGTKW6A> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

[mtakili]

Hello @fernandojvdasilva,

Thanks for these new tweets! Unfortunately, we need them to be complete. Can you send them without the ellipsis, please?

Best,
Marianne

[fernandojvdasilva]

Hi Marianne! Oh, I see! Actually I got the ellipsis by using the API, by I 'm going to check if I can get the full tweet. I just need some more days this time, because I may need to change something in the code. Best regards, Fernando. Em seg, 18 de jul de 2022 04:42, mtakili ***@***.***> escreveu:

…

Hello @fernandojvdasilva <https://github.com/fernandojvdasilva>, Thanks for these new tweets! Unfortunately, we need them to be complete. Can you send them without the ellipsis, please? Best, Marianne — Reply to this email directly, view it on GitHub <#61 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKECB6W7MTFOAH3ZUOTLFLVUUDHLANCNFSM5LGTKW6A> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[mtakili]

Hello Fernando! Thanks for letting me know. No worries, see you in a few days, then. =) Best, Marianne Le mar. 19 juil. 2022 à 15:50, Fernando Vieira da Silva < ***@***.***> a écrit :

…

Hi Marianne! Oh, I see! Actually I got the ellipsis by using the API, by I 'm going to check if I can get the full tweet. I just need some more days this time, because I may need to change something in the code. Best regards, Fernando. Em seg, 18 de jul de 2022 04:42, mtakili ***@***.***> escreveu: > Hello @fernandojvdasilva <https://github.com/fernandojvdasilva>, > > Thanks for these new tweets! Unfortunately, we need them to be complete. > Can you send them without the ellipsis, please? > > Best, > Marianne > > — > Reply to this email directly, view it on GitHub > < #61 (comment) >, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/AAKECB6W7MTFOAH3ZUOTLFLVUUDHLANCNFSM5LGTKW6A > > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> > — Reply to this email directly, view it on GitHub <#61 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AZ3DRQMRJ3OQTEWT3OQOVADVU2XD7ANCNFSM5LGTKW6A> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[mtakili]

Hey @fernandojvdasilva!

Status update: we'll publish your article Friday, next week. :)

I'll send you the link once it's live.

Thanks for your flexibility, cooperation, and work!

Cheers
Marianne

[mtakili]

Hi @fernandojvdasilva!

Can you fill the thumbnail section of your article? I need at least a picture of you that you like so I use it for this article's promotion.

Talk soon,
Marianne

[fernandojvdasilva]

Hi Marianne! I'm sorry for my delay in reply... Actually, I just figured out why the tweets were ending with ellipsis and I had to do a minor change in my code and an important add to the article. I'll send another version by tomorrow. Sure, I'll send the thumbnail too! After all, I'm very excited the article is about to be published! I just hope you can still consider this final change, because I think it's important for readers to be able to retrieve full tweets. Best regards, Fernando Vieira da Silva Em sex., 29 de jul. de 2022 às 13:45, mtakili ***@***.***> escreveu:

…

Hi @fernandojvdasilva <https://github.com/fernandojvdasilva>! Can you fill the thumbnail section of your article? I need at least a picture of you that you like so I use it for this article's promotion. Talk soon, Marianne — Reply to this email directly, view it on GitHub <#61 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKECBY6OIJHOMC4FCL5A6LVWQDDPANCNFSM5LGTKW6A> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[fernandojvdasilva]

Hi Marianne! I've just uploaded a new version of the article in the shared drive: https://drive.google.com/drive/folders/17LIG1q9c2rTR-TP8IDQpWMzWKYkw00f8?usp=sharing The changes are marked in yellow in Page 4 only, along with a piece of code in the same part of the text. Regarding the thumbnail, I've added one into the presentation that I uploaded into the same directory, is that correct or should I add it somewhere else? In any case, I'm sending the same thumbnail here too: Also, here are some full tweets for the case you want to update (collected with the Streaming API): He didn't "test positive again," he's still sick. The gross CDC's guideline to isolate for 10 days after symptom onset is grossly inadequate because 10 days might not be long enough for the virus to clear out of your system. 1% of people here wear masks. All of them get Covid over and over again. Unless they avoid people. Guatemala was amazing (except for the Covid I contracted while there). Back home now fully recovered and happy. Nope, only people who returned to work, not yet even with pre-covid I think it was George Soros. He actually talked about Covid back in the 80’s when he was donating up democrats. They were clearly funding virus research. 😳 Biden's cough returns on fourth day of COVID 'rebound' battle Best regards! Fernando Vieira da Silva Em ter., 2 de ago. de 2022 às 17:44, Fernando J V da Silva < ***@***.***> escreveu:

…

Hi Marianne! I'm sorry for my delay in reply... Actually, I just figured out why the tweets were ending with ellipsis and I had to do a minor change in my code and an important add to the article. I'll send another version by tomorrow. Sure, I'll send the thumbnail too! After all, I'm very excited the article is about to be published! I just hope you can still consider this final change, because I think it's important for readers to be able to retrieve full tweets. Best regards, Fernando Vieira da Silva Em sex., 29 de jul. de 2022 às 13:45, mtakili ***@***.***> escreveu: > Hi @fernandojvdasilva <https://github.com/fernandojvdasilva>! > > Can you fill the thumbnail section of your article? I need at least a > picture of you that you like so I use it for this article's promotion. > > Talk soon, > Marianne > > — > Reply to this email directly, view it on GitHub > <#61 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAKECBY6OIJHOMC4FCL5A6LVWQDDPANCNFSM5LGTKW6A> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

[mtakili]

Hi @fernandojvdasilva,

Thanks for these updates.

We'll have to review the additional content you've created before posting it. As many of our teammates are off, it's likely to take a week or more. Sorry for this!

I'll let you know when we've determined a publishing date.

Best,
Marianne

[fernandojvdasilva]

Hi Marianne! Thank you for considering these updates! I understand it takes time to evaluate, but the important thing is that this change will make things easier for readers I guess (as they won't have problems and they won't have to research more than the article already provides). Best regards, Fernando Vieira da Silva Em qua., 3 de ago. de 2022 às 08:33, Marianne ***@***.***> escreveu:

…

Hi @fernandojvdasilva <https://github.com/fernandojvdasilva>, Thanks for these updates. We'll have to review the additional content you've created before posting it. As many of our teammates are off, it's likely to take a week or more. Sorry for this! I'll let you know when we've determined a publishing date. Best, Marianne — Reply to this email directly, view it on GitHub <#61 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKECB2CDWTB3NSWQ57SUM3VXJKJHANCNFSM5LGTKW6A> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[theodu]

Hi @fernandojvdasilva
Sorry for the time to give an answer and a review, a part of the team is currently in holidays.
Thank you to have taken the initiative to add this feature to show the whole tweet with your scrapper! It will indeed help people to use the code more easily!

Also, In the V2 version, in the doc here: https://docs.google.com/document/d/1nL3zfPAvv1lgq1ZYfGUdDPlkcgGssRur/edit, we allowed ourself to add a new part "Importing your dataset in Kili", to show how to import a tweet in our platform with a nice format. We have a feature to customize the format with which we import text assets in the platform and I tried to reproduce a tweet aspect. As it can be quite advanced to take this into hands, I spare you this and I wrote the code myself. Could you have a look at it and say if you accept to add this at the end of your article ?

Best
Théo

[fernandojvdasilva]

Hi Théo! I think it was a great add for sure! I loved how it fits so well to the article! :D Best regards, Fernando Vieira da Silva Em ter., 16 de ago. de 2022 às 06:29, theodu ***@***.***> escreveu:

…

Hi @fernandojvdasilva <https://github.com/fernandojvdasilva> Sorry for the time to give an answer and a review, a part of the team is currently in holidays. Thank you to have taken the initiative to add this feature to show the whole tweet with your scrapper! It will indeed help people to use the code more easily! Also, In the V2 version, in the doc here: https://docs.google.com/document/d/1nL3zfPAvv1lgq1ZYfGUdDPlkcgGssRur/edit, we allowed ourself to add a new part "Importing your dataset in Kili", to show how to import a tweet in our platform with a nice format. We have a feature to customize the format with which we import text assets in the platform and I tried to reproduce a tweet aspect. As it can be quite advanced to take this into hands, I spare you this and I wrote the code myself. Could you have a look at it and say if you accept to add this at the end of your article ? Best Théo — Reply to this email directly, view it on GitHub <#61 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKECB3FBKQQSOXNSKLSVZ3VZNNQJANCNFSM5LGTKW6A> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[mtakili]

Hello @fernandojvdasilva!

Thanks again for this nice addition and your flexibility! We're very happy to announce to you it has been published!

Here's the link: https://kili-technology.com/blog/creating-a-dataset-of-tweets. You can now send your invoice as mentioned in the guidelines.

Also, can you send me your @ on Twitter and LinkedIn, so we're sure to mention you correctly when sharing this on social? Additionally, accordingly to the guidelines as well, please make sure to provide us with a photograph of you that you like –so we can include it in the visuals to be used.

Thanks for your help and congrats again. ;)

[mtakili]

Hello @fernandojvdasilva!

Can you also email me (marianne.tavel@kili-technology.com) your bill? We've been experiencing some bugs and I'd like to make sure you get paid asap.

Best,
Marianne

[fernandojvdasilva]

Hi Marianne! I'm very happy to see the article was published! I would like to thank you and the Kili team for this opportunity! =D I'm sending a picture that I use on LinkedIn, but with a high resolution (please let me know if you need one with lower resolution). My user on Twitter is @FernandoJVSilva and my user on LinkedIn is https://www.linkedin.com/in/fernando-n2vec/ Regarding the invoice, I just need a couple of days to check the information for international transfer with my bank, so I'll fill up the invoice. Best regards, Fernando. [image: linkedin_fernando.jpg] Em sex., 26 de ago. de 2022 às 03:10, Marianne ***@***.***> escreveu:

…

Hello @fernandojvdasilva <https://github.com/fernandojvdasilva>! Can you also email me ***@***.***) your bill? We've been experiencing some bugs and I'd like to make sure you get paid asap. Best, Marianne — Reply to this email directly, view it on GitHub <#61 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKECB7ESICSQCRA6NT4BRLV3BNW7ANCNFSM5LGTKW6A> . You are receiving this because you were mentioned.Message ID: ***@***.***>