- A bit about us and the NYT Weather Data Team (also: We're hiring!)
- Monitoring real-time warnings
- Finding and using historical data
- Documentation: https://www.weather.gov/documentation/services-web-api
- Click on "Specification"
- Building a URL:
- Base endpoint:
https://api.weather.gov/alerts/active
- We want actual warnings, not tests:
?status=actual
- Area? Let's say Maryland, Virginia and D.C. You can get fancier here, but states are easy:
&area=MD,DC,VA
- Code. This is the warning type. Tornado warning, tornado watch, etc. List is here.
- Limit.
&limit=500
- Base endpoint:
- But what about monitoring it ... for free!
- Github Actions
- Makefiles
- Slack
- Sign into Github (or quickly make an account if you haven't already)
- Go to github.com/jkeefe/nicar2024-weather
- Chose the "Fork" button
- Note that the owner is now you. Click "Create fork"
- After a minute, you will have a new screen. Note that your name is up at the top! This is your copy. You can use this now or just watch and return to it later. (If you see jkeefe instead of your name, you're on the wrong screen. Go find your copy in your github account.)
- Now click the green "<> Code" button and, after you do, the "Codespaces" tab under it.
- Click "Create Codespace on Main"
- File list on the left side
- Coding happens in the big window
- There's a terminal window at the bottom.
This is an ephemperal instance! The instance will live in your account for a few days, but unless you take active steps, it will disappear. Which is good! You need to actively save your code back to the repo to make sure you have it.
- Save all of the files you want to commit to your code
- Maybe even close them to make sure!
- Click the github source-control button
- Enter a commit message, like "edited readme"
- Use the blue dropdown arrow
- Pick "Commit & Sync"
- You will be warned that there are no changes staged, and do you want to stage and all of your changes. Say "Yes"
Running computers cost money. You get 60 hours free every month and 15 gigabytes of storage. But let's not waste those free hours.
- Go to (github.com/codespaces)[https://github.com/codespaces]
- Pick the three dots next to the Active codespace.
- Chose "Stop codespace"
- If you forget, don't worry: It'll shut down automatically after 30 minutes. But why waste that?
- Go back to the main tab, and you'll see it's gone
- Can restart
- Open the
warnings
folder - Look at the Makefile in that folder
- Open the Terminal
cd warnings
make clean
make download
Don't do make warnings
yet. Let's look at what this does!
- okay, when John says so, do
make warnings
Try each of these and then run make warnings
again.
- Add a full description: uncomment line 87 (delete the
//
) - Add a map link, using geojson.io! uncomment lines 89-92
- Ignore warnings we've seen before: uncomment the others ...
- line 57
- lines 72-79
- line 94
- line 98
You need to make a Slack app (it's easier than that sounds) and get a "bot token."
The only catch is that depending on your existing Slack setup, you may need to get an administrator to approve the creation of an app. The good thing is that you are only requesting the ability to chat:write
, which is simply posting into a channel.
Also? This works in a free Slack workspace. So you can do this all on your own if you want; just make a new workspace at slack.com
.
OK, here's what to do:
- Do steps 1, 2 and 3 in this Slack app quickstart
- In step 2, "Requesting Scopes" you just need the
chat:write
scope and it will only work in channels where the bot is invited. - The thing you want is the "Bot User OAuth Token" which always starts
oxob-
. That's the token. - Locally, you can enter the following to test it directly into your Slack:
export SLACK_TOKEN=[your_token]
For example:
export SLACK_TOKEN=xoxb-123-456-abc-zyz
Now try make slack
!
Github actions allow you to run your code in the cloud really easily.
The driver of any github action is a yaml file in the .github/workflows
directory of a repo, like this one.
In short, here's what our warnings
Github Action does:
- It starts running according to a cron statement ([here every 10 minutes(https://github.com/jkeefe/nicar2024-weather/blob/39ae476058f19021be90a70dbc59b60cef120fd5/.github/workflows/warnings.yml#L5C1-L5C103)])
- Spins up a computer running ubuntu.
- Checks out this repo
- Loads node.js and installs packages (or pulls them from a cache if nothing has changed).
- Reads a SLACK_TOKEN secret
- Runs make all just like we did in the terminal
- Commits the new data to the repo (saving our
seen.json
for next run)
To get this working, you need to do two key things:
Let the Action write back to the repo
- Settings > Actions > General > Workflow Permissions > Read and Write permissions > SAVE
- Don't forget to click "Save!"
![Screenshot 2024-03-06 at 9 49 13 PM](https://private-user-images.githubusercontent.com/312347/310738284-daa35671-8fec-4dde-a1d0-d769733097ca.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjE2ODYyMzMsIm5iZiI6MTcyMTY4NTkzMywicGF0aCI6Ii8zMTIzNDcvMzEwNzM4Mjg0LWRhYTM1NjcxLThmZWMtNGRkZS1hMWQwLWQ3Njk3MzMwOTdjYS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzIyJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcyMlQyMjA1MzNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1hYTgxM2NkZWFlZjNhY2QwNjljMGQ0OTljNzJiMDhmZGZjZjZhMmNiZjdmN2M4NzhjZmZkNzdhNWY3MDMzYzMwJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.qZwF6Z05uUIxbybcY50wpiUEZzwXza5BoszUrPm8oXw)
![Screenshot 2024-03-06 at 10 11 08 PM](https://private-user-images.githubusercontent.com/312347/310737404-64df0600-0ae7-4857-99d0-45510ee0faa4.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjE2ODYyMzMsIm5iZiI6MTcyMTY4NTkzMywicGF0aCI6Ii8zMTIzNDcvMzEwNzM3NDA0LTY0ZGYwNjAwLTBhZTctNDg1Ny05OWQwLTQ1NTEwZWUwZmFhNC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzIyJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcyMlQyMjA1MzNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0xM2Q2NTQzYmM2ZWYxZTJiY2JkMDcxOTdkMDNlNDQyZjU5YTg0YjZmNDU1MmUwMTcyYTg3NTgzZDY0NDkxNWEwJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.-jhDCTLtW0U-zZE0-MVJ2roQEVTg6zK6bY8IDJLaWNw)
Let the Action know your Slack Token
- Settings > Secrets and varialbes > Actions > Repository Secrets > New Repository Secret
![Screenshot 2024-03-06 at 9 41 52 PM](https://private-user-images.githubusercontent.com/312347/310737606-3d1873cf-2bc5-420f-898d-30265864e6a9.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjE2ODYyMzMsIm5iZiI6MTcyMTY4NTkzMywicGF0aCI6Ii8zMTIzNDcvMzEwNzM3NjA2LTNkMTg3M2NmLTJiYzUtNDIwZi04OThkLTMwMjY1ODY0ZTZhOS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzIyJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcyMlQyMjA1MzNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0wN2FiODQwM2VjZDAxNTllNDUzODEzNGNkMGE5YTk1ZWYzZjMxZmMwZTQ2MWNkN2NjM2EzY2M2NTVkODUxYjA2JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.JGTN4cQcCMKRmgfIaXY5JsbI2jvGmHTQeVEuBKiRX-U)
![Screenshot 2024-03-06 at 9 43 08 PM](https://private-user-images.githubusercontent.com/312347/310737675-599a8e19-1cde-49ff-89b0-9c3e20d84177.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjE2ODYyMzMsIm5iZiI6MTcyMTY4NTkzMywicGF0aCI6Ii8zMTIzNDcvMzEwNzM3Njc1LTU5OWE4ZTE5LTFjZGUtNDlmZi04OWIwLTljM2UyMGQ4NDE3Ny5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzIyJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcyMlQyMjA1MzNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT05M2VkY2RiMmFjYjhjNzBjYzFkYWViN2VhN2VkNTk4ZWRhYTRhM2ZmODcwNzlmMDg1ZTM2NDhkNzlhNmU2MDQ0JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.kpZzasPeELFOvikI9kS3T37lPX4shGrR1j4Rj7vJNNY)
- Enter
SLACK_TOKEN
in the top box - Paste your "Bot User OAuth Token" which always starts
oxob-
into the larger box
![Screenshot 2024-03-06 at 9 44 53 PM](https://private-user-images.githubusercontent.com/312347/310737716-99a855c4-46d7-482e-ad55-5123920e0a0f.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjE2ODYyMzMsIm5iZiI6MTcyMTY4NTkzMywicGF0aCI6Ii8zMTIzNDcvMzEwNzM3NzE2LTk5YTg1NWM0LTQ2ZDctNDgyZS1hZDU1LTUxMjM5MjBlMGEwZi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzIyJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcyMlQyMjA1MzNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0xNmRmMjcyYjk5N2UxOWJmMjE1MzM0ZjU3Njk4OWMzNDRjOTY3ZDBjNDhkNjA3NmY4OWU2NGM4YmMwNTViMmU2JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.Ln3YG-p72C-2slHsWreW626MfRWFL2x_cXPFQQxFMLE)
Then ... run your action:
- Actions > warnings > Run workflow dropdown > Run workflow button
![Screenshot 2024-03-06 at 9 40 45 PM](https://private-user-images.githubusercontent.com/312347/310737902-6db8365f-30be-4a49-9c31-5b3fa28e2068.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjE2ODYyMzMsIm5iZiI6MTcyMTY4NTkzMywicGF0aCI6Ii8zMTIzNDcvMzEwNzM3OTAyLTZkYjgzNjVmLTMwYmUtNGE0OS05YzMxLTViM2ZhMjhlMjA2OC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzIyJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcyMlQyMjA1MzNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT02ODA0ZmMxNmM1ZTU5OTAyYzY2NTJmMzE1OWY5OGYxMTQwNDBiZTNjOWQxYzVlMDE4N2ExYzI4MmEyNzkzZDdhJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.iJieNnKCFood7y1re-Hg6BqkQOXrxKYrzcr5EXUPxxI)
![Screenshot 2024-03-06 at 9 40 54 PM](https://private-user-images.githubusercontent.com/312347/310737976-5b535489-b742-43d5-8e8f-ea57a2533ece.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjE2ODYyMzMsIm5iZiI6MTcyMTY4NTkzMywicGF0aCI6Ii8zMTIzNDcvMzEwNzM3OTc2LTViNTM1NDg5LWI3NDItNDNkNS04ZThmLWVhNTdhMjUzM2VjZS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzIyJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcyMlQyMjA1MzNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT02YmZiMDE1NzU4NDYwNWU5Zjk5NmE0N2JhOTNhNzViMzlhOWJiMjM2Njc2ZDk1NzkxM2U0YjYwMTNlNzE1MTg2JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.Q2o-qcwsMh2MadjueA0Vml5nFIirW8e0wt9yh0uRKbM)
- Click the "warnings" label next to the yellow dot to watch it in action
To automate it ...
- Uncomment these two lines in the
warnings.yaml
file - Adjust the cron syntax if necessary (here's a good site for that)
- Commit your changes and push to Github.
- Look at
threadex/Makefile
- Open the Terminal
cd ../threadex
make initialize
- Optionally to generate station ids:
make stations
make scrape
ACIS documentation: http://www.rcc-acis.org/docs_webservices.html
Query builder: https://builder.rcc-acis.org/