Welcome to the first virtual Hack Day hosted by Materialize and our good friends at Redpanda and dbt Labs! The goal of this event is to encourage knowledge-sharing between our communities (we've already learned so much just putting it together!), and give you a taste of what building streaming analytics pipelines with this stack looks like.
Maybe you've never used dbt. Maybe you're new to streaming. Maybe you're even new to dbt and streaming. But guess what: it doesn't really matter!
We'll kick things off with a quick intro to each of the projects and go over the guidelines for the event to make sure you're all set! We are of course also giving you somewhere to start!
At the end of the event, we encourage you to share your projects, experiments and learnings in Show and tell!
Throughout the day, folks from all three projects will be available to bounce off ideas, support you with your project or just...chat. To get in touch with us, join the official Slack channel or reach out in Troubleshooting!
TODO: agree on the minimum requirements for each type of swag (?) and make it clear in this section how to claim it.
Our goal was to guarantee that everyone is able to get up and running in a reasonable amount of time, as well as find something fun to work on regardless of their level of expertise with each tool. For this reason, you can find a sample project in the repo with enough plumbing to spin up an end-to-end setup that you can play around with, extend or completely modify:
To get started, fork this repo, clone it and navigate to the sample_project directory:
git clone https://github.com/<github-username>/mz-hack-day-2022.git
cd mz-hack-day-2022/sample_project
To get the sample project working all the way, you need to at least add a new dbt model that creates a materialized view. But there's a lot more you can do — choose your own Hack Day adventure! 🤠 In case you need some ideas, here are a few challenges:
| Tool | Challenge |
|---|---|
| Materialize | Replace the JSON file with a Postgres database that pushes changes to the aircraft reference data into Materialize, either through Redpanda+Debezium or directly |
| Materialize | Push data from a materialized view to a web app using TAIL. You can use our Node.js and Materialize guide as a reference! |
| Redpanda | Create a producer for a new data source. Here is an example. |
| Redpanda | Adopt one or more sources to use pandaproxy instead. Look here for an example. |
| dbt | Add a materialized view model that joins streaming data from Redpanda with the static reference data. Check the provided examples and templates for pointers. |
| dbt | Incorporate macros from the materialize-dbt-utils package into your models |
| Source | Requires authentication? | Rate limited? | URL |
|---|---|---|---|
| Example | Example | Example | Example |
If you know about other cool data sources you'd like to add to the list, feel free to open an issue or a pull request with suggestions!