Pathway is an open framework for high-throughput and low-latency real-time data processing. It is used to create Python code which seamlessly combines batch processing, streaming, and real-time API's for LLM apps. Pathway's distributed runtime (🦀-🐍) provides fresh results of your data pipelines whenever new inputs and requests are received.
Pathway provides a high-level programming interface in Python for defining data transformations, aggregations, and other operations on data streams. With Pathway, you can effortlessly design and deploy sophisticated data workflows that efficiently handle high volumes of data in real time.
Pathway is interoperable with various data sources and sinks such as Kafka, CSV files, SQL/noSQL databases, and REST API's, allowing you to connect and process data from different storage systems.
Typical use-cases of Pathway include realtime data processing, ETL (Extract, Transform, Load) pipelines, data analytics, monitoring, anomaly detection, and recommendation. Pathway can also independently provide the backbone of a light LLMops stack for real-time LLM applications.
In Pathway, data is represented in the form of Tables. Live data streams are also treated as Tables. The library provides a rich set of operations like filtering, joining, grouping, and windowing.
For any questions, you will find the community and team behind the project on Discord.
Pathway requires Python 3.9 or above.
You can install the current release of Pathway using pip:
$ pip install -U pathway
To use Pathway, you only need to import it:
import pathway as pwNow, you can easily create your processing pipeline, and let Pathway handle the updates. Once your pipeline is created, you can launch the computation on streaming data with a one-line command:
pw.run()You can then run your Pathway project (say, main.py) just like a normal Python script: $ python main.py. Alternatively, use the pathway'ish version:
$ pathway spawn python main.py
Pathway natively supports multithreading. To launch your application with 3 threads, you can do as follows:
$ pathway spawn --threads 3 python main.py
To jumpstart a Pathway project, you can use our cookiecutter template.
import pathway as pw
# Using the `demo` module to create a data stream
table = pw.demo.range_stream(nb_rows=50)
# Storing the stream into a CSV file
pw.io.csv.write(table, "output_table.csv")
# Summing all the values in a new table
sum_table = table.reduce(sum=pw.reducers.sum(pw.this.value))
# Storing the sum (which is a stream) in another CSV file
pw.io.csv.write(sum_table, "sum_table.csv")
# Now that the pipeline is built, the computation is started
pw.run()Run this example in Google Colab!
Pathway comes with a monitoring dashboard that allows you to keep track of the number of messages sent by each connector and the latency of the system. The dashboard also includes log messages.
This dashboard is enabled by default; you can disable it by passing monitoring_level = pathway.MonitoringLevel.NONE to pathway.run().
In addition to Pathway's built-in dashboard, you can use Prometheus to monitor your Pathway application.
See also: Pathway Developer Resources webpage (including API Docs).
- Core concepts of Pathway
- Basic operations
- Joins
- Groupby
- Windowby
- Transformer classes
- Input and output connectors
- Coming from pandas
- API docs
- Troubleshooting
- Linear regression on a Kafka Stream (video)
- Joins:
- Connectors:
- Monitoring Pathway with Prometheus
- Time between events in a multi-topic event stream
- Real-time linear regression (Data Engineering Weekly)
- Realtime server logs monitoring (Data Engineering Weekly)
- Data enrichment with fuzzy joins (Data Engineering Weekly)
▶️ How to do Realtime Twitter Sentiment Analysis in Python (video)
If you would like to share with us some Pathway-related content, please give an admin a shout on Discord.
Manuls (aka Pallas's Cats) are creatures with fascinating habits. As a tribute to them, we usually read pw, one of the most frequent tokens in Pathway code, as: "paw".
Pathway is made to outperform state-of-the-art technologies designed for streaming and batch data processing tasks, including: Flink, Spark, and Kafka Streaming. It also makes it possible to implement a lot of algorithms/UDF's in streaming mode which are not readily supported by other streaming frameworks (especially: temporal joins, iterative graph algorithms, machine learning routines).
If you are curious, here are some benchmarks to play with.
If you try your own benchmarks, please don't hesitate to let us know. We investigate situations in which Pathway is underperforming on par with bugs (i.e., to our knowledge, they shouldn't happen...).
Pathway continues to evolve and gain new capabilities. Here are some exciting new features that we plan to incorporate in the near future:
- Enhanced monitoring, observability, and data drift detection (integrates with Grafana visualization and other dashboarding tools).
- New connectors: interoperability with Delta Lake and Snowflake data sources.
- Easier connection setup for MongoDB.
- More performant garbage collection.
Stay up to date with the latest developments and news surrounding Pathway on our blog, or [subscribe to our newsletter].
Pathway is made to run in a "clean" Linux/MacOS + Python environment. When installing the pathway package with pip (from a wheel), you are likely to encounter a small number of Python package dependencies, such as sqlglot (used in the SQL API) and python-sat (useful for resolving dependencies during compilation). All necessary Rust crates are pre-built; the Rust compiler is not required to install Pathway, unless building from sources. A modified version of Timely/Differential Dataflow (which provides a dataflow assembly layer) is part of this repo.
Pathway is distributed on a BSL 1.1 License which allows for unlimited non-commercial use, as well as use of the Pathway package for most commercial purposes, free of charge. Code in this repository automatically converts to Open Source (Apache 2.0 License) after 4 years. Some public repos which are complementary to this one (examples, libraries, connectors, etc.) are licensed as Open Source, under the MIT license.
If you develop a library or connector which you would like to integrate with this repo, we suggest releasing it first as a separate repo on a MIT/Apache 2.0 license.
For all concerns regarding core Pathway functionalities, Issues are encouraged. For further information, don't hesitate to engage with Pathway's Discord community.