/pgwire

PostgreSQL wire protocol implemented as a rust library.

Primary LanguageRustApache License 2.0Apache-2.0

pgwire

CI Docs

Build Postgres compatible access layer for your data service.

This library implements PostgreSQL Wire Protocol, and provide essential APIs to write PostgreSQL comptible servers and clients.

This library is a work in progress and in its relatively early stage. There is no guarantee for API stability. I'm constantly introducing break changes during this period. And also sorry for lack of API docs, examples may get you familiar with its usage.

Status

  • Message format
    • Frontend-Backend protocol messages
    • Logical replication streaming protocol message
  • Backend TCP/TLS server on Tokio
  • Frontend-Backend interaction over TCP
    • SSL Request and Response
    • Startup
      • No authentication
      • Clear-text password authentication
      • Md5 Password authentication
      • SASL SCRAM authentication (optional feature server-api-scram-ring or server-api-scram-aws-lc-rs)
        • SCRAM-SHA-256
        • SCRAM-SHA-256-PLUS
    • Simple Query and Response
    • Extended Query and Response
      • Parse
      • Bind
      • Execute
      • Describe
      • Sync
    • Termination
    • Cancel
    • Error and Notice
    • Copy
    • Notification
  • Logical replication over TCP
  • APIs
    • Startup APIs
      • AuthSource API, fetching and hashing passwords
      • Server parameters API, ready but not very good
    • Simple Query API
    • Extended Query API
      • QueryParser API, for transforming prepared statement
      • PortalStore API, for caching statements and portals
    • ResultSet builder/encoder API
    • Query Cancellation API
    • Error and Notice API
    • Copy API
      • Copy-in
      • Copy-out
      • Copy-both
    • Logical replication server API

About Postgres Wire Protocol

Postgres Wire Protocol is a relatively general-purpose Layer-7 protocol. There are 3 parts of the protocol:

  • Startup: client-server handshake and authentication.
  • Simple Query: The text-based query protocol of postgresql. Query are provided as string, and server is allowed to stream data in response.
  • Extended Query: A new sub-protocol for query which has ability to cache the query on server-side and reuse it with new parameters. The response part is identical to Simple Query.

Also note that Postgres Wire Protocol has no semantics about SQL, so literally you can use any query language, data formats or even natural language to interact with the backend.

The response are always encoded as data row format. And there is a field description as header of the data to describe its name, type and format.

Usage

Server/Backend

To use pgwire in your server application, you will need to implement two key components: startup processor and query processor. For query processing, there are two kinds of queries: simple and extended. By adding SimpleQueryHandler to your application, you will get psql command-line tool compatibility. And for more language drivers and additional prepared statement, binary encoding support, ExtendedQueryHandler is required.

Examples are provided to demo the very basic usage of pgwire on server side:

  • examples/sqlite.rs: uses an in-memory sqlite database at its core and serves it with postgresql protocol. This is a full example with both simple and extended query implementation.
  • examples/duckdb.rs: similar to sqlite example but with duckdb backend. Note that not all data types are implemented in this example.
  • examples/gluesql.rs: uses an in-memory gluesql at its core and serves it with postgresql protocol.
  • examples/server.rs: demos a server that always returns fixed results.
  • examples/secure_server.rs: demos a server with ssl support and always returns fixed results.
  • examples/scram.rs: demos how to configure more secure authentication mechanism: SCRAM
  • examples/datafusion.rs: Now moved to datafusion-postgres

Client/Frontend

I think in most case you do not need pgwire to build a postgresql client, existing postgresql client like rust-postgres should fit your scenarios. Please rise an issue if there is a scenario.

Projects using pgwire

  • GreptimeDB: Cloud-native time-series database
  • risinglight: OLAP database system for educational purpose
  • PeerDB Postgres first ETL/ELT, enabling 10x faster data movement in and out of Postgres
  • CeresDB CeresDB is a high-performance, distributed, cloud native time-series database from AntGroup.
  • dozer a real-time data platform for building, deploying and maintaining data products.

License

This library is released under MIT/Apache dual license.