Pulsar is a distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API.
Learn more about Pulsar at https://pulsar.apache.org
- Horizontally scalable (Millions of independent topics and millions of messages published per second)
- Strong ordering and consistency guarantees
- Low latency durable storage
- Topic and queue semantics
- Load balancer
- Designed for being deployed as a hosted service:
- Multi-tenant
- Authentication
- Authorization
- Quotas
- Support mixing very different workloads
- Optional hardware isolation
- Keeps track of consumer cursor position
- REST API for provisioning, admin and stats
- Geo replication
- Transparent handling of partitioned topics
- Transparent batching of messages
This repository is the main repository of Apache Pulsar. Pulsar PMC also maintains other repositories for components in the Pulsar ecosystem, including connectors, adapters, and other language clients.
- pulsar master branch
Pulsar Components | Java Version |
---|---|
Broker | 17 |
Functions / IO | 17 |
CLI | 17 |
Java Client | 8 or 11 or 17 |
- 2.8 <= pulsar ver <= 2.10
Pulsar Components | Java Version |
---|---|
Broker | 11 |
Functions / IO | 11 |
CLI | 8 or 11 |
Java Client | 8 or 11 |
- pulsar ver 2.8 <
Pulsar Components | Java Version |
---|---|
All | 8 or 11 |
Requirements:
- Java JDK 17
- Maven 3.6.1+
- zip
Note: this project includes a Maven Wrapper that can be used instead of a system installed Maven. Use it by replacing
mvn
by./mvnw
on Linux andmvnw.cmd
on Windows in the commands below.
Compile and install:
$ mvn install -DskipTests
Compile and install individual module
$ mvn -pl module-name (e.g: pulsar-broker) install -DskipTests
mvn install -Pcore-modules,-main -DskipTests
Run Unit Tests:
$ mvn test
Run Individual Unit Test:
$ mvn -pl module-name (e.g: pulsar-client) test -Dtest=unit-test-name (e.g: ConsumerBuilderImplTest)
Run Selected Test packages:
$ mvn test -pl module-name (for example, pulsar-broker) -Dinclude=org/apache/pulsar/**/*.java
Start standalone Pulsar service:
$ bin/pulsar standalone
Check https://pulsar.apache.org for documentation and examples.
Docker images must be built with Java 8 for branch-2.7
or previous branches because of
issue 8445.
Java 11 is the recommended JDK version in branch-2.8
, branch-2.9
and branch-2.10
.
Java 17 is the recommended JDK version in master
.
This builds the docker images apachepulsar/pulsar-all:latest
and apachepulsar/pulsar:latest
.
mvn clean install -DskipTests
mvn package -Pdocker,-main -am -pl docker/pulsar-all -DskipTests
After the images are built, they can be tagged and pushed to your custom repository.
Here's an example of a bash script that tags the docker images with the current version and git revision and
pushes them to localhost:32000/apachepulsar
.
image_repo_and_project=localhost:32000/apachepulsar
pulsar_version=$(mvn initialize help:evaluate -Dexpression=project.version -pl . -q -DforceStdout)
gitrev=$(git rev-parse HEAD | colrm 10)
tag="${pulsar_version}-${gitrev}"
echo "Using tag $tag"
docker tag apachepulsar/pulsar-all:latest ${image_repo_and_project}/pulsar-all:$tag
docker push ${image_repo_and_project}/pulsar-all:$tag
docker tag apachepulsar/pulsar:latest ${image_repo_and_project}/pulsar:$tag
docker push ${image_repo_and_project}/pulsar:$tag
Apache Pulsar is using lombok so you have to ensure your IDE setup with required plugins.
-
Open Project Settings.
Click File -> Project Structure -> Project Settings -> Project.
-
Select the JDK version.
From the JDK version drop-down list, select Download JDK... or choose an existing recent Java 17 JDK version.
-
In the download dialog, select version 17. You can pick a version from many vendors. Unless you have a specific preference, choose Eclipse Temurin (AdoptOpenJDK (Hotspot)).
-
Open Maven Importing Settings dialog by going to Settings -> Build, Execution, Deployment -> Build Tools -> Maven -> Importing.
-
Choose Use Project JDK for JDK for Importer setting. This uses the Java 17 JDK for running Maven when importing the project to IntelliJ. Some of the configuration in the Maven build is conditional based on the JDK version. Incorrect configuration gets chosen when the "JDK for Importer" isn't the same as the "Project JDK".
-
Validate that the JRE setting in Maven -> Runner dialog is set to Use Project JDK.
-
Open Annotation Processors Settings dialog box by going to Settings -> Build, Execution, Deployment -> Compiler -> Annotation Processors.
-
Select the following buttons:
- Enable annotation processing
- Obtain processors from project classpath
- Store generated sources relative to: Module content root
-
Set the generated source directories to be equal to the Maven directories:
- Set "Production sources directory:" to "target/generated-sources/annotations".
- Set "Test sources directory:" to "target/generated-test-sources/test-annotations".
-
Click OK.
-
Install the lombok plugin in intelliJ.
-
Open Code Style Settings dialog box by going to Settings -> Editor -> Code Style.
-
Click on the ⚙️ symbol -> Import scheme -> IntelliJ IDEA code style XML
-
Pick the file
${pulsar_dir}/src/idea-code-style.xml
-
On the dialog box that opens, click OK.
-
Ensure the scheme you just created is selected in Scheme dropdown then click OK.
-
Install the Checkstyle-IDEA plugin.
-
Open Checkstyle Settings dialog box by going to Settings -> Tools -> Checkstyle.
-
Set Checkstyle version to 8.37.
-
Set Scan scope to Only Java sources (including tests).
-
Click + button in the Configuration section to open a dialog to choose the checkfile file.
- Enter a Description. For example, Pulsar.
- Select Use a local checkstyle file.
- Set File to buildtools/src/main/resources/pulsar/checkstyle.xml.
- Select Store relative to project location.
- Click Next -> Next -> Finish.
-
Activate the configuration you just added by toggling the corresponding box.
-
Click OK.
-
When working on the Pulsar core modules in IntelliJ, reduce the number of active projects in IntelliJ to speed up IDE actions and reduce unrelated IDE warnings.
- In IntelliJ's Maven UI's tree view under "Profiles"
- Activate "core-modules" Maven profile
- De-activate "main" Maven profile
- Run the "Reload All Maven Projects" action from the Maven UI toolbar. You can also find the action by the name in the IntelliJ "Search Everywhere" window that gets activated by pressing the Shift key twice.
- In IntelliJ's Maven UI's tree view under "Profiles"
-
Run the "Generate Sources and Update Folders For All Projects" action from the Maven UI toolbar. You can also find the action by the name in the IntelliJ "Search Everywhere" window that gets activated by pressing the Shift key twice. Running the action takes about 10 minutes for all projects. This is faster when the "core-modules" profile is the only active profile.
-
In the case of compilation errors with missing Protobuf classes, ensure to run the "Generate Sources and Update Folders For All Projects" action.
-
All of the Pulsar source code doesn't compile properly in IntelliJ and there are compilation errors.
- Use the "core-modules" profile if working on the Pulsar core modules since the source code for those modules can be compiled in IntelliJ.
- Sometimes it might help to mark a specific project ignored in IntelliJ Maven UI by right-clicking the project name and select Ignore Projects from the menu.
- Currently, it is not always possible to run unit tests directly from the IDE because of the compilation issues. As a workaround, individual test classes can be run by using the
mvn test -Dtest=TestClassName
command.
-
The above steps have all been performed, but a test still won't run.
- In this case, try the following steps:
- Close IntelliJ.
- Run
mvn clean install -DskipTests
on the command line. - Reopen IntelliJ.
- If that still doesn't work:
- Verify Maven is using a supported version. Currently, the supported version of Maven is specified in the section of the main pom.xml file.
- Try "restart and clear caches" in IntelliJ and repeat the above steps to reload projects and generate sources.
- In this case, try the following steps:
Follow the instructions here to configure your Eclipse setup.
Tip
For how to make contributions to Pulsar documentation, see Pulsar Documentation Contribution Guide.
Name | Scope | |||
---|---|---|---|---|
users@pulsar.apache.org | User-related discussions | Subscribe | Unsubscribe | Archives |
dev@pulsar.apache.org | Development-related discussions | Subscribe | Unsubscribe | Archives |
Pulsar slack channel at https://apache-pulsar.slack.com/
You can self-register at https://apache-pulsar.herokuapp.com/
To report a vulnerability for Pulsar, contact the Apache Security Team. When reporting a vulnerability to security@apache.org, you can copy your email to private@pulsar.apache.org to send your report to the Apache Pulsar Project Management Committee. This is a private mailing list.
https://github.com/apache/pulsar/security/policy contains more details.
Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0
This distribution includes cryptographic software. The country in which you currently reside may have restrictions on the import, possession, use, and/or re-export to another country, of encryption software. BEFORE using any encryption software, please check your country's laws, regulations and policies concerning the import, possession, or use, and re-export of encryption software, to see if this is permitted. See http://www.wassenaar.org/ for more information.
The U.S. Government Department of Commerce, Bureau of Industry and Security (BIS), has classified this software as Export Commodity Control Number (ECCN) 5D002.C.1, which includes information security software using or performing cryptographic functions with asymmetric algorithms. The form and manner of this Apache Software Foundation distribution makes it eligible for export under the License Exception ENC Technology Software Unrestricted (TSU) exception (see the BIS Export Administration Regulations, Section 740.13) for both object code and source code.
The following provides more details on the included cryptographic software: Pulsar uses the SSL library from Bouncy Castle written by http://www.bouncycastle.org.