Getting connection refused when trying to run https://github.com/linkedin/brooklin/wiki/Streaming-Text-Files-to-Kafka

Question

Getting connection refused when trying to run https://github.com/linkedin/brooklin/wiki/Streaming-Text-Files-to-Kafka

revanthpobala opened this issue 4 years ago · 13 comments

Subject of the issue

Describe your issue here.
I am evaluating brooklin and I used this link https://github.com/linkedin/brooklin/wiki/test-driving-brooklin. I followed the steps in installing the kafka and unzipping brooklin. I ran the following command

(base) stlwm224773:brooklin-1.0.2 ei6201$ bin/brooklin-server-start.sh config/server.properties >/dev/null 2>&1 &
[1] 49693

and when I ran this command https://github.com/linkedin/brooklin/wiki/Streaming-Text-Files-to-Kafka#3-create-a-datastream

bin/brooklin-rest-client.sh -o CREATE -u http://localhost:32311/ -n first-file-datastream -s NOTICE -c file -p 1 -t kafkaTransportProvider -m '{"owner":"test-user"}'

I am getting the following error.

	at java.base/java.lang.Thread.run(Thread.java:834)
[2020-07-09 11:31:18,330] ERROR Create Datastream {metadata={owner=test-user, datastreamUUID=f4830b02-c9b2-4b4b-8a96-d728e4b1262f}, transportProviderName=kafkaTransportProvider, name=first-file-datastream, destination={}, connectorName=file, source={connectionString=NOTICE, partitions=1}} failed with error. (com.linkedin.datastream.DatastreamRestClient)
com.linkedin.r2.RemoteInvocationException: com.linkedin.r2.RemoteInvocationException: Failed to get response from server for URI http://localhost:32311/datastream
	at com.linkedin.restli.internal.client.ExceptionUtil.wrapThrowable(ExceptionUtil.java:135)
	at com.linkedin.restli.internal.client.ResponseFutureImpl.getResponseImpl(ResponseFutureImpl.java:130)

Your environment

Operating System -- MacOS
Brooklin version -- 1.0.2
Java version - 8
Kafka version -- kafka_2.12-2.2.0
ZooKeeper version -- zookeeper is run as a service from homebrew

Steps to reproduce

Tell us how to reproduce this issue.

Follow steps in https://github.com/linkedin/brooklin/wiki/Streaming-Text-Files-to-Kafka#3-create-a-datastream

Expected behaviour

Tell us what should happen

Brooklin should start and transfer the file

Actual behaviour

Tell us what happens instead

It's throwing a connection request exception.

Answer 1 · 2020-07-09T18:35:59.000Z

t java.base/java.lang.Thread.run(Thread.java:834)
[2020-07-09 11:31:18,330] ERROR Create Datastream {metadata={owner=test-user, datastreamUUID=f4830b02-c9b2-4b4b-8a96-d728e4b1262f}, transportProviderName=kafkaTransportProvider, name=first-file-datastream, destination={}, connectorName=file, source={connectionString=NOTICE, partitions=1}} failed with error. (com.linkedin.datastream.DatastreamRestClient)
com.linkedin.r2.RemoteInvocationException: com.linkedin.r2.RemoteInvocationException: Failed to get response from server for URI http://localhost:32311/datastream
	at com.linkedin.restli.internal.client.ExceptionUtil.wrapThrowable(ExceptionUtil.java:135)
	at com.linkedin.restli.internal.client.ResponseFutureImpl.getResponseImpl(ResponseFutureImpl.java:130)
	at com.linkedin.restli.internal.client.ResponseFutureImpl.getResponse(ResponseFutureImpl.java:94)
	at com.linkedin.datastream.DatastreamRestClient.lambda$createDatastream$2(DatastreamRestClient.java:276)
	at com.linkedin.datastream.common.PollUtils.poll(PollUtils.java:162)
	at com.linkedin.datastream.DatastreamRestClient.createDatastream(DatastreamRestClient.java:268)
	at com.linkedin.datastream.tools.DatastreamRestClientCli.main(DatastreamRestClientCli.java:263)
Caused by: com.linkedin.r2.RemoteInvocationException: Failed to get response from server for URI http://localhost:32311/datastream
	at com.linkedin.r2.transport.http.common.HttpBridge$1.onResponse(HttpBridge.java:66)
	at com.linkedin.r2.transport.http.client.rest.ExecutionCallback.lambda$onResponse$0(ExecutionCallback.java:64)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:418)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:454)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: com.linkedin.r2.RetriableRequestException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:32311
	at com.linkedin.r2.transport.http.client.common.ChannelPoolLifecycle.lambda$create$0(ChannelPoolLifecycle.java:103)

Answer 2 · 2020-07-13T16:10:49.000Z

I am also evaluating Brooklin.
I tried on multiple machines suspecting a port blocking issue on a work laptop.

Downloaded all 3 versions from the releases page: https://github.com/linkedin/brooklin/releases/

Modified the brooklin-{version_num}/config/server.properties file trying a lower port as well.

All of the versions throw the same exception that @revanthpobala posted above.

Answer 3 · 2020-07-14T01:35:36.000Z

Hello @revanthpobala and @dholtz ,

Thanks for your interest. I tried the instructions on the referenced wiki page but could not reproduce the problem you're seeing.

Are you able to hit http://localhost:32311/datastream in your browser?

Answer 4 · 2020-07-14T14:44:55.000Z

I am unable to hit the http://localhost:32311/datastream in any browser.

Attaching the full server log:
server.log

Answer 5 · 2020-07-14T16:28:30.000Z

The error you're seeing means that either Brooklin isn't running or something (firewall?) is blocking connections to the port in question.

Two things I would try:

Confirm that Brooklin is running by checking if the relevant port is in use:
```
 sudo lsof -i -P -n | grep 32311
```
If no process is listening on that port, I'd recommend removing the redirection to /dev/null from the command that starts Brooklin to see what issues it's running into:
```
bin/brooklin-server-start.sh config/server.properties
```
Check if you have an active firewall that may be blocking connections to that port.

Answer 6 · 2020-07-14T16:42:38.000Z

@ahmedahamid I checked that no process is running on port 32311 and when I removed the /dev/null option, I am getting the following errors.

(base) stlwm224773:brooklin-1.0.2 ei6201$  sudo lsof -i -P -n | grep 32311
Password:
(base) stlwm224773:brooklin-1.0.2 ei6201$ bin/brooklin-server-start.sh config/server.properties
[0.002s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:/Users/ei6201/ESRX/Tools/brooklin-1.0.2/bin/../logs/brooklin-gc.log instead.
Unrecognized VM option 'PrintGCDateStamps'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
(base) stlwm224773:brooklin-1.0.2 ei6201$

Answer 7 · 2020-07-14T16:52:20.000Z

@revanthpobala Got it. May I ask which JDK version you're using? My guess is it must be 9+.

You can either try setting JAVA_HOME to any JDK 8 on your box or comment out this line in bin/run-class.sh as a workaround.

Answer 8 · 2020-07-14T19:30:50.000Z

I updated the line you suggested and things seem to progress a little farther.

However, brooklin doesn't seem to be able to connect with Zookeeper:

[2020-07-14 14:21:57,462] INFO Waiting for keeper state SyncConnected (org.I0Itec.zkclient.ZkClient)
[2020-07-14 14:21:57,470] INFO Opening socket connection to server localhost/<unresolved>:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-07-14 14:21:57,478] WARN Session 0x0 for server localhost/<unresolved>:2181, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.nio.channels.UnresolvedAddressException
	at java.base/sun.nio.ch.Net.checkAddress(Net.java:139)
	at java.base/sun.nio.ch.SocketChannelImpl.checkRemote(SocketChannelImpl.java:727)
	at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:741)
	at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:277)
	at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:287)
	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1021)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1064)

Kafka doesn't seem to have any issues establishing a connection with Zookeeper:

[2020-07-14 14:15:15,785] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket)
[2020-07-14 14:15:15,793] INFO zookeeper.request.timeout value is 0. feature enabled= (org.apache.zookeeper.ClientCnxn)
[2020-07-14 14:15:15,798] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2020-07-14 14:15:15,803] INFO Opening socket connection to server localhost/[0:0:0:0:0:0:0:1]:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-07-14 14:15:15,811] INFO Socket connection established, initiating session, client: /[0:0:0:0:0:0:0:1]:61488, server: localhost/[0:0:0:0:0:0:0:1]:2181 (org.apache.zookeeper.ClientCnxn)
[2020-07-14 14:15:15,978] INFO Session establishment complete on server localhost/[0:0:0:0:0:0:0:1]:2181, sessionid = 0x100a45337240000, negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn)

Answer 9 · 2020-07-14T21:29:02.000Z

@dholtz Thanks for the update.

This error means Java is unable to resolve localhost:2181, which is the same address Kafka should be connecting to (kafka_2.12-2.2.0/config/server.properties).

java.nio.channels.UnresolvedAddressException
	at java.base/sun.nio.ch.Net.checkAddress(Net.java:139)

It won't be easy to investigate this issue further without access to an environment where it's reproducible.

As an alternative, would you be open to using Vagrant to auto-provision a local VM with Brooklin, Kafka, and ZooKeeper up and running? If you are, you should find the instructions for doing that right here. That wiki page discusses setting up Brooklin to mirror Kafka clusters but you can always tinker with Brooklin's config after everything is provisioned in the local VM to repeat the steps in the text-to-Kafka demo.

Answer 10 · 2020-07-14T22:26:02.000Z

I tried the Vagrant approach as well.
I can't tell if the vagrant up worked or not. There appear to be errors in the startup, but the VM appears to be running in VirtualBox

    default: Complete!
    default: Downloading Kafka
    default: 
    default: gzip: stdin: not in gzip format
    default: tar: Child returned status 1
    default: tar: Error is not recoverable: exiting now
    default: Encountered an error while uncompressing Kafka tarball
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.

Logs of vagrant up:
vagrant_startup.txt

As expected, the rest-client call also failed.
Logs of running the command: bin/brooklin-rest-client.sh -o CREATE -u http://localhost:32311/ -n first-mirroring-stream -s "kafka://localhost:9093/^(first|second)-topic$" -c kafkaMirroringConnector -t kafkaTransportProvider -m '{"owner":"test-user","system.reuseExistingDestination":"false"}' 2>/dev/null

brooklin_rest_client.txt

Answer 11 · 2020-07-15T01:16:42.000Z

@dholtz I apologize for the inconvenience. The Kafka archive specified in the Vagrant file is no longer valid. I sent out PR #729 to fix this.

In the meantime, please feel free to make the same change in your local Vagrantfile and then do:

vagrant destroy  # delete existing VM
vagrant up       # create new VM
vagrant ssh      # ssh into it

then run the steps from the tutorial on the local VM.

Answer 12 · 2020-07-15T01:40:35.000Z

In the meantime,

Fix has been checked in. Feel free to pull it and carry out the rest of the steps above.

Answer 13 · 2020-07-23T02:05:53.000Z

@dholtz @revanthpobala are you guys unblocked?