Execution stops at "We will now try fetching results for Campaign ID" with no errors thrown
Closed this issue · 20 comments
When using the script I am seeing no errors and no results files are being created. This applies to DOCXs and CSVs and quick reports.
I can generate errors by using invalid campaigns IDs, so I know the script it working. I'd expect to see the message "We have successfully pulled the campaign details for ID .." but it looks like the script is not getting this far.
I have tried the API request in the browser using the same host and API Key as the config file, and I am seeing the JSON results in the browser, so I know the path is correct and authenticates.
I wonder if it is because the returned payload is quite large - I am returning campaigns with several 1000s of emails.
Thanks for reporting this. Do you see the script complete or does it just appear to be running forever?
If you have a large campaign, some of the checks could take a while. GoReport does a few extra things like getting location data for IP addresses. Useful, but it adds overheard.
I have used GoReport to process very large campaigns before, so I don't think the numbers should be an issue. It might depends on how many 1,000s of emails we're talking about here.
It's always difficult to troubleshoot problems because I can't see your data or test with it. There's not even an easy way to generate solid test data with GoPhish either. So, any information you can provide would be helpful.
No, the script almost immediately returns to the command prompt, so the execution appears to have ended. It does this in the blink of an eye, rather than as if it's waiting for the round trip to the API. We are talking less than 10,000 emails. So, not HUGE. The messages in the terminal are:
- We will now try fetching results for Campaign ID...
- We have successfully pulled the campaign details for ID...
Back to command prompt
Looking at the source, I'd expect to see some form of error message. I can't understand (because I am not a python person) the path the script is taking to get to that point. i.e. how the script is managing to end without throwing an error or reporting anything else to the terminal.
Thank you for the details. I think I see the problem, at least why you're not receiving feedback.
The "We have successfully pulled the campaign details for ID..." response is returned when GoReport is initiating the connection to GoPhish and pulling the campaign data. There is a try/except check here. GoReport tries to check the "success" value to see if it is False. According to the API documentation, success is returned with a value of False when an API key is not present or is rejected. There is no True state. If the connection is successful, GoReport just gets the campaign info.
For this reason, GoReport tries to check "success" at this stage. If there is an error, GoReport assumes "success" isn't a valid key and proceeds as if all is well. There may be a situation that is not being accounted for here. I actually have some debug code in here from a time I must have been double checking this and then forgot about it. I'm going to investigate it today and will let you know when I push some changes.
I just reproduced this situation by intentionally providing an unreachable address. I have added some additional error handling to prevent this confusion, but there are some things you can do while I finish up some code changes.
I didn't have the API connection attempt wrapped in a try/except. Oops.
This doesn't throw a fatal error because the GoPhish Python library must handle the error gracefully and not make it apparent a problem has occurred. So, GoReport happily continues forward, sees it has nothing to do, and finishes. GoReport will now check to see if the API call was successful and print details if there is a connection error.
What you can do right now: check the IP address and port provided for the API connection. make sure the computer executing GoReport can hit the GoPhish admin panel on port 3333.
GoPhish, by default, listens on 127.0.0.1:3333 for the admin panel. You can not connect to the admin panel or the API from an external connection. If you're using a remote server to run GoPhish, then you'll need to SSH to that server and use port forwarding to connect to the admin panel.
ssh -L 3333:127.0.0.1:3333 user@remote_server_IP
Then make sure GoReport is setup to connect to https://127.0.0.1:3333 in the config file and you can open https://127.0.0.1:3333 in a web browser and view GoPhish.
Please let me know if that helps you. Otherwise, I'll push some changes soon and then you can update and at least see error details that we can use to investigate further.
I was playing around with it yesterday too and had more-or-less reached the same conclusion: that it was an unreachable address.
Since you are in the code making changes, it would be really useful if you could output to the terminal the full URI that the script is trying to hit the API with. That way I can test it. I've been guessing what the full URI is and trying wget from the terminal (it works) and using that URI directly in the browser (it works too). So, I am either not using the same URI as the GoReport script, or something else is afoot...
Sure thing; I added this to GoReport. The API endpoint will now be displayed in your terminal when you run GoReport.
The endpoint is: /api/campaign/?api_key=<YOUR_API_KEY>
So, if you're using SSH -L 3333:127.0.0.1:3333 root@gophish to forward local port 3333 to port 3333 on your GoPhish server, the full address is:
https://127.0.0.1:3333/api/campaign/?api_key=<YOUR_API_KEY>
Use HTTP if you didn't setup a certificate for GoPhish to use HTTPS. That is important if you are using HTTP on your server and GoReport's configuration file is set to use an HTTPS address. It won't work.
I'll be pushing a big update soon. I'm actually reworking quite a few things and have some new features from user feedback I have received. This update will include printing the API endpoint and the try/except for the issue you're encountering. However, I strongly suspect the problem is likely the connection, maybe an HTTP/HTTPS disagreement. Worth double checking that just in case.
This hasn't been updated in a while, so I am going to assume the problem was either related to what is discussed above or was fixed in one of the recent updates.
Please do not hesitate to let me know if you have any issues!
I was waiting for you to push the updated code - containing the printed API endpoint. It's still an unresolved issue for us. Thanks
The code has been pushed to the Dev branch for the time being. Please give that version a shot and let me know how it goes.
Okay, there is lots more feedback from the script now. Very useful. Thanks.
As we suspected, it's an unreachable address. Can you please confirm:
- what the IP address should be if GoReport is installed on the same box as the GoPhish server?
- whether the connection MUST be over https?
Thanks
Let's say your config.json file for GoPhish has this block:
"admin_server" : {
"listen_url" : "127.0.0.1:3333",
"use_tls" : true,
"cert_path" : "gophish.crt",
"key_path" : "gophish.key"
},
The connection does not need to be HTTPS, but the address you provide in the GoReport config file must match your GoPhish config.json settings. With "use_tls" enabled, you must use https://. If it is disabled, use http://.
If you're running GoReport on the same server, the IP address for GoReport is 127.0.0.1. With GoPhish running, make sure GoPhish is listening on port 127.0.0.1:3333.
With an invalid ID:
[+] Connecting to GoPhish at http://localhost:3333 L.. The API Authorization endpoint is: http://localhost:3333/api/campaigns/?api_key=FOO_BAR [+] A total of 1 campaign IDs have been provided for processing. [+] GoReport will process IDs 20000 [+] Now fetching results for Campaign ID 20000 (1/1). [!] Failed to get results for campaign ID 20000 L.. Details: Campaign not found
With a valid ID:
[+] Connecting to GoPhish at http://localhost:3333 L.. The API Authorization endpoint is: http://localhost:3333/api/campaigns/?api_key=FOO_BAR [+] A total of 1 campaign IDs have been provided for processing. [+] GoReport will process IDs 200 [+] Now fetching results for Campaign ID 200 (1/1). [!] There was a problem fetching this campaign ID's details. Make sure your URL and API key are correct. Check HTTP vs HTTPS! L.. Details: 'NoneType' object is not iterable
So, it feels like it is connecting and getting the response (I can wget the API endpoint from the same folder as GoReport), but then I don't know what's happening after that.
In the interest of full disclosure: I have just worked out that this is a docker installation of GoPhish :-0
How does GoLang fetch a URL? Is a test with wget a fair test?
Yes, you can use wget or even visit the link in a browser to see the JSON for the campaign.
This error indicates what came back was empty, None. I am unable to reproduce this. If it was an HTTP/HTTPS mix-up, GoPhish usually returns:
('Connection aborted.', BadStatusLine('\x15\x03\x01\x00\x02\x02\n',))
An API key problem will return:
Invalid API Key
This error suggests the campaign is somehow empty. The ID exists, but the campaign is empty. That shouldn't be possible. I can't reproduce the error any other way, so please try this.
What do you see if you visit this link in a web browser?
Replace the last bit with your API key.
I can't use a browser because this is running on a server, not locally.
I can "wget" the API endpoints for /campaigns and for /campaigns/{id} and both results are populated and match the structure in the API documentation.
(I tried to wget the same URL with an HTTPS URL and you get an "unable to establish SSL connection" error. So, I agree, this is not an HTTP/HTTPS error.)
It looks to me like the code is throwing an error, rather than the results being empty. The error being thrown is in the try-catch block is at the line:
self.campaign = self.api.campaigns.get(campaign_id=CAM_ID)
Hope this helps.
Yes, the error itself is related to Python not liking that the variable is None. However, no variable used in this step should ever be None. GoReport would stop before it got here if that were the case. The campaign ID variable is is present and valid because we see it in the status text. The api variable should be all set because a) the API setup step passed without issue and b) you can provide an invalid ID and get a response from GoPhish.
The only thing that happens in here, in the GoReport code anyway, is setting a campaign variable equal to the results of an API get request for the campaign details, which all uses variables that we know are not None. However, there is more going on that is somewhat behind the scenes for GoReport, the code executed by the gophish Python library. It's looking like something is going wrong in the get request for the valid campaign ID that leads to the library trying to do something with a None variable.
Please check to make sure you have the latest gophish library installed (pip --upgrade install gophish). A new version was released not too long ago with some updates that GoReport takes advantage of. It's possible something in how it fetches campaigns changed as well. If you do a pip freeze you should have gophish==0.1.3.
If that doesn't help, and you are comfortable making the change, please try removing the try/catch block. Assuming you have the latest code from the dev branch and this problem is indeed occurring on line 337, you would just need to comment out line 335 and lines 338-342. Then unindent line 337 one tab. If you do this, let me know what the new error message is. It'll be like the one you have seen, but should be more verbose with some information.
It's probably out of date code then, in which case this is probably a waste of your time, but it would be useful to draw a line underneath it. Here is the trace:
Traceback (most recent call last): File "GoReport.py", line 1453, in <module> parse_options() File "/usr/local/lib/python3.4/dist-packages/click/core.py", line 722, in __call__ return self.main(*args, **kwargs) File "/usr/local/lib/python3.4/dist-packages/click/core.py", line 697, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.4/dist-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/lib/python3.4/dist-packages/click/core.py", line 535, in invoke return callback(*args, **kwargs) File "/usr/local/lib/python3.4/dist-packages/click/decorators.py", line 17, in new_func return f(get_current_context(), *args, **kwargs) File "GoReport.py", line 166, in parse_options gophish.run(id, combine, complete) File "GoReport.py", line 337, in run self.campaign = self.api.campaigns.get(campaign_id=CAM_ID) File "/usr/local/lib/python3.4/dist-packages/gophish/api/campaigns.py", line 16, in get return super(API, self).get(resource_id=campaign_id) File "/usr/local/lib/python3.4/dist-packages/gophish/api/api.py", line 49, in get return self._cls.parse(response.json()) File "/usr/local/lib/python3.4/dist-packages/gophish/models.py", line 65, in parse setattr(campaign, key, Template.parse(val)) File "/usr/local/lib/python3.4/dist-packages/gophish/models.py", line 191, in parse Attachment.parse(attachment) for attachment in val] TypeError: 'NoneType' object is not iterable
I can't update the gophish library because the installation is a docker container.
No, never a waste of my time. If there is an issue with the gophish lib I can report it to Jordan Wright. If I have an idea of what the issue is, that's even better for him. Worst case scenario, I can look into using web requests instead of the library. The library just makes it easier for me to manage the web requests.
This stack trace helps clear things up. Thank you.
The issue begins here, in GoReport:
line 337, in run self.campaign = self.api.campaigns.get(campaign_id=CAM_ID) File "/usr/local/lib/python3.4/dist-packages/gophish/api/campaigns.py",
We knew that. The self.api is the GoPhish API object and self.api.campaign.get() fetches the campaign results using the gophish lib. We see that happening in /gophish/api/api.py here in the next piece of the trace:
line 16, in get return super(API, self).get(resource_id=campaign_id) File "/usr/local/lib/python3.4/dist-packages/gophish/api/api.py",
Then there is some JSON parsing happening in /gophish/models.py:
ine 49, in get return self._cls.parse(response.json()) File "/usr/local/lib/python3.4/dist-packages/gophish/models.py",
Finally, there is more parsing in /gophish/models.py and this is where things breaK:
line 65, in parse setattr(campaign, key, Template.parse(val)) File "/usr/local/lib/python3.4/dist-packages/gophish/models.py",
Right here:
line 191, in parse Attachment.parse(attachment) for attachment in val] TypeError: 'NoneType' object is not iterable
We now know the problem is happening in the GoPhish Python API, but before we can know if Jordan can help we need to know which version you have.
I understand you can't update -- no problem. Can you run pip freeze and see which version of gophish is reports?
Sorry, pip freeze doesn't tell me anything about gophish, probably because it's running in docker. I don't really understand docker.
Hey there!
Good discussion @andyg1 and @chrismaddalena. Sorry for the delay in hopping into this conversation. I thought I was watching this repo but it looks like I was mistaken.
@chrismaddalena is correct- there have been some changes and fixes with the latest version of the Python client, so the first step will be to make sure you're using the latest version.
When you say gophish is running in docker, is that just gophish itself? The Python API library is different than the gophish server.
If, however, you are using the Python API library (and subsequently) Goreport in docker, then you should be able to see what version you're running using something like:
docker exec -it <container_id_or_name> pip freeze
If this shows an older version of the library, you'll need to update that. The best way is probably to update your dockerfile (or something - I'm not a docker expert either 😄 ). But, for troubleshooting you could try:
docker exec -it <container_id_or_name> pip install -U gophish
Hopefully the only thing running in Docker is the Gophish server, in which case the Python library and Goreport are actually just on your host, so you wouldn't have to fiddle with docker in that case. We'll hope for the best!
Let us know if that helps make any progress and if you have any questions about those steps.
The pip freeze command doesn't work in the docker container either because pip doesn't look to be installed.
I managed to get into the README file for the installation of gophish in the docker image though, and assuming the README correctly reflects the installed version, it looks like gophish here is 0.1.2
Based on what you said @jordan-wright I thought to myself "Am I missing something though?"... I don't really understand, but doesn't GoReport just use an imported library? So why does the docker container matter? Couldn't it just run on it's own, as if it was on a different server connecting via the API?
With this in mind, I stopped obsessing about trying to get the pip info for the docker image. Using pip in the host where GoReport is, I've checked and ungraded gophish to ensure that v0.1.3 is installed. It is. And the error is still there, and the stack trace looks the same as above. So, the error is still there, even though the version is supposedly 0.1.3.
(Just incase this is relevant: for some reason, I have to precede all my pip commands with python3 -m pip because I have to force the python3 version of everything, and I forget why. I think something wasn't working when I first tried to get GoReport running - something wasn't compatible with the python2 version that comes with this box I'm using. I should've written it down because it might be relevant to this discussion...)