Problem parsing JSON

Question

Problem parsing JSON

Haytam95 opened this issue 6 years ago · 52 comments

When parsing a received a JSON from Java (Spring) REST service it doesn't receive nothing. I thought it was because of nulls on Json, so i filled them with empty strings and arrays but still.

Java Entity

@Entity
public class Usuario implements Serializable {

  @Id
  @GeneratedValue(strategy = GenerationType.IDENTITY)
  private Long id;

  @Column
  @NotNull
  private String username;

  @Column
  @NotNull
  private String contrasenia;

  @Column
  private String correo;

  @Column
  private Date fechaAlta;

  @Column
  private Float salud;

  @Column
  private Float blindaje;

  @Column
  private Integer energia;

  @Column
  private Integer saciedadHambre;

  @Column
  private Integer conectado;

  @JoinColumn
  @ManyToOne
  private Crew crew;

  @JoinColumn
  @ManyToOne
  private RangoCrew rangoCrew;


  @JoinColumn
  @OneToMany(fetch = FetchType.LAZY)
  private List<Vehiculo> vehiculoList;

  @JoinColumn
  @OneToMany(fetch = FetchType.LAZY)
  private List<Propiedad> propiedadList;

  @JoinColumn
  @OneToMany(fetch = FetchType.LAZY)
  private List<ItemInventario> itemInventarioList;

  @JoinColumn
  @ManyToMany(fetch = FetchType.LAZY)
  private List<Afeccion> afeccionList;
}

Java REST:

@PostMapping("findUsuarioByUsername")
  public Usuario findUsuarioByUsername(@RequestBody Map<String, String> body) {

    String username = body.get("username");

    return usuarioRepository.findUsuarioByUsername(username);
  }

JSON comming from service (Not null protected):

{
    "username": "Elba_Nanero",
    "contrasenia": "lala",
    "correo": null,
    "fechaAlta": null,
    "salud": null,
    "blindaje": null,
    "energia": null,
    "saciedadHambre": null,
    "conectado": 1,
    "crew": null,
    "rangoCrew": null,
    "vehiculoList": null,
    "propiedadList": null,
    "itemInventarioList": null,
    "afeccionList": null
}

Json null protected

{
    "username": "Elba_Nanero",
    "contrasenia": "lala",
    "correo": "asdasd",
    "fechaAlta": "2018-11-22T13:11:30.996+0000",
    "salud": 0,
    "blindaje": 0,
    "energia": 0,
    "saciedadHambre": 0,
    "conectado": 1,
    "crew": {},
    "rangoCrew": {},
    "vehiculoList": [],
    "propiedadList": [],
    "itemInventarioList": [],
    "afeccionList": []
}

SAMP - Request:

public OnGameModeInit()
{
client = RequestsClient("http://localhost:8080/", RequestHeaders(
        "Content-Type", "application/json",
        "Accept", "application/json"
   	));
    
    new Node:node = JsonObject("username", JsonString("Elba_Nanero"));
    
	RequestJSON(client, "usuario/findUsuarioByUsername", HTTP_METHOD_POST, "OnServerResponse", node, RequestHeaders("Content-Type", "application/json",
        "Accept", "application/json"));
return 1;

}

SAMP - Process Response:

public OnServerResponse(id, status, node) {

	new strResponse[512];
	new strData[128];
	
    JsonGetString(node, "contrasenia", strData);
	format(strResponse, 512, "Recibi: '%s'", strData);
	print(strResponse);
}

SAMP - Error | Empty

A little hand please?

Also, the status code is 2 (WTF)

Update

After some testing, ~~i found out that only strings are the problem~~. Look at this:

~~Float and Integers work fine~~ Neither float and integers are working, they come as "0" always

Answer 1 · 2018-11-22T13:57:00.000Z

If you have curl available, could you post the output of curling that endpoint with the -v flag? Since nothing seems to be working here, I suspect it's some edge case with the details of the response.

If you don't have curl, instead modify your code to make use of the basic Request function instead of RequestJSON and simply print out the response body and check that it's actually the expected JSON payload.

Also, ensure your headers include a Content-Type: application/json which I would assume they would since Spring should handle that automatically, but it's good to check just in case.

Answer 2 · 2018-11-22T14:18:43.000Z

I just editted my code, i found something a little strange.

Strings after response data concatenation aren't showing. Maybe encoding problem?

Also, ensure your headers include a Content-Type: application/json which I would assume they would since Spring should handle that automatically, but it's good to check just in case.

Also at Spring, i have:

@RestController
@RequestMapping(value = "/usuario/", produces = "application/json;charset=UTF-8")
public class UsuarioController {

Edit

I just wrote the response in a file...

Answer 3 · 2018-11-22T15:16:06.000Z

That first issue you noticed would be the culprit - the response contains invalid characters. If you open the output file in a hex viewer (Notepad++ can do this I think and Sublime has a plugin for it) you'll see the exact binary values.

But it seems like the problem is with the server, not the client.

Answer 4 · 2018-11-22T17:02:34.000Z

Just checked characters in response using Postman. It's ok, so i'm out of ideas

Answer 5 · 2018-11-22T17:06:10.000Z

Also running SAMP default HTTP request, it worked ok:

Answer 6 · 2018-11-23T09:40:54.000Z

Just to make sure: have you seen any error in OnRequestFailure?

Answer 7 · 2018-11-23T11:25:21.000Z

Noupe, OnRequestFailure is not being called.

And the data gets to Java service, it hit even the breakpoint and all.

Answer 8 · 2018-11-23T11:45:33.000Z

I was thinking, if there is a way in which we could debug the plugin locally with breakpoints (Using the source code)? Just to see what happens on the response.

I don't know how to program on C++ or develop SAMP Plugins, but if you give me a few hints i'm sure i'll be able to debug it and find out what's going on.

At this point, i'm pretty sure that:

SAMP to Java data is getting fine
Java response is correct and is encoded UTF-8 (tested with Postman)
HTTP native SAMP is getting Java data fine

Do you have something in mind?

Or if you prefer, i can build a .war of my service, give you some endpoints and help you out whatever you need debugging. (It uses Hibernate & JPA, so there is no need to worry about creating databases or populate with data. Just run a Tomcat server and drop it there)

Answer 9 · 2018-11-24T19:23:37.000Z

I noticed you used HTTP_METHOD_POST in the code, but in the Postman and SA:MP standard HTTP call you used a GET method, is this intentional? Ensure your endpoint can deal with receiving data through a POST method if you need to send data to it. If this is the case, there is still something going wrong with the response not raising an error.

Another thing that probably won't help here but is good practice is to check the return value of Request/RequestJSON functions, a non-zero return value indicates an error.

If none of this helps, I'll do some digging tomorrow.

Answer 10 · 2018-11-24T23:18:31.000Z

I noticed you used HTTP_METHOD_POST in the code, but in the Postman and SA:MP standard HTTP call you used a GET method, is this intentional? Ensure your endpoint can deal with receiving data through a POST method if you need to send data to it. If this is the case, there is still something going wrong with the response not raising an error.

Yeah, it was intentional when i tried default HTTP request SAMP.

Another thing that probably won't help here but is good practice is to check the return value of Request/RequestJSON functions, a non-zero return value indicates an error.

RequestJson is returning 0 and Java service is getting data ok.

If none of this helps, I'll do some digging tomorrow.

As i say, if i can help you with anything just say it!

Answer 11 · 2018-11-27T09:22:50.000Z

Just a quick update to keep you in the loop: I didn't get a chance to look at this at the weekend, took longer than usual to get my Windows laptop set up (not used it in months, had to install Visual Studio and dev tools etc). Will hopefully get time to do this soon.

Answer 12 · 2018-11-27T11:11:05.000Z

Okay thanks for the update!

Waiting for it!

Answer 13 · 2018-11-29T11:24:36.000Z

Any news? I know you're probably busy, but i'm depending on this to start a development of a gamemode

As i say, if you are too busy, just give me a hand to set up my development enviroment and i'll try to fix it

Answer 14 · 2018-11-29T19:13:22.000Z

No progress yet - the reason this is taking a while is I build the plugin on a machine I don't own any more, and going through the process of setting up the build was something I neglected to document properly...

As far as I know, it should be a case of installing vcpkg, cpprestsdk, cmake and Visual Studio. I'm currently just awaiting the cpprestsdk installation then I shall try and get the source building.

If you can provide me with a .jar/.war/.exe to run a simple server with some endpoints, it will help with debugging!

Answer 15 · 2018-11-29T19:32:04.000Z

Update: I have the environment set up and I'm ready to debug, hopefully we can get this bug fixed ASAP!

Answer 16 · 2018-11-29T19:33:45.000Z

As far as I know, it should be a case of installing vcpkg, cpprestsdk, cmake and Visual Studio. I'm currently just awaiting the cpprestsdk installation then I shall try and get the source building.

That's what i tried, but... It didn't go so well, i couldn't make the IDE recognize the libraries and more mayhem.

If you can provide me with a .jar/.war/.exe to run a simple server with some endpoints, it will help with debugging!

Of course! ~~Today i'm full, but i think tomorrow i'll be able to build a war~~

I'll try to make some room in an hour to build a war!

Answer 17 · 2018-11-29T19:35:55.000Z

Update: I have the environment set up and I'm ready to debug, hopefully we can get this bug fixed ASAP!

Perhaps it's a good time to review the build environment documentation haha

Answer 18 · 2018-11-29T19:38:03.000Z

Yep haha I'm doing that now so I don't forget for next time!

Answer 19 · 2018-11-29T20:20:16.000Z

Documentation updated with development environment setup guide: https://github.com/Southclaws/pawn-requests/blob/master/README.md#development

Answer 20 · 2018-11-29T20:31:28.000Z

So I've realised where the status code 2 is coming from, I encoded error values into the status field when calling OnRequestFailure: https://github.com/Southclaws/pawn-requests/blob/master/src/impl.cpp#L116 but the strange thing is the log should display ERROR: General error: (cause)

Answer 21 · 2018-11-29T23:41:03.000Z

I'm kind on a rush, so i could make a simple Spring application with some services (Couldn't build the war, tomorrow i have three exams at university and i think they're going to waste me !)

https://github.com/Haytam95/service_test

Just clone & put it to run in Intellij Idea

Answer 22 · 2018-11-29T23:41:39.000Z

Anyway, tomorrow i'll build the war propertly, sorry!!!

Answer 23 · 2018-12-04T15:32:31.000Z

Quick update!

I'm having some troubles building the war. For some reason, the service works when running from IDE, but i couldn't deploy the war in a Tomcat.

Did you see the repo (https://github.com/Haytam95/service_test) ? Or are waiting for the war?

I'll try harder to find a solution to the war problem

Answer 24 · 2018-12-04T15:38:03.000Z

Yeah I cloned it and ran it in IntelliJ, I couldn't get it to run as it was throwing build errors (I did resolve them but after that, couldn't get it to run for some other reason. I'm at work currently so I can't give you any more details currently)

Answer 25 · 2018-12-04T16:02:50.000Z

Don't worry.

It's my mess haha, i'll fix it and improve documentation to make it run.

Once i update everything, i'll leave a comment

Answer 26 · 2018-12-04T16:25:49.000Z

Ready! So... In a perfect world, if you just pull from the repository, everything should work (At least in intellij. I'm still struggling making a war).

If you run in any trouble, let me know! (I messed up with the POM)

Answer 27 · 2018-12-04T17:29:37.000Z

Thanks, works, I will run some tests on an evening this week (I can't tonight as I'm busy).

Answer 28 · 2018-12-05T09:25:01.000Z

Here's a quick test I ran with curl. First a call to httpbin which is used in the unit tests: https://github.com/Southclaws/pawn-requests/blob/master/test.pwn#L42

➜  ~ curl -v http://httpbin.org/get
*   Trying 34.206.253.53...
* TCP_NODELAY set
* Connected to httpbin.org (34.206.253.53) port 80 (#0)
> GET /get HTTP/1.1
> Host: httpbin.org
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Connection: keep-alive
< Server: gunicorn/19.9.0
< Date: Wed, 05 Dec 2018 09:22:48 GMT
< Content-Type: application/json
< Content-Length: 214
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
< Via: 1.1 vegur
<
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Connection": "close",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.54.0"
  },
  "origin": "91.217.237.56",
  "url": "http://httpbin.org/get"
}
* Connection #0 to host httpbin.org left intact

Then a call to the Java app:

➜  ~ curl -v localhost:8080/testGet
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> GET /testGet HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200
< Content-Type: application/json;charset=UTF-8
< Transfer-Encoding: chunked
< Date: Wed, 05 Dec 2018 09:08:29 GMT
<
* Connection #0 to host localhost left intact
{"id":"asd","name":"asdasdasd","moreData":[]}

Ignoring the request headers and response body, here's the difference between the response headers:

➜  ~ diff -y <(curl -v localhost:8080/testGet 2>&1 | grep '< ') <(curl -v http://httpbin.org/get 2>&1 | grep '< ')
< HTTP/1.1 200  					      |	< HTTP/1.1 200 OK
< Content-Type: application/json;charset=UTF-8  	      |	< Connection: keep-alive
< Transfer-Encoding: chunked    			      |	< Server: gunicorn/19.9.0
< Date: Wed, 05 Dec 2018 09:22:02 GMT   			< Date: Wed, 05 Dec 2018 09:22:02 GMT
							      >	< Content-Type: application/json
							      >	< Content-Length: 214
							      >	< Access-Control-Allow-Origin: *
							      >	< Access-Control-Allow-Credentials: true
							      >	< Via: 1.1 vegur
<       							<

I doubt the missing OK on the status would break it. The keep-alive I'm not sure about. Another thing I'm wondering is whether that Content-Type with the charset directive is something cpprestsdk isn't dealing with properly. There's also the missing Content-Length header which cpprestsdk may require to properly decode the JSON response or something.

However, I still haven't had a chance to properly debug this on the plugin so these are all just assumptions. If there's an issue in cpprestsdk (the library that underpins the plugin) there may not be anything I can do other than wait for a fix.

Answer 29 · 2018-12-05T11:13:54.000Z

Okay. So this is what i understand: Java is receiving data and cpprestsdk is receiving the response. BUT, it's going trough an exception because "it thinks" something failed, because the headers are different. Am i right?

If they can comunicate, maybe we could try:

Match the headers adding annotations to Java server
Insolate the exception and make it run anyway (if we are certain that the data is not corrupted. I think some information like the charset can be hardcoded, and the length could be scripted)

I'll search about how can i play a little with Spring headers and see how it goes.

I'll wait for more! Good job!

Answer 30 · 2018-12-05T11:39:54.000Z

With Spring and cpprestsdk both being widely used libraries, I would expect both of them to conform to standards and work fine.

Another thing is I think the current binaries are using cpprestsdk v2.10.1 however the Linux builds use the latest pulled from GitHub so Linux vs Windows may behave differently. This may be worth investigating, if the issue doesn't occur on Linux then it's likely fixed in a later version of cpprestsdk.

Answer 31 · 2018-12-05T15:36:28.000Z

Update:

Just tried including Content-Length to spring headers response. Nothing

New spring service:

Spring new response with headers:

Samp new call to service:

Samp response handling & console:

We can discard that it's because of Content-Length header.

Answer 32 · 2018-12-05T15:40:24.000Z

Also i just pushed the new service & filter that adds the content-length header.

In case you need to test, it's very easy add new headers.

Answer 33 · 2018-12-10T10:53:53.000Z

Hi there! How's it going? Any news?

Answer 34 · 2018-12-10T19:32:55.000Z

Unfortunately I can't get the demo server project to run on Windows: java.lang.IllegalStateException: Error processing condition on org.springframework.boot.autoconfigure.context.PropertyPlaceholderAutoConfiguration.propertySourcesPlaceholderConfigurer

Answer 35 · 2018-12-10T19:40:32.000Z

In the repo, there is the .jar compiled. https://github.com/Haytam95/service_test/tree/master/jar

Just install Java, open a CMD console and

java -jar demo.jar

Answer 36 · 2018-12-10T19:49:40.000Z

Thanks, it's running now.

It seems to be working fine:

*** Test OnGetData

{"id":"asd","name":"asdasdasd","moreData":[]}

PASS!

Using the following code:

#define RUN_TESTS

#include <a_samp>
#include <YSI\y_testing>

#include "requests.inc"

new Request:OnGetData_ID;
main() {
    new RequestsClient:client = RequestsClient("http://localhost:8080", RequestHeaders());
    OnGetData_ID = Request(
        client,
        "testGet",
        HTTP_METHOD_GET,
        "OnGetData",
        .headers = RequestHeaders()
    );
}
forward OnGetData(Request:id, E_HTTP_STATUS:status, data[], dataLen);
public OnGetData(Request:id, E_HTTP_STATUS:status, data[], dataLen) {
    print("*** Test OnGetData\n");

    ASSERT(id == OnGetData_ID);
    ASSERT(status == HTTP_STATUS_OK);
    print(data);

    print("\nPASS!");
}

public OnRequestFailure(Request:id, errorCode, errorMessage[], len) {
    printf("Request %d failed with %d: '%s'", _:id, errorCode, errorMessage);
}

Answer 37 · 2018-12-10T20:11:09.000Z

Dammit, i literally copied your script and run it in my PC and it still refuses to work.

I tried in three different PC: Two at my work and the notebook on home

Answer 38 · 2018-12-10T20:17:48.000Z

I suppose the latest build has some very minor changes then. I would put this down to a cpprestsdk bug that was fixed in a recent version.

I will provide new binaries for the updated version.

Answer 39 · 2018-12-10T20:20:58.000Z

Thanks, i'll wait for your update. I hope we could close this issue, it has been a pain in the ass get to the bug hahahaha

Answer 40 · 2018-12-12T18:59:23.000Z

There was any problems with the new release? Or should just download the lastest? (I don't use SAMPCTL)

Answer 41 · 2018-12-12T19:29:35.000Z

https://github.com/Southclaws/pawn-requests/releases/tag/0.8.5

Answer 42 · 2018-12-12T19:29:46.000Z

Please let me know if this fixes the issue!

Answer 43 · 2018-12-12T20:19:12.000Z

The plugin doesn't load. There's some kind of trouble with the "entry point of method" bla bla bla

Answer 44 · 2018-12-13T01:10:59.000Z

In case it's useful, vcpkg can wrap up a binary zip via the export command:

>vcpkg export --zip cpprestsdk zlib --triplet x64-windows

(other options include 7zip and nuget). The zip contains the cmake toolchain at the same relative path, so if you can build with a vcpkg repo, you can build with a vcpkg export :)

Answer 45 · 2018-12-16T14:51:02.000Z

@Southclaws in the dependencies folder you left the old cpprest.dll file (cpprest_2_10). I think that's why i'm getting the error.

Could you provide the correct dll please?

Answer 46 · 2018-12-17T15:52:33.000Z

Unfortunately I'm travelling at the moment and don't have my Windows laptop with me. I'm not going to be back until after the new year so maybe someone in the community can do a build for you if it's urgent, sorry!

Answer 47 · 2019-01-02T18:45:00.000Z

I've updated the latest release for Windows - the one that didn't work was built using Boost 66 whereas the bug was fixed in Boost 67. The latest release includes Boost 68.

Answer 48 · 2019-01-02T21:44:57.000Z

Happy new year! And nice new avatar, by the way

Now it's starting & the request reach fine to the service. BUT

It can't find the public function in the AMX to retrieve the info. It was working previously (When the data was coming corrupted). Something has changed?

I also tried using the native HTTP request (Just to check for misspelling) and it's calling the public.

When a request is created, the address of the
AMX it is called from is stored into the request object which is then
passed on to the response object when the request has finished.

Maybe #20 & #21 solution bring a new problem?

Answer 49 · 2019-01-03T08:53:03.000Z

There are no other changes between versions, all I did was recompile with newer dependencies: 0.8.4...0.8.5

The changes in #21 are on a different branch because it's a pull request, those changes were made later yesterday evening after releasing the binary. I can't reproduce this issue unfortunately but I'll have a look again later.

Answer 50 · 2019-01-05T19:50:20.000Z

0.8.6 has been released which contains the changes for #21 which fixes #20 - it passes all unit tests, let me know if this issue has been resolved.

Answer 51 · 2019-01-06T16:03:40.000Z

Yeah... But... hahaha just joking, you nailed it!

It was a tough issue, uh? Thank you so much for your time and dedication!

Answer 52 · 2019-01-06T16:37:59.000Z

Great to hear! 🎉