Importing a .vespene file the variables are stored using single quotes instead of double quotes making the json invalid

[tommy-couzens]

Bug description
When importing a project, via the organisations tab in the GUI, using a .vespene I got an error when the project running and looked at the variables and saw they were stored in single quotes instead of double quotes. After changing it to double quotes the project ran fine.

Steps to reproduce

My .vespene file is the following:

name: "Tommys Second Project"
script: "scripts/example.sh"
timeout: 60
repo_branch: master
variables:
   testvar: "hehehoho"
   luggage_code: "5150"
worker_pool: 'tutorial-pool'

This builds me a project that I should then be able to run.

Actual behavior

After importing the variables in the project are as follows:

When I run the project I get this error pop up:

Traceback (most recent call last):
  File "/opt/vespene/vespene/plugins/variables/common.py", line 39, in _get_variables
    obj_variables = json.loads(obj.variables)
  File "/usr/lib64/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.6/json/decoder.py", line 355, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/vespene/vespene/views/project.py", line 141, in project_start
    start_project_from_ui(request, obj)
  File "/opt/vespene/vespene/manager/jobkick.py", line 81, in start_project_from_ui
    start_project(project, launch_answers=launch_answers, request=request)
  File "/opt/vespene/vespene/manager/jobkick.py", line 94, in start_project
    variables = compute_variable_chain(project)
  File "/opt/vespene/vespene/manager/jobkick.py", line 163, in compute_variable_chain
    return VariableManager(project).compute()
  File "/opt/vespene/vespene/common/variables.py", line 22, in compute
    computed = plugin.compute(self.project, results)
  File "/opt/vespene/vespene/plugins/variables/common.py", line 60, in compute
    variables.update(self._get_variables(project))
  File "/opt/vespene/vespene/plugins/variables/common.py", line 41, in _get_variables
    raise Exception("failed to parse JSON from object: %s" % obj.variables)
Exception: failed to parse JSON from object: {'testvar': 'hehehoho', 'luggage_code': '5150'}

I changed the single quotes to double quotes manually as so:

And the project without problems.

Operating system and version
Amazon Linux 2

[mpdehaan]

Hi @AicyDC - it looks like I didn't actually try using the "variables" portion of the .vespene file - good catch and thanks for the report.

The code in question is https://github.com/vespene-io/vespene/blob/master/vespene/workers/importer.py

As you can see, it has no special value handling in 'attr_manage()' for the variables field, and it should.

What we should do is check if the type of any field is a dict, and if so, save the JSON version of that value.

What is happening is you are getting the stringification of the dictionary field, which is obviously no good.

I should be able to get to this in the next few days unless you want to have a go at it.

[mpdehaan]

essentially it just needs something like this:

if type(value) == dict:
    value = json.dumps(value)

[mpdehaan]

or better, probably:

value = json.dumps(value, indent=4)

(This preserves format because we are still storing those as text for slightly better DB compatibility, and because of some issues with the forms code and the JSON field class)

[tommy-couzens]

Thanks for the quick reply, unfortunately I do not know python very well so will not be having a go at it myself.

[mpdehaan]

No problem!

[mpdehaan]

Just wanted to say I haven't forgotten about this one. Various other things coming up but this should still be an easy change. Thanks for all the forum feedback!

[mpdehaan]

Apologies on losing track of this one, I've just pushed the two line fix mentioned above.

Thanks!