Webapp gets incomplete data from the MariaDB for Jenkins

[ian-jarrett]

Tried to build a simple syslog config using this JSON data:

{"syslog": [{"loghost": "100.123.0.16","facility": "daemon","level":"info"},{"loghost": "100.123.0.16","facility": "any","level": "warning"}]}

Run write_yaml_files.py from within the jenkins container to generate ./group_vars/spines.yaml:

syslog:
- loghost: 100.123.0.16
  facility: daemon
  level: info
- loghost: 100.123.0.16
  facility: any
  level: warning

And use this J2 template to build the Junos config:

system {
  syslog {
    {% for host in syslog %}
      host {{ host.loghost }} {
        {{ host.facility }} {{ host.level }};
      }
    {% endfor %}
  }
}

If running from the shell, the build process works correctly. If running from the Webapp, Ansible fails with this error:

AnsibleUndefinedVariable: 'ansible.parsing.yaml.objects.AnsibleUnicode object' has no attribute 'loghost'"

This error is because the YAML file was not fully built by the "Build" job. However, the Excel data is all correctly stored in the "ngcn_worksheets" table inside the "Sites" database (running in the MariaDB container). So somehow, this data is not getting extracted correctly by the webapp when it runs the Jenkins job.

[wildsubnet]

Confirming issue. MariaDB has both rows of data but the data.json that is generated when the ansible-playbook is run by jenkins only gets the last row of data.

Tested against ansible 2.10 as well just in case it is some oddball issue with ansible. Same result. Saw same ansible error reported above, but in both webapp and via container shell.

This issue is not unique to the specific use case Ian raised. I was able duplicate it with "management interfaces" tab as well by adding both em0 and em1 interfaces to a spine switch. In that case, em1 was only exposed to jenkins.

[hribeiro-jnpr]

Yes that was my understanding as well, pretty sure the issue is around views.py where the webapp gets the info from mariadb and generates the data.json.

[wildsubnet]

Confirmed based on Debug info that data is being truncated in views.py code. Output in log shows data set at line 207 but my 924 the extra rows are gone. Querying Maria DB directly using the same selector/order by as the python code returns the data as expected. So something from the data retrieval into json file is losing fidelity.

line 922 calls the function to create temp.xlsx. This file seems correct to me. Has the data as expected in temp.xlsx.
line 923 converts this temp file to yaml. This is almost certainly where it goes south but I believe this to be logical error rather than an outright programming bug. The dictionary key must be unique. However, the logic is building the dictionary keys to be the sheet name PER device name. For example, if I assign the data as follows with two syslog entries

foo = ('host_vars/dc1-spine1.yaml', ('syslog', OrderedDict([('loghost', '100.123.0.17'), ('facility', 'daemon'), ('level', 'info')])), ('syslog', OrderedDict([('loghost', '100.123.0.17'), ('facility', 'any'), ('level', 'warning')])))

Python only stores the second entry in memory:

('host_vars/dc1-spine1.yaml', ('syslog', OrderedDict([('loghost', '100.123.0.17'), ('facility', 'daemon'), ('level', 'info')])), ('syslog', OrderedDict([('loghost', '100.123.0.17'), ('facility', 'any'), ('level', 'warning')])))

I think this is what is happening here. I say logic error because I do not believe this can work as intended and needs to be explored further as a change to how we process these sheets when specific hosts have "duplicate" entries.

[wildsubnet]

Potential workaround for now...for multi-line entries like syslog you can create additional columns in the syslog tab. I did facility2 and level2. Updated the template to use both:

	syslog {
		host {{ syslog.loghost }} {
			{{ syslog.facility }} {{ syslog.level }};
		}
		host {{ syslog.loghost }} {
			{{ syslog.facility2 }} {{ syslog.level2 }};
		}
	}

This generated the proper yaml file :

---
OS_version: 18.1R3-S5.3
healthbot_device_group: dc1-spines
loopback_ip: 10.30.100.3
management_interface:
  int: em0
  ip: 100.123.24.3
  mask: 16
syslog:
  facility: any
  facility2: daemon
  level: warning
  level2: info
  loghost: 100.123.0.17

And Jenkins ran the job and generated junos.conf file properly.

This is less clunky than the first workaround I proposed earlier (and since removed). Still clunky but I believe this should become an enhancement request rather than a bug as we will need to do some work on how the data retrieved and stored in memory to make the use case as specified above to work properly.

[ian-jarrett]

Thanks Matt, I've tried your workaround and I can confirm that it works, thank you very much. We can caveat this behaviour in the docs for now and move on. Whether it's an ER or a bug I don't know, I'll defer to you since you spent longer looking into the code than I did :-)

[aburston]

Doing a basic test on the example data definitely shows that this problem is not caused by yaml2xls:

test.yaml

syslog:
- loghost: 100.123.0.16
  facility: daemon
  level: info
- loghost: 100.123.0.16
  facility: any
  level: warning

yaml2xls.py test.yaml test.xlsx produces a file test.xlsx

You can then you can convert this file back to yaml by doing:
mkdir tmp && xls2yaml.py test.xlsx tmp

and that produces the following:
tmp/test.yaml

---
syslog:
- loghost: 100.123.0.16
  facility: daemon
  level: info
- loghost: 100.123.0.16
  facility: any
  level: warning

Which has a slightly different start but nothing important has changed.

[wildsubnet]

This seems to be resolved by an undocumented feature in NITA. If we change the column header on syslog to be host+.facility and host+.level the views.py subroutines use ordered_dicts as fields instead of strings. So we go from this:

('syslog', OrderedDict([('loghost', '100.123.0.17'), ('facility', 'daemon'), ('level', 'info')]))])),

To this:

('syslog', OrderedDict([('loghost', '100.123.0.17'), ('host', [OrderedDict([('facility', 'any'), ('level', 'warning')]), OrderedDict([('facility', 'daemon'), ('level', 'info')])])]))])),

Then after that I believe it should just be updating the j2 template to iterate through each element like the evpn_port template does in the examples. If you look at the example files for the data center use case evpn_port tab there is "+" on the tab name too. I assume it’s doing some sort of larger grouping that may not matter for the syslog example (I’m almost sure that’s it honestly). However, this worked without the tab name being "syslog+". Something to explore further while working on developer's guide.

[aburston]

This sounds like a documentation bug against nita-yaml-to-excel