Juniper/jsnapy

The module results from a test run do not link test names to test results only test name pass or fail

jnpr-bowen opened this issue · 4 comments

jnpr-bowen commented

Description of Issue/Question

The Operator class returned by a test run only associates a test name to a pass or fail value. The test_details/test_results dictionaries are keyed to the command/rpc and do not refer to the test name that uses them.
Using jsnapy as a module does not allow you to produce an output like the command line does that prints the test name, the command/rpc and then the info or err messages tied to a test result.
The Operator class needs a dictionary that associates the test name with the results of the test so that some thing like Ansible can use it and provide output that ties info and err messages tied to test names.

Setup

Any test run from the python module will show that the Operator class does not have any association of test name to test details.

Steps to Reproduce Issue

Versions Report

1.3.2

jnpr-bowen commented

Found the problem. Line 44 of operator.py file breaks connection of jsnapy test name to the results stored in test_details.
del(test['test_name'])
If you take that line out, you can write a script that will be able to loop through all the test results by test name and present any of the err and info messages that are defined in a test. I took the sample script in the "admin guide" https://www.juniper.net/documentation/en_US/junos-snapshot1.0/topics/example/automation-junos-snapshot-python-module-example.html and added the following to test this:

print("Testing result details:")
for res in snapchk:
    print(colorama.Fore.BLUE + "****** Device: " + res.device + " ********")
    print("Tests passed: {} Tests Failed: {}".format(res.no_passed, res.no_failed))
    for test_name, test_list in res.testname_results.items():  # test_list should be a list of 1 of rpc/cmmd
        print("Test details for: " + test_name)
        for test in test_list:
            print (colorama.Fore.BLUE + "******** Command or RPC " + test['command'] + " ********")
            print ("\tTest info messages:")
            for details in test['passed']:
                print (colorama.Fore.GREEN + "\t\t" + details['message'])
            print ("\tTest error messages:")
            for details in test['failed']:
                print (colorama.Fore.RED + "\t\t" + details['message'])
sidhujasminder commented

@jnpr-bowen https://github.com/Juniper/jsnapy/wiki/4.-Module, I think the example 6 on this page might be helpful. Please check if this helps.

jnpr-bowen commented
chidanandpujar commented

Hi @jnpr-bowen
After commenting out the del(test['test_name']), able to print the results multiple times.

cat mult_tests.py 
from jnpr.jsnapy import SnapAdmin
from pprint import pprint
from jnpr.junos import Device
import colorama

config_data = """
hosts:
  - device: x.x.x.x 
    username : xyz
    port : 22
    passwd: xyz
tests:
  - test_show_chassis_hardware_no_diff_2.yml 
"""

dev = Device(host='x.x.x.x', user='xyz', password='xyz', port=22)
dev.open()

js = SnapAdmin()
js.snap(config_data, "pre", dev)

js.snap(config_data, "post", dev)

snapchk = js.check(config_data, "pre", "post", dev)

print("Testing result details:")
for res in snapchk:
    print(colorama.Fore.BLUE + "****** Device: " + res.device + " ********")
    print("Tests passed: {} Tests Failed: {}".format(res.no_passed, res.no_failed))
    for test_name, test_list in res.testname_results.items():  # test_list should be a list of 1 of rpc/cmmd
        print("Test details for: " + test_name)
        for test in test_list:
            print (colorama.Fore.BLUE + "******** Command or RPC " + test['command'] + " ********")
            print ("\tTest info messages:")
            for details in test['passed']:
                print (colorama.Fore.GREEN + "\t\t" + details['message'])
            print ("\tTest error messages:")
            for details in test['failed']:
                print (colorama.Fore.RED + "\t\t" + details['message'])


print("Testing result details:")
for res in snapchk:
    print(colorama.Fore.BLUE + "****** Device: " + res.device + " ********")
    print("Tests passed: {} Tests Failed: {}".format(res.no_passed, res.no_failed))
    for test_name, test_list in res.testname_results.items():  # test_list should be a list of 1 of rpc/cmmd
        print("Test details for: " + test_name)
        for test in test_list:
            print (colorama.Fore.BLUE + "******** Command or RPC " + test['command'] + " ********")
            print ("\tTest info messages:")
            for details in test['passed']:
                print (colorama.Fore.GREEN + "\t\t" + details['message'])
            print ("\tTest error messages:")
            for details in test['failed']:
                print (colorama.Fore.RED + "\t\t" + details['message'])


:~/ansible_release_v105_final/jsnapy/tests# python mult_tests.py 
Taking snapshot of COMMAND: show chassis hardware 
Taking snapshot of COMMAND: show chassis hardware 
**************************** Device: 10.220.24.28 ****************************
Tests Included: test_chassis_hardware 
*********************** Command: show chassis hardware ***********************
PASS | All "description" exists at xpath "//chassis/chassis-module" [ 4 value matched ]
PASS | INFO: chassis description for {{ id_0 }} is {{ pre['description'] }}
PASS | All "serial-number" is same in pre and post snapshot [ 4 value matched ]
------------------------------- Final Result!! -------------------------------
test_chassis_hardware : Passed
Total No of tests passed: 2
Total No of tests failed: 0 
Overall Tests passed!!! 
Testing result details:
****** Device: 10.220.24.28 ********
Tests passed: 2 Tests Failed: 0
Test details for: test_chassis_hardware
******** Command or RPC show chassis hardware ********
	Test info messages:
		INFO: chassis description for Routing Engine 0 is Control Board
		INFO: chassis description for FPC 1 is FPC-JNP10003-LOGICAL
		INFO: chassis description for SIB 0 is SIB-JNP10003
		INFO: chassis description for SIB 1 is SIB-JNP10003
	Test error messages:
******** Command or RPC show chassis hardware ********
	Test info messages:
		INFO: chassis serial-number same for ['SIB 1']
		INFO: chassis serial-number same for ['SIB 0']
		INFO: chassis serial-number same for ['FPC 1']
		INFO: chassis serial-number same for ['Routing Engine 0']
	Test error messages:
Testing result details:
****** Device: 10.220.24.28 ********
Tests passed: 2 Tests Failed: 0
Test details for: test_chassis_hardware
******** Command or RPC show chassis hardware ********
	Test info messages:
		INFO: chassis description for Routing Engine 0 is Control Board
		INFO: chassis description for FPC 1 is FPC-JNP10003-LOGICAL
		INFO: chassis description for SIB 0 is SIB-JNP10003
		INFO: chassis description for SIB 1 is SIB-JNP10003
	Test error messages:
******** Command or RPC show chassis hardware ********
	Test info messages:
		INFO: chassis serial-number same for ['SIB 1']
		INFO: chassis serial-number same for ['SIB 0']
		INFO: chassis serial-number same for ['FPC 1']
		INFO: chassis serial-number same for ['Routing Engine 0']
	Test error messages:

Thanks
Chidanand