Results for FSE
Closed this issue · 4 comments
sibat119 commented
sibat119 commented
Crash report from Issue #13
Statistics
Out-of-Scope Modules: 45,
Interpreter-related Crashes: 88,
Inadequate Artifacts: 22,
Network: 11,
Playbook Semantics: 14
Confirm Bug: 8
Total: 188
Full Report
| Sample Num | Evaluation | Reason | Crash Type |
|---|---|---|---|
| 0 | Not a bug | Error comes from not finding a user-provided file name | Inadequate Artifacts |
| 1 | Confirmed a bug | field explicitly asks for int but byte string was provided |
N/A |
| 2 | Not a bug | playbook tries to reference undefined variable | Inadequate Artifacts |
| 3 | Not a bug | tries to access undefined host (ubuntu2) | Network |
| 4 | Not a bug | insufficient file privileges | Inadequate Artifacts |
| 5 | Not a bug | user-provided file name does not exist | Inadequate Artifacts |
| 6 | Confirmed a bug | - | N/A |
| 7 | Not a bug | user-provided file name does not exist | Inadequate Artifacts |
| 8 | Not a bug | user-provided file name does not exist | Inadequate Artifacts |
| 9 | May not apply | module documentation requires RESTCONF system service be enabled. This may be what is causing the error. |
- |
| 10 | Not a bug | user-provided file name does not exist | Inadequate Artifacts |
| 11 | Confirmed a bug | socket error | N/A |
| 12 | Not a bug | user-provided file name does not exist | Inadequate Artifacts |
| 13 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 14 | Not a bug | file not found error | Inadequate Artifacts |
| 15 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 16 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 17 | Not a bug | Failure message is like how it is shown in the documentation | - |
| 18 | Not a bug | http error | Network |
| 19 | Not a bug | http error | Network |
| 20 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 21 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 22 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 23 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 24 | Not a bug | previously encountered. requires cpanminus to be installed |
Interpreter-related Crashes |
| 25 | Not a bug | previously encountered. requires cpanminus to be installed |
Interpreter-related Crashes |
| 26 | Not a bug | user-provided file name does not exist | Inadequate Artifacts |
| 27 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 28 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 29 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 30 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 31 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 32 | Confirmed a bug | module is community.general.ejabberd_user | N/A |
| 33 | already reported | same as 32 | N/A |
| 34 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 35 | Maybe a bug | triggers traceback when pip cannot be found | Interpreter-related Crashes |
| 36 | Not a bug | user-provided file name does not exist | Inadequate Artifacts |
| 37 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 38 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 39 | Not a bug | user-provided file name does not exist | Inadequate Artifacts |
| 40 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 41 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 42 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 43 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 44 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 45 | leaning no but not sure | file not found error but could be due to a specified dependency not being installed | Inadequate Artifacts |
| 46 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 47 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 48 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 49 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 50 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 51 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 52 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 53 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 54 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 55 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 56 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 57 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 58 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 59 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 60 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 61 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 62 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 63 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 64 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 65 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 66 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 67 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 68 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 69 | Not a bug | Traceback triggered from read/write permissions | Inadequate Artifacts |
| 70 | Not a bug | Traceback triggered from read/write permissions | Inadequate Artifacts |
| 71 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 72 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 73 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 74 | Not a bug | user-provided file name does not exist | Inadequate Artifacts |
| 75 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 76 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 77 | Not a bug | graceful failure on http connection | Network |
| 78 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 79 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 80 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 81 | Maybe a bug | raises KeyError: 'body' but no argument with that name is used. software is not installed and cannot be tested due to it being proprietary. |
- |
| 82 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 83 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 84 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 85 | Maybe a bug | argument casing causes locale is not available error. this could be fixed by either ignoring case for checking or giving a better error message |
- |
| 86 | Maybe a bug | module fails catastrophically when locale-gen has not been run on the system first. this error could be avoided by checking for the file /etc/locale.gen first. instead, a vague error is given |
Inadequate Artifacts |
| 87 | need further inspection | module is made to be used for callback but does raise a weird error | - |
| 88 | Confirmed a bug | module error says to check stdout/stderr but could easily inform the user of the connection refused error generated from the traceback |
N/A |
| 89 | Maybe a bug | same module as 88 but error is different |
- |
| 90 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 91 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 92 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 93 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 94 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 95 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 96 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 97 | Maybe a bug | getting vague module error but reason could be that required software is not installed | Interpreter-related Crashes |
| 98 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 99 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 100 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 101 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 102 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 103 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 104 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 105 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 106 | need further inspection | weird socket-related error | Network |
| 107 | maybe a bug | module related to 106. connection refused error but that can only be seen from inspecting the stack trace |
Network |
| 108 | Not a bug | vague error message: JSONDecodeError. Testing the module further requires an api key |
- |
| 109 | Not a bug | error message:"MODULE FAILURE- See stdout/stderr for the exact error". hard-to-interpret traceback but further testing requires a subscription to OVH cloud services |
- |
| 110 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 111 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 112 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 113 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 114 | Not a bug | properly display connection failed message | Network |
| 115 | Maybe a bug | error relates to ansible.builtin.pip instead of the target module. error is arguably vague. |
Interpreter-related Crashes |
| 116 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 117 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 118 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 119 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 120 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 121 | Not a bug | file not found error | Inadequate Artifacts |
| 122 | Not a bug | file not found error | Inadequate Artifacts |
| 123 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 124 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 125 | Not a bug | reasonable error message | - |
| 126 | Maybe a bug | vague error but cannot test further without installing CA Spectrum software |
- |
| 127 | Not a bug | already determined to not be a bug | - |
| 128 | Not a bug | already determined to not be a bug | - |
| 129 | Not a bug | already determined to not be a bug | - |
| 130 | Not a bug | already determined to not be a bug | - |
| 131 | Maybe a bug | vague error message: JSONDecodeError. Testing the module further requires an api key |
- |
| 132 | Not a bug | proper database connection error message | Inadequate Artifacts |
| 133 | Not a bug | proper database connection error message | Inadequate Artifacts |
| 134 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 135 | Not a bug | tried to read a file that does not exist | Inadequate Artifacts |
| 136 | Not a bug | vague error message. stacktrace shows socket-related error | Network |
| 137 | Not a bug | same as 136 |
Network |
| 138 | Not a bug | same as 136 |
Network |
| 139 | Not a bug | same as 136 |
Network |
| 140 | skip | paid service | Out-of-Scope Modules |
| 141 | skip | paid service | Out-of-Scope Modules |
| 142 | skip | paid service | Out-of-Scope Modules |
| 143 | skip | paid service | Out-of-Scope Modules |
| 144 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 145 | Not a bug | gracefully failed on not having required python package installed | Interpreter-related Crashes |
| 146 | skip | cloud service software | Out-of-Scope Modules |
| 147 | skip | cloud service software | Out-of-Scope Modules |
| 148 | skip | cloud service software | Out-of-Scope Modules |
| 149 | skip | cloud service software | Out-of-Scope Modules |
| 150 | skip | cloud service software | Out-of-Scope Modules |
| 151 | skip | cloud service software | Out-of-Scope Modules |
| 152 | skip | cloud service software | Out-of-Scope Modules |
| 153 | skip | cloud service software | Out-of-Scope Modules |
| 154 | skip | cloud service software | Out-of-Scope Modules |
| 155 | skip | cloud service software | Out-of-Scope Modules |
| 156 | skip | cloud service software | Out-of-Scope Modules |
| 157 | skip | cloud service software | Out-of-Scope Modules |
| 158 | skip | cloud service software | Out-of-Scope Modules |
| 159 | skip | cloud service software | Out-of-Scope Modules |
| 160 | skip | cloud service software | Out-of-Scope Modules |
| 161 | Confirmed a bug | bigmon_chain module | N/A |
| 162 | skip | module is for huwawei cloud switches | Out-of-Scope Modules |
| 163 | skip | module is for huwawei cloud switches | Out-of-Scope Modules |
| 164 | skip | module targets lenovo CNOS devices | Out-of-Scope Modules |
| 165 | skip | module targets lenovo CNOS devices | Out-of-Scope Modules |
| 166 | skip | module targets lenovo CNOS devices | Out-of-Scope Modules |
| 167 | skip | module targets lenovo CNOS devices | Out-of-Scope Modules |
| 168 | skip | module targets lenovo CNOS devices | Out-of-Scope Modules |
| 169 | skip | module targets lenovo CNOS devices | Out-of-Scope Modules |
| 170 | skip | module targets lenovo CNOS devices | Out-of-Scope Modules |
| 171 | skip | module targets lenovo CNOS devices | Out-of-Scope Modules |
| 172 | skip | module targets cisco FTD devices | Out-of-Scope Modules |
| 173 | skip | module targets Itential Automation Platform | Out-of-Scope Modules |
| 174 | skip | module targets Ruckus ICX devices | Out-of-Scope Modules |
| 175 | skip | module targets Ruckus ICX devices | Out-of-Scope Modules |
| 176 | skip | module targets Ruckus ICX devices | Out-of-Scope Modules |
| 177 | skip | module targets Ruckus ICX devices | Out-of-Scope Modules |
| 178 | skip | module targets Cisco switches | Out-of-Scope Modules |
| 179 | skip | module targets virtual routers | Out-of-Scope Modules |
| 180 | skip | module targets virtual routers | Out-of-Scope Modules |
| 181 | skip | module targets virtual routers | Out-of-Scope Modules |
| 182 | skip | module targets virtual routers | Out-of-Scope Modules |
| 183 | skip | module targets virtual routers | Out-of-Scope Modules |
| 184 | skip | module targets virtual routers | Out-of-Scope Modules |
| 185 | skip | module targets virtual routers | Out-of-Scope Modules |
| 186 | skip | module targets virtual routers | Out-of-Scope Modules |
| 187 | skip | module targets virtual routers | Out-of-Scope Modules |
sibat119 commented
TELER based.
Out-of-Scope Modules: 0,
Interpreter-related Crashes: 3,
Inadequate Artifacts: 2,
Network: 0,
Playbook Semantics: 2,
Confirm Bug: 0
Total: 7
| index | id | title | comment | Crash Type |
|---|---|---|---|---|
| 0 | - | - | Not a bug. Module not installed in python interpreter hence causing crash | Interpreter-related Crashes |
| 1 | - | - | Not a bug. The code introduces divide by zero error. | Playbook Semantics |
| 2 | - | - | Not a bug. Proper Module is not installed. | Interpreter-related Crashes |
| 3 | - | - | Not a bug. The code gives non existent file path and invalid conditional check. | Inadequate Artifacts |
| 4 | - | - | Not a bug. Provides undefined variable. | Playbook Semantics |
| 5 | - | - | Not a bug. Setuptools Module not installed. | Interpreter-related Crashes |
| 6 | - | - | Not a bug. Invalid filepath given. | Inadequate Artifacts |
sibat119 commented
Latent Bugs from #12
Total Latent Bugs to Report in FSE: 9
| No. | Repo | issue ID | status | issue URL | Heuristic Category |
|---|---|---|---|---|---|
| 1 | Community General | 6949 | Fixed | community.general.ejabberd_user | Type |
| 2 | Ansible Core / Builtin | 81463 | Fixed | ansible.builtin.debug | Type |
| 3 | Ansible Core / Builtin | 81349 | Fixed | ansible.builtin.script | Type |
| 4 | Community General | 7207 | Fixed | community.general.blockinfile | State Inquiry |
| 5 | Community Network | 554 | In-Progress | community.network.bigmon_chain | Type |
| 6 | Ansible Netcommon | 582 | In-Progress | ansible.netcommon.telnet | Type |
| 7 | Community General | 7122 | In-Progress | community.general.mail | Type |
| 8 | Community General | 7121 | Rejected/Not-a-Bug | community.general.locale_gen | Type |
| 9 | Ansible Core / Builtin | 81647 | Rejected/Not-a-Bug | ansible.builtin.set_facts | Inventory |
The following bugs will not be reported in FSE.
| Repo | issue ID | status | issue URL |
|---|---|---|---|
| Community Docker | 681 | In-Progress (This issue was originated using Level 3 prompt so, we will not count this one) | community.docker.docker_image |
| Ansible Core / Builtin | 81767 | In-Progress (This issue is resubmission of the issue: 7207. As the same code and output was used in the bug report, we will not consider this as a latent bug.) | ansible.builtin.blockinfile |
| Ansible Core / Builtin | 81648 | Rejected/Not-a-Bug (This issue was originated using Level 3 prompt so, we will not count this one) | ansible.builtin.debug |
sibat119 commented
Non-Type Heuristics
Out-of-Scope Modules: 0,
Interpreter-related Crashes: 1,
Inadequate Artifacts: 6,
Network: 0,
Playbook Semantics: 14,
Confirm Bug: 2
Total: 23
Traceback Filtered for Lv2:
| Sample No. | Issue No. | Issue Title | Comment | Level | Evaluation | Crash Type |
|---|---|---|---|---|---|---|
| 2 | 37022 | `nxos_config idempotency issue with FEX on Nexus 5K | Ansible have command: shell: "echo {{ 1/0 }}" which throws divide by zero error. Not a Bug | 2 | Not a Bug | Playbook Semantics |
| 4 | 4709 | `lookup error exception in /jinja2/environment.py" for apt_key module since last weeks commits | Covered in conditional bug 4 | 2 | Latent Bug | N/A |
| 6 | 8552 | `ec2_vpc issues with boto 2.32.x | the code produces Divide by zero error. Not a Bug. | 2 | Not a Bug | Playbook Semantics |
| 16 | 22261 | `symlinks to bin/ansible with link name like 'ansible-3.6' cause errors | invalid template path given. Not a bug. | 2 | Not a Bug | Inadequate Artifacts |
| 20 | 37267 | `EC2 NoSuchVersion error when running elb_target_group or elb_application_lb modules | Not a Bug: provides non existent group name. | 2 | Not a Bug | Inadequate Artifacts |
| 23 | 42501 | `s3_bucket error with Minio - GetBucketVersioning NotImplemented | Not a bug: The playbook includes code that trigger divide by zero error. | 2 | Not a Bug | Playbook Semantics |
| 24 | 47371 | `SBC reboot time incorrectly parsed when TZ is set | Not a bug: The playbook includes code that trigger divide by zero error. | 2 | Not a Bug | Playbook Semantics |
| 28 | 59147 | `GCP Inventory plugin error when service_account_email is not set. | Not a Bug: The notebook have multiline content which it tries to save into a destination file. During the execution of this task the compiler produces PermissionError because the file already exists with another owner. | 2 | Not a Bug | Inadequate Artifacts |
| 31 | 7204 | `playbook hangs when --ask-sudo-pass not supplied and is required (vs error msg raised) [edited description] | Not a bug: uses apt for incorrect Linux distribution. | 2 | Not a Bug | Playbook Semantics |
| 37 | 15926 | `bug: included nested block with conditions results in unexpeced tasks execution | Not a Bug: set fact introduces a divide by zero error. | 2 | Not a Bug | Playbook Semantics |
| 38 | 19672 | `Ansible:Synchronize module throwing UNREACHABLE! error | Not a Bug: the playbook uses a non-existing file which throws AnsibleFileNotFound Exception. | 2 | Not a Bug | Inadequate Artifacts |
| 40 | 33693 | `Variable undefined error when import_role is used within a child play | Maybe a bug: list index out of range error. might not be a bug but i'm not sure how to read the jinja2 syntax here. | 2 | Not a Bug | **Playbook Semantics |
| 41 | 41129 | `npm module suddenly changed default behavior | Not a bug: reasonable error message: couldn't locate file | 2 | Not a Bug | Inadequate Artifacts |
| 46 | 73959 | ``uri` module fails with import error | Maybe a bug: exception encountered during another exception | 2 | Not a Bug | **Playbook Semantics |
| 47 | 5445 | Segmentation fault when running apt update-cache=yes` on OSX with 1.4.3 |
Not a bug: proper response from apt module | 2 | Not a Bug | Playbook Semantics |
| 48 | 7074 | `authorized_key module not reporting error when path to key is invalid | Not a bug: division by 0 but error message seems reasonable | 2 | Not a Bug | Playbook Semantics |
| 49 | 7322 | `apt_repository module error and usage of proxies when behind a firewall | Not a bug: proper response from apt module | 2 | Not a Bug | Playbook Semantics |
| 52 | 22119 | `hashi_vault returns Fatal error when the secret does not exist | Not a bug: issue and fix identified by error message | 2 | Not a Bug | Playbook Semantics |
| 58 | 41738 | `firewalld module does not permanently set an interface zone on CentOS / RedHat due to upstream bugs | Maybe a minor bug: error message shows permission denied when trying to write to a protected directory. it is also classified as an 'unhandled exception' which may be unnecessary | 2 | Not a Bug | Inadequate Artifacts |
| 64 | 68809 | `SSL issue running fortios_firewall_adress | Not a bug: tries to call command 'localhost' | 2 | Not a Bug | Playbook Semantics |
| 69 | 29529 | `dellos9_command ansible hangs after reload command issued to remote device. | Not a bug: division by 0 but error message seems reasonable | 2 | Not a Bug | Playbook Semantics |
| 73 | 36444 | `cloud:azure: error while using azure_rm_networkinterface for creating a vnic without public IP | Not a bug: failed module import | 2 | Not a Bug | Interpreter-related Crashes |
| 77 | 71444 | `iptables comment parameter incorrectly positioned in iptables | Maybe a bug: exception encountered during another exception | 2 | Latent Bug | N/A |