Migration between Overlays - Decrypt Issue
Closed this issue · 13 comments
Hey guys,
we run 3 stages in our environment.
Stage A = DEV, Stage B = Quality Insurance and Stage C = Productive Stage.
This means:
We deploy or develop new settings / features in the DEV stage. If completed we move the changes to Stage B and will test these settings with a testing plan. If the tests are successfully we will migrate the changes finally to the Stage C / PROD.
What we do today is:
We backup all settings with SASTRE on Stage A / DEV and restore the whole settings in Stage B / Quality Insurance.
During the restore of the settings we get always 2 errors in the feature templates AAA and SNMP.
The errors are always the same if we try to migrate the setting from Stage A to B.
javax.crypto.BadPaddingException: CiscoJCEJNI/Source/Block_Ciphers/Block_Cipher.cpp:do_evp_final: Bad ciphertext padding provided.: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt
If we stay within the same stage the issue isn't present.
I think that the issue is related to the decrypted mechanism inside the vManage DB and in my point of view the API nor SASTRE can fix this behavior but what SASTRE could do:
During the restore process SASTRE could ask the user to input the new password or something like that. What do you think about it?
Hi, would it be possible to share the AAA or SNMP template you're using along with the changes done between stage A and B? Also, please share the logs from Sastre as the problem happens. You can send them to sastre-support@cisco.com.
Hey, the settings between the stages are identically in both feature templates (AAA + SNMP). Only difference is the specific password for the users.
xy@server:/data/projects/sastre$ python3.9 sdwan.py -a [prod-stage.com](http://prod-stage.com/) -u api_user --port 443 --verbose restore all --update --regex "^Cloudcon" --workdir /data/projects/sastre/data/backup_qs-stage.com_20220406
vManage password:
INFO: Restore task: Local workdir: "/data/projects/sastre/data/backup_qs-stage.com_20220406" -> vManage URL: https://prod-stage.com/
INFO: Loading existing items from target vManage
INFO: Identifying items to be pushed
INFO: Inspecting template_device items
INFO: Inspecting template_feature items
INFO: Inspecting policy_vsmart items
INFO: Inspecting policy_vedge items
INFO: Inspecting policy_security items
INFO: Inspecting policy_voice items
INFO: Inspecting policy_customapp items
INFO: Inspecting policy_definition items
INFO: Inspecting policy_profile items
INFO: Inspecting policy_list items
INFO: Pushing items to vManage
INFO: Updating feature template Cloudconnect_AAA requires reattach of affected templates
INFO: Template attach: Cloudconnect_C1111_Primary (rnlcloudconnect01), Cloudconnect_C1111_Secondary (rnlcloudconnect03)
INFO: Reattaching templates
INFO: Waiting...
INFO: Waiting...
WARNING: Failed Cloudconnect_C1111_Primary, Cloudconnect_C1111_Secondary: rnlcloudconnect01: [6-Apr-2022 17:11:39 CEST] Configuring device with feature template: Cloudconnect_C1111_Primary, [6-Apr-2022 17:11:39 CEST] Checking and creating device in vManage, [6-Apr-2022 17:11:40 CEST] Generating configuration from template, [6-Apr-2022 17:11:45 CEST] Device is online, [6-Apr-2022 17:11:45 CEST] Updating device configuration in vManage, [6-Apr-2022 17:11:46 CEST] Sending configuration to device, [6-Apr-2022 17:11:54 CEST] javax.crypto.BadPaddingException: CiscoJCEJNI/Source/Block_Ciphers/Block_Cipher.cpp:do_evp_final: Bad ciphertext padding provided.: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt, rnlcloudconnect03: [6-Apr-2022 17:11:39 CEST] Configuring device with feature template: Cloudconnect_C1111_Secondary, [6-Apr-2022 17:11:39 CEST] Checking and creating device in vManage, [6-Apr-2022 17:11:40 CEST] Generating configuration from template, [6-Apr-2022 17:11:46 CEST] Device is online, [6-Apr-2022 17:11:46 CEST] Updating device configuration in vManage, [6-Apr-2022 17:11:46 CEST] Sending configuration to device, [6-Apr-2022 17:11:52 CEST] javax.crypto.BadPaddingException: CiscoJCEJNI/Source/Block_Ciphers/Block_Cipher.cpp:do_evp_final: Bad ciphertext padding provided.: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt
ERROR: Failed Update feature template Cloudconnect_AAA: Failed reattaching templates
INFO: Updating feature template Cloudconnect_SNMP requires reattach of affected templates
INFO: Template attach: Cloudconnect_C1111_Primary (rnlcloudconnect01), Cloudconnect_C1111_Secondary (rnlcloudconnect03)
INFO: Reattaching templates
INFO: Waiting...
INFO: Waiting...
WARNING: Failed Cloudconnect_C1111_Primary, Cloudconnect_C1111_Secondary: rnlcloudconnect01: [6-Apr-2022 17:12:02 CEST] Configuring device with feature template: Cloudconnect_C1111_Primary, [6-Apr-2022 17:12:02 CEST] Checking and creating device in vManage, [6-Apr-2022 17:12:03 CEST] Generating configuration from template, [6-Apr-2022 17:12:08 CEST] Device is online, [6-Apr-2022 17:12:08 CEST] Updating device configuration in vManage, [6-Apr-2022 17:12:09 CEST] Sending configuration to device, [6-Apr-2022 17:12:14 CEST] javax.crypto.BadPaddingException: CiscoJCEJNI/Source/Block_Ciphers/Block_Cipher.cpp:do_evp_final: Bad ciphertext padding provided.: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt, rnlcloudconnect03: [6-Apr-2022 17:12:02 CEST] Configuring device with feature template: Cloudconnect_C1111_Secondary, [6-Apr-2022 17:12:02 CEST] Checking and creating device in vManage, [6-Apr-2022 17:12:03 CEST] Generating configuration from template, [6-Apr-2022 17:12:08 CEST] Device is online, [6-Apr-2022 17:12:08 CEST] Updating device configuration in vManage, [6-Apr-2022 17:12:09 CEST] Sending configuration to device, [6-Apr-2022 17:12:15 CEST] javax.crypto.BadPaddingException: CiscoJCEJNI/Source/Block_Ciphers/Block_Cipher.cpp:do_evp_final: Bad ciphertext padding provided.: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt
ERROR: Failed Update feature template Cloudconnect_SNMP: Failed reattaching templates
INFO: Task completed with caveats: 2 errors, 2 warnings
Thank you, I'll try recreating this during the week to come up with some options to address the problem.
Hi, as I understand, you have particular templates Cloudconnect_AAA and Cloudconnect_SNMP, which have the same name between your environments but different contents (the passwords).
Assuming the above is correct, I think this is better addressed by maintaining different names for AAA and SNMP templates in stage A and B.
Keeping the templates between 2 environments with the same name and different payload is against what restore --update is meant for.
Another option could be to use the regular restore (without --update) and always rename the items that need to be updated (appending a version number to the name for instance).
hello Marcelo,
the point is that we like to synchronize all our configuration with SASTRE between the environments / overlays. That's why use the --update option.
For simplicity let's just focus on the SNMP template. The TACACS keys and all other SNMP settings are identical between the stages / overlays and we are hitting the decryption error here as well. For the AAA we use due security reasons individual passwords for each environment but please don't let us focus on the AAA template because as described in the previous sentence the issue is also visible in the SNMP template.
Hi, sorry for taking so long to circle back. Specially because of the SNMP template issue, I suspect this is a bug on vManage. I couldn't replicate that problem in my environment. If you send me the specific vManage release you're running as well as your SNMP template I can try replicating it.
hello @reismarcelo!
Excuse my late response.
We are using vManage 20.6.3.
We are using the following SNMP configuration in the feature template:
- Shutdown = No
- Contact Person = Custom
- Location of Device = Variable
- SNMP Version = 2
- View = view_v1
- Object Identifier = 1.3.6.1
- Community
- Name = XYZ
- Authorization = read-only
- View = view_v1
Could you please reopen the issue and try to reproduce it in your environment?
Thanks in advance
filip
Thank you for the info Filip, I'll try to recreate it in the next few days and get back.
Hi Filip, I was able to recreate the issue on my environment, thanks for the additional info. I'll now investigate how the GUI generates the encrypted value (for community name in the example you gave) to see if there's a way we can automatically make the conversion.
Hi, any progress in the issue?
We have someone working on it for the past couple of weeks, should have a PR soon. Targeting it for 1.23, which will likely be for around September timeframe
To be clear, during import from vManage 20.9.3 to 20.9.3 the import itself is successful, but during attach of the template:
[23-Jun-2023 9:18:37 UTC] Configuring device with feature template: TEMPLATE1
[23-Jun-2023 9:18:38 UTC] Checking and creating device in vManage
[23-Jun-2023 9:18:39 UTC] Generating configuration from template
[23-Jun-2023 9:19:02 UTC] Device is online
[23-Jun-2023 9:19:02 UTC] Updating device configuration in vManage
[23-Jun-2023 9:19:02 UTC] Sending configuration to device
[23-Jun-2023 9:19:09 UTC] Error encountered during the decryption, please update below fields in templates
Cisco AAA: Tacacs server shared key
Cisco AAA: Tacacs server shared key
Solution: remove tacacs' servers and group, then add them once again with the same passwords.
Support for this was added in 1.23 via encrypt and transform tasks.