NMC3 Support [Need Testers]
Closed this issue · 24 comments
Hello,
I tried this for my NMC3 v3.0.0.12 card
PS D:\Downloads\apc-p15-tool_windows_amd64> ./apc-p15-tool install --keyfile my.org.key --certfile my.org.fullchain --apchost 192.168.1.120:22 --username apc --password 12345678 --fingerprint xxxx --debug
apc-p15-tool v0.4.2
install: making apc p15 file from pem
install: successfully loaded pem files
install: apc p15 file data succesfully generated
debug: ssh: remote server key fingerprint (b64): redacted
debug: ssh: remote server key fingerprint (hex): redacted
install: apc p15 file installed on 192.168.1.120:22
install: done
apc-p15-tool done
But even after rebooting the NMC interface it does not recognize the new cert.
Have you tried making the file and then uploading it manually? Does the NMC use the same wizard the nmc2 does?
Yes, I did try uploading manually but it did not work. And according to CLI docs
https://www.se.com/us/en/download/document/SPD_CCON-AYCELJ_EN/
it should work
Looking at that command and the others it sounds like the nmc3 preference is to upload the p15 key and then the cert separately. The tool combines both into one file.
Have you successfully used the security wizard tool to make a p15 key+cert file and upload it?
If you do in fact need to upload the p15 key and the cert separately I’d need to modify the tool to support that.
Please also ssh into the device and cd to the ‘ssl/‘ path and tell me the file names that are in there.
Have you successfully used the security wizard tool to make a p15 key+cert file and upload it?
I tried with Security Wizard 1.04. For private key, it is asking for a p15 key which I don't have so I am stuck there.
I have a wildcard certificate. Not sure if it will work for wildcard certs.
My cert files
Please also ssh into the device and cd to the ‘ssl/‘ path and tell me the file names that are in there.
I assume you’re using a self signed cert generated by the card. Can you send me that defaultcert.p15 file? I can compare against what the tool is doing.
Also try uploading the tool p15 via the web ui.
Also fyi the gui security wizard you posted is pretty old I think. You might want to try the CLI version. If you can’t get it working with the official tool I’m not sure there will be much I can do. You might need to post on the vendor support forum.
I tried to use my trusted LE issued wildcard certs with the tool. So no self signed certs. I cannot copy or read the "defaultcert.p15" file even with "apc" super user.
Are you using any extra Certificate Extensions (such as Must Staple)? You might also want to try without the wildcard (i.e. set the subject name / common name to your fqdn without wildcard and don't specify any other alt names).
At the end of the day, if you're unable to produce a working file with the official tools, I'm just going to be guessing what the problem is. Your best bet is to take the problem to the official Schneider Electric forums and ask for help with the official tools first. https://community.se.com/
If you get that working come back and I can compare the working file against what my tool generates and can find the difference(s).
I see, thanks for the information. Let me see, what is going to come out of official channel.
Good luck!
Hello
I confirm the problem.
Uploading the p15 file via SSH to the NMC3 is successful, but the NMC3 appears to be ignoring the file.
In NMC3 we need to download the certificate and private key file separately.
It would be nice if your utility had the ability to download such a thing.
Thank you.
Are you able to create a private key and cert file that work in your NMC3? I would need them to reverse engineer the files. This involves uploading a private key here, so certainly don't keep/trust the files on your actual device, but I do need to know the files actually work, for sure, on your device.
Additionally, please confirm if you need to restart the network interface after uploading the files, for them to actually go into use. If you do, please upload the two files to the NMC, but BEFORE you restart the interface, login via SSH and run cd ssl and then dir and let me know all of the file names returned. For example:
apc@apc>dir
E000: Success
0 Mar 25 2022 ./
0 Mar 25 2022 ../
2544 May 13 3:09 defaultcert.p15
I am guessing that perhaps the UPS firmware converts the two staged files into the defaultcert.p15 file on the device.
If you can do these things and are willing to test, I can try to add support.
Are you able to create a private key and cert file that work in your NMC3? I would need them to reverse engineer the files.
I think there is nothing magical in my key and certificate files that can be examined.
A few of my observations.
Using OpenSSL, I generated a regular private key (openssl.exe genrsa -out file.key 2048) and created a request to the local certificate authority (openssl.exe req -config my.cfg -new -key file.key.pem -out file. req). Then, based on the request file file.req, I received a certificate from the CA in Base-64 format. So I have 2 files in PEM format - file.key.pem and file.cert.pem
The good news is that using your utility I was able to create a p15 file (apc-p15-tool.exe create --keyfile file.key.pem --certfile file.cert.pem --outfile file.p15) and this the file is successfully received via the NMC3 web interface. NMC3 works with this p15 file.
The bad news is that I can't see where the downloaded p15 file ends up and what happens to it on the NMC3 storage
The general structure of NMC3 storage looks like this:
apc>dir
E000: Success
1024 May 12 2023 apc_hw21_aos_2.5.0.8.bin
6795900 May 12 2023 apc_hw21_su_2.5.0.6.bin
45000 Jun 2 13:12 config.ini
0 Oct 18 2021 db/
0 Oct 18 2021 ssl/
0 Oct 18 2021 ssh/
0 Oct 18 2021 logs/
0 Oct 18 2021 sec/
0 Oct 18 2021 fwl/
0 Oct 18 2021 email/
0 Oct 18 2021 eapol/
0 Oct 18 2021 tmp/
0 Oct 18 2021 upsfw/
0 Aug 1 2023 certs/
The ssl directory is empty:
apc>cd ssl
E000: Success
apc>dir
E000: Success
0 Oct 18 2021 ./
0 Oct 18 2021 ../
The certs directory and its subdirectories are empty:
apc>cd certs
E000: Success
apc>dir
E000: Success
0 Aug 1 2023 ./
0 Aug 1 2023 ../
0 Aug 1 2023 ca/
0 Aug 1 2023 private/
apc>cd private
E000: Success
apc>dir
E000: Success
0 Aug 1 2023 ./
0 Aug 1 2023 ../
apc>cd ../ca
E000: Success
apc>dir
E000: Success
0 Aug 1 2023 ./
0 Aug 1 2023 ../
But the installed certificate works.
After rebooting the NMC, nothing changes.
These directories remain empty.
The sec directory is also empty.
The only place where there is something similar to certificates is the db directory, there is some file keyset.p15
apc>cd db
E000: Success
apc>dir
E000: Success
0 Oct 18 2021 ./
0 Oct 18 2021 ../
70656 May 9 10:29 eventcfg2.db
30 Aug 1 2023 appid.dat
2048 Aug 1 2023 slcache.db
6112 Aug 1 2023 net4.db
104 Jun 2 12:25 hw.dat
6112 Aug 1 2023 fwver.db
53248 Jun 2 12:24 eventdef2.db
4081 Jun 2 12:19 keyset.p15
10208 Jun 2 13:24 cfg4.db
5088 Jun 2 12:17 trkeys.db
5088 Jun 2 13:15 user2.db
They must have changed something in the firmware since the defaultcert.p15 file doesn't even exist on your device, but it has an SSL cert. Are you able to download the keyset.p15 file? You can use the Putty SCP executable to try this: ./pscp user@ups.example.com:/db/keyset.p15 ./keyset.p15. I'm guessing it won't work because in other firmware I've tested they don't allow download of the sensitive .p15 files but it is worth a shot.
If you can't download that file and provide it I am at a loss. You could try uploading the .p15 file from my tool as keyset.p15, but I can't advise that. The file size of the keyset.p15 file is significantly larger than the defaultcert.p15 so there must be something else going on in that file. For what it is worth, my NMC2 doesn't have that file, so I'm guessing part of the file is the thing we're interested in + some amount of "other" stuff.
Finally, for NMC3 you might not even need my tool. See: https://community.se.com/t5/APC-UPS-Data-Center-Enterprise/SSL-Cert-for-NMC3-Web-Interface/td-p/459844 which seems to suggest you can upload certs using the native tooling.
I do see the instructions mentioning uploading a key in the .p15 format from the Security Wizard Tool. I can actually add the functionality to generate that file from a .pem key. I considered doing it originally, but it seemed unnecessary. I will work on that sometime soon, if it is helpful. Let me know.
Yes, I tried downloading this keyset.p15 file. And of course, without success :)
If you look at the document "Network Management Card 3 - CLI (Command Line Interface) Guide" https://www.se.com/us/en/download/document/SPD_CCON-AYCELJ_EN/ then there we will see the commands ssl key, ssl csr, ssl cert. We are interested in the commands ssl key -i and ssl cert - i, with the help of which, I think, we can try to upload our own certificate and key in p15 format to the NMC3.
I have several NMC3 boards in work and we could try to expand the capabilities of the apc-p15-tool utility
I have added the additional key.p15 output file when using the create command and adding the --keyp15 flag. Since I have no idea how the NMC3 processes the files, I can't make the install function work. You'll have to use the native SSH commands to install the created files, but it looks like it should be pretty reasonable to script.
Please try this build and let me know.
Thank you.
After several unsuccessful attempts, I found a working scheme.
So, the sequence of actions will be like this:
Convert the PEM (Base-64) private key to P15 format with specific APC headers:
> apc-p15-tool-amd64.exe create --keyfile D:\Certs\custom.key" \
--certfile "D:\Certs\custom.cer" --keyp15 --outkeyfile "D:\Certs\custom.key.p15"
apc-p15-tool v0.5.0
create: making apc p15 file from pem
create: successfully loaded pem files
create: apc p15 file data succesfully generated
create: apc p15 key+cert file apctool.p15 written to disk
create: apc p15 key file D:\Certs\custom.key.p15 written to disk
create: done
apc-p15-tool done
The command is executed successfully. A key file in a specific P15 format is created.
Please note here that an extra file "apctool.p15" is created. This is a logical error. In our case, such a file should not exist.
Next, using SCP we send files via SSH to NMC3.
Important note: when transferring a key file, we cannot use a name that contains p15 (for example *.p15 or *.p15.key). I noticed that NMC3 immediately deletes such a file. Therefore we will change the name of the final file.
> scp "D:\Certs\custom.key.p15" apc@nmc3.host.fqdn:/ssl/nmc.key
> scp "D:\Certs\custom.cer" apc@nmc3.host.fqdn:/ssl/nmc.crt
Now we connect via SSH to NMC3 and check the files.
We see that the files are in place:
apc>cd ssl
E000: Success
apc>dir
E000: Success
0 Jul 15 2021 ./
0 Jul 15 2021 ../
2016 Jun 5 8:22 nmc.crt
1420 Jun 5 8:25 nmc.key
In the current ssl directory, install the key and then the certificate:
apc>ssl key -i nmc.key
E000: Success
apc>ssl cert -i nmc.crt
E000: Success
Now we check that the certificate is installed and used as the current one:
apc>ssl cert -s
E000: Success
Certificate
-----------
Serial Number: 4a000....
Issuer: CN=Corp-Sub-CA1
Validity:
Not Before: Wed Jun 5 04:22:57 2024 UTC
Not After : Sat Jun 5 04:32:57 2025 UTC
Subject: CN=nmc3.host.fqdn, ...
Subject Public Key Info:
Public Key Algorithm: RSA (2048 bit)
Modulus:
bc:22:64:b8:...
... 2:29
Exponent: 65537 (0x10001)
Subject Alternative Name:
DNS: nmc3.host.fqdn
DNS: nmc3
IP Address: 10.10.5.2
Thumbprint: 1d461f...
Fingerprint: ed98cee...
apc>exit
There is no need to reboot the NMC3.
The certificate starts working immediately.
Thanks, this is very helpful. I can probably actually update the install function using this information.
Also, I'd left the certificate.p15 generating because I thought you'd need both files. I might just remove the key flag and always make it create both files. I don't know. We'll see :)
Give this one a shot. I think install should work now. Let me know how it goes.
Were you able to try preview 2?
Hello.
Does not work.
The process starts but does not end.
C:\Temp\v0.5.0-preview2>apc-p15-tool-amd64.exe install --keyfile "d:\certs\up006.holding.com.key" --certfile "d:\certs\up006.holding.com.cer" --apchost "up006.holding.com:22" --username "apc" --password "MyPW!0rd" --debug --fingerprint "7Z3baEepWWbGRZjq7iSshMEuUHzghnFx4HBVNdNX69g"
apc-p15-tool v0.5.0
install: making apc p15 file from pem
install: successfully loaded pem files
install: apc p15 file data succesfully generated
At this point the process hangs
I adjusted a little bit more (and found a typo in my regex). Please try attached preview 3.
If this doesn't work -- please SSH into your UPS (using MobaXterm, Putty, or whatever client you prefer) and then type ssl and push enter.
Then copy everything in the terminal up to that point and provide it here. I'm guessing something that works on my NMC2 isn't quite the same on NMC3. Please do not redact anything (there isn't anything sensitive) as I need to know the exact content. If you absolutely must change something please make it clear exactly what changed and how. For example, substitute example.com for your FQDN.
Here is an example of what I'm asking for (from my NMC2):
Schneider Electric Network Management Card AOS v7.1.2
(c) Copyright 2023 All Rights Reserved Smart-UPS & Matrix-UPS APP v7.1.2
-------------------------------------------------------------------------------
Name : apc9138A8 Date : 06/18/2024
Contact : Unknown Time : 10:32:26
Location : Unknown User : Super User
Up Time : 11 Days 21 Hours 52 Minutes Stat : P+ N4+ N6+ A+
-------------------------------------------------------------------------------
IPv4 : Enabled IPv6 : Enabled
Ping Response : Enabled
-------------------------------------------------------------------------------
HTTP : Disabled HTTPS : Enabled
FTP : Disabled Telnet : Disabled
SSH/SCP : Enabled SNMPv1 : Disabled
SNMPv3 : Disabled
-------------------------------------------------------------------------------
Super User : Enabled RADIUS : Disabled
Administrator : Disabled Device User : Disabled
Read-Only User : Disabled Network-Only User : Disabled
Type ? for command listing
Use tcpip command for IP address(-i), subnet(-s), and gateway(-g)
apc@apc>ssl
E101: Command Not Found
apc@apc>
It worked.
apc-p15-tool v0.5.0
install: making apc p15 file from pem
install: successfully loaded pem files
install: apc p15 file data succesfully generated
install: connected to ups ssh, installing ssl key and cert...
install: apc p15 file installed on up006.holding.com:22
apc-p15-tool done
On the NMC3 web page, after F5, the new certificate is used.
@Aleksey-Maksimov awesome! Thanks for confirming. I will include this in the next full release, likely in the coming days.
@Salvora can you test v0.5.0-preview3.zip with your NMC3 ?
@Aleksey-Maksimov awesome! Thanks for confirming. I will include this in the next full release, likely in the coming days.
@Salvora can you test v0.5.0-preview3.zip with your NMC3 ?
Currently, I am away from home and wil lbe back in August. I can test this only when I get back. I will get back to you regarding the results.