Edit dtb to allow eSATA volumes
Opened this issue · 20 comments
Hi @007revad,
I appreciate a lot what you have done for Synology community and this script is another example of hard work. Thank you very much for your effort.
I noticed the script downloads and runs /bin/dtc. Would be grateful if you could describe at least in the readme file what /bin/dtc is, where it comes from. Even better, if you could share its source code, if you are the author.
I don't have any official Synology expansion unit, so I can't actually test your script. However, I have 2 SATA SSDs connected to my DS1522+ using eSATA cables. These is a storage volume inside them (created by means of putting these disks inside the device), and it actually works great, if I leave 2 problems aside: 1) each time the device reboots, it requires me to manually press on assemble the volume (storage pool) button in the storage manager; 2) the disks aren't listed in the storage manager, so I have to either use command line tools to manage them, or remove some internal disks and put the external ones inside.
Given you understand how DSM gathers information about external units and know how to make changes to the device model (dtb), my feature request is to allow eSATA volumes on directly connected HDDs/SSDs (i.e. make DSM think these disks are internal ones). What do you think about my idea?
Regards,
dtc is a device tree compiler from Linux. It's used to decompile and compile device tree blobs.
I got the dtc binary from one of these repos, which all have the exact same dtc binary:
- https://github.com/pocopico/tcrp-addons/tree/main/dtbpatch/releases
- https://github.com/PeterSuh-Q3/tcrp-addons/tree/main/dtbpatch/releases
- https://github.com/jumkey/redpill-load/tree/develop/redpill-dtb/releases
I've just asked pocopico and PeterSuh-Q3 about the scource code for the dtc binary. I have searched the Xpenology forum but searching for 'dtc source' found too many irrelevant results.
There is source code available from https://git.kernel.org/pub/scm/utils/dtc/dtc.git for the v1.6.1 used by a few of my scripts. It's been decades since I last compiled a binary file from C source so when I found the pre-compiled binary on the xpenology boot loader repos, and it worked for what I needed, I just used it. If you're able to compile dtc v1.6.1 from the source here I'd love to know if it creates the same file.
Let me know if you find the source for dtc as I'd like to add it my repo, or at least a link to the source.
Do you have 2 eSATA drives or 1 eSATA device that contains 2 SSD drives?
I do intend updating this script to make any eSATA drive supported, and show up in storage manager.
dtc is a device tree compiler from Linux. It's used to decompile and compile device tree blobs.
Thank you for clarification. Curious why Synology decided to use (a part of?) this tool under the hood of DSM. At least on my DS1522+ with 7.2.1-69057 Update 4 this binary is not found.
There is source code available from https://git.kernel.org/pub/scm/utils/dtc/dtc.git for the v1.6.1 used by a few of my scripts.
It's been decades since I last compiled a binary file from C source so when I found the pre-compiled binary on the xpenology boot loader repos, and it worked for what I needed, I just used it. If you're able to compile dtc v1.6.1 from the source here I'd love to know if it creates the same file.
After a quick research I've found the dtc repository which seems to be actively maintained. It visually has all the necessary files to be compiled and installed using standard developer tools. But I haven't checked it myself.
DSM doesn't include dtc. That's why this script (and a couple of my other scripts) include dtc (and download dtc if it's not installed and not with the script).
Do you have 2 eSATA drives or 1 eSATA device that contains 2 SSD drives?
I have 2 SATA drives (Samsung SSD 870 EVO). Each one is connected to DS1522+ using a SATA to eSATA cable with USB power like this. Both are visible from DSM console just like internal devices (i.e. /dev/sataX, where X is 6 and 7 for eSATA-connected drives in my case), but not listed by synodisk --enum.
About a year ago I tested a dual-bay 2.5" to 3.5" SATA adapter with my DS1522+. It supports a JBOD mode, but I didn't manage to make my Synology device show more than 1 drive either due to a lack of port multiplier support in the SATA controller, or due to Synology checking some additional info about one-to-many adapters.
Additional debug info that might be useful for understanding:
Linux SATA ports mapping
ash-4.4# ls -al /sys/block/sata*
lrwxrwxrwx 1 root root 0 Mar 26 21:57 /sys/block/sata1 -> ../devices/pci0000:00/0000:00:01.5/0000:0a:00.0/ata1/host0/target0:0:0/0:0:0:0/block/sata1
lrwxrwxrwx 1 root root 0 Mar 26 21:57 /sys/block/sata2 -> ../devices/pci0000:00/0000:00:01.5/0000:0a:00.0/ata2/host1/target1:0:0/1:0:0:0/block/sata2
lrwxrwxrwx 1 root root 0 Mar 26 21:57 /sys/block/sata3 -> ../devices/pci0000:00/0000:00:01.5/0000:0a:00.0/ata3/host2/target2:0:0/2:0:0:0/block/sata3
lrwxrwxrwx 1 root root 0 Mar 26 21:57 /sys/block/sata4 -> ../devices/pci0000:00/0000:00:08.2/0000:0c:00.0/ata6/host5/target5:0:0/5:0:0:0/block/sata4
lrwxrwxrwx 1 root root 0 Mar 26 21:57 /sys/block/sata5 -> ../devices/pci0000:00/0000:00:08.2/0000:0c:00.0/ata7/host6/target6:0:0/6:0:0:0/block/sata5
lrwxrwxrwx 1 root root 0 Mar 26 21:57 /sys/block/sata6 -> ../devices/pci0000:00/0000:00:01.5/0000:0a:00.0/ata5/host4/target4:0:0/4:0:0:0/block/sata6
lrwxrwxrwx 1 root root 0 Mar 26 21:57 /sys/block/sata7 -> ../devices/pci0000:00/0000:00:01.5/0000:0a:00.0/ata4/host3/target3:0:0/3:0:0:0/block/sata7
Synology slot mapping
ash-4.4# syno_slot_mapping System Disk Internal Disk 01: /dev/sata3 02: /dev/sata2 03: /dev/sata1 04: /dev/sata4 05: /dev/sata5Esata port count: 2
Esata port 1
01: /dev/sata7Esata port 2
01: /dev/sata6USB Device
01:
02: /dev/usb1
Internal SSD Cache:
01: /dev/nvme0n1
02: /dev/nvme1n1
model.dts (generated by dtc -q -I dtb -O dts -o ./model.dts /etc.defaults/model.dtb)
/dts-v1/;/ {
compatible = "Synology";
model = "synology_r1000_1522+";
version = <0x01>;
syno_spinup_group = <0x02 0x01 0x01 0x01>;
syno_spinup_group_delay = <0x0b>;
syno_hdd_powerup_seq = "true";
syno_cmos_reg_secure_flash = <0xe0>;
syno_cmos_reg_secure_boot = <0xe2>;
power_limit = "14.85,14.85";DX517 { compatible = "Synology"; model = "synology_dx517"; pmp_slot@1 { libata { EMID = <0x00>; pmp_link = <0x00>; }; }; pmp_slot@2 { libata { EMID = <0x00>; pmp_link = <0x01>; }; }; pmp_slot@3 { libata { EMID = <0x00>; pmp_link = <0x02>; }; }; pmp_slot@4 { libata { EMID = <0x00>; pmp_link = <0x03>; }; }; pmp_slot@5 { libata { EMID = <0x00>; pmp_link = <0x04>; }; }; }; internal_slot@1 { protocol_type = "sata"; power_pin_gpio = <0x04 0x00>; detect_pin_gpio = <0x0a 0x01>; led_type = "atmega1608"; ahci { pcie_root = "00:01.5,00.0"; ata_port = <0x02>; signal_data_gen2 = <0x1e5>; signal_data_gen3 = <0x7e5>; set_ssc_off; }; led_green { led_name = "syno_led0"; }; led_orange { led_name = "syno_led1"; }; }; internal_slot@2 { protocol_type = "sata"; power_pin_gpio = <0x05 0x00>; detect_pin_gpio = <0x0b 0x01>; led_type = "atmega1608"; ahci { pcie_root = "00:01.5,00.0"; ata_port = <0x01>; signal_data_gen2 = <0x1e4>; signal_data_gen3 = <0x3e5>; set_ssc_off; }; led_green { led_name = "syno_led2"; }; led_orange { led_name = "syno_led3"; }; }; internal_slot@3 { protocol_type = "sata"; power_pin_gpio = <0x08 0x00>; detect_pin_gpio = <0x0d 0x01>; led_type = "atmega1608"; ahci { pcie_root = "00:01.5,00.0"; ata_port = <0x00>; signal_data_gen2 = <0x1e4>; signal_data_gen3 = <0x3e5>; set_ssc_off; }; led_green { led_name = "syno_led4"; }; led_orange { led_name = "syno_led5"; }; }; internal_slot@4 { protocol_type = "sata"; power_pin_gpio = <0x09 0x00>; detect_pin_gpio = <0x0e 0x01>; led_type = "atmega1608"; ahci { pcie_root = "00:08.2,00.0"; ata_port = <0x00>; }; led_green { led_name = "syno_led6"; }; led_orange { led_name = "syno_led7"; }; }; internal_slot@5 { protocol_type = "sata"; power_pin_gpio = <0x5b 0x00>; detect_pin_gpio = <0x10 0x01>; led_type = "atmega1608"; ahci { pcie_root = "00:08.2,00.0"; ata_port = <0x01>; }; led_green { led_name = "syno_led8"; }; led_orange { led_name = "syno_led9"; }; }; esata_port@1 { ahci { pcie_root = "00:01.5,00.0"; ata_port = <0x03>; signal_data_gen2 = <0x1e4>; signal_data_gen3 = <0x1ef>; set_ssc_off; }; }; esata_port@2 { ahci { pcie_root = "00:01.5,00.0"; ata_port = <0x04>; signal_data_gen2 = <0x1e5>; signal_data_gen3 = <0x3ef>; set_ssc_off; }; }; usb_slot@1 { vbus { syno_gpio = <0x2a 0x01>; }; usb2 { usb_port = "1-2"; }; usb3 { usb_port = "2-2"; }; }; usb_slot@2 { vbus { syno_gpio = <0x28 0x01>; }; usb2 { usb_port = "1-3"; }; usb3 { usb_port = "2-3"; }; }; i2c_bus@0 { acpi_hid = "AMDI0010"; acpi_uid = "2"; }; i2c_bus@1 { acpi_hid = "AMDI0010"; acpi_uid = "3"; i2c_device@1 { i2c_device_name = "rtc-s35390a"; i2c_address = "0x30"; }; i2c_device@2 { i2c_device_name = "atmega1608"; i2c_address = "0x60"; }; }; nvme_slot@1 { pcie_root = "00:01.2"; port_type = "ssdcache"; }; nvme_slot@2 { pcie_root = "00:01.3"; port_type = "ssdcache"; };
};
I wish I could get these for $1.50. Prices here range from $10 to $80!?!?! for the same cable.
The Synology Does the eSATA port of Synology products support eSATA disk enclosures with port multipliers? page says:
All Synology products do not support eSATA disk enclosures with port multipliers. The installed drives on eSATA disk enclosures with port multipliers may not be recognized by Synology products. If you would like to use external disk enclosures with multiple drives, we recommend you to use Synology Expansion Units.
So it sounds like if you can find a eSATA multi-drive enclosure that has the same port multiplier chip as the DX and RX expansion units it might work.
for the same cable.
I ordered two cables looking the same on Ali.
So it sounds like if you can find a eSATA multi-drive enclosure that has the same port multiplier chip as the DX and RX expansion units it might work.
It is not only the chip that matters, but also its firmware. As far as I remember, last time after many hours of googling I bumped into a small piece of info that DSM validates some data inside eSATA multi-drive enclosure in order to consider it original. And the author of that message didn't manage to make it work as intended.
But my idea in this issue was actually not about non-original multi-drive eSATA enclosures, but about directly connected eSATA drives. I suggest you could extend the enable_eunit script functionality to make external drives be shown/treated by DSM as internal ones enabling users to use spare eSATA ports as slots for connecting additional drives.
Is it safe to replace
esata_port@1 {
ahci {
pcie_root = "00:01.5,00.0";
ata_port = <0x03>;
signal_data_gen2 = <0x1e4>;
signal_data_gen3 = <0x1ef>;
set_ssc_off;
};
};
esata_port@2 {
ahci {
pcie_root = "00:01.5,00.0";
ata_port = <0x04>;
signal_data_gen2 = <0x1e5>;
signal_data_gen3 = <0x3ef>;
set_ssc_off;
};
};
with
internal_slot@6 {
protocol_type = "sata";
ahci {
pcie_root = "00:01.5,00.0";
ata_port = <0x03>;
signal_data_gen2 = <0x1e4>;
signal_data_gen3 = <0x1ef>;
set_ssc_off;
};
};
internal_slot@7 {
protocol_type = "sata";
ahci {
pcie_root = "00:01.5,00.0";
ata_port = <0x04>;
signal_data_gen2 = <0x1e5>;
signal_data_gen3 = <0x3ef>;
set_ssc_off;
};
};
inside model.dts, than recompile model.dtb and copy it to /etc.defaults/model.dtb? Is it probable that I will brick my device? If bricked, what options are there to restore it?
I suppose maxdisks="5" should also be replaced with maxdisks="7" in /etc.defaults/synoinfo.conf.
In case you are interested in a non-original eSATA port multipliers topic, I think it was a small piece of info on Reddit by mchamst3r that I mentioned above:
When I did the research on this, I found that synology actually does support eSATA port multipliers but they vendor locked it to their expansion models.
The port multiplier chip they use has an external 24c series eeprom with their vendor code in a specific address. All that one needs to do is read the vendor code from the eeprom and then write it to the same eeprom on the port multipliers you can find on eBay for like $29. I figured this out by looking at the circuit boards from tear down photos. If you already had the expansion card, it would be like $0.25 in parts.
Is it safe to replace
No. Don't try that. You may brick the Synology.
EDIT Actually you would not brick the Synology, But you may lose your storage pool and volume(s).
That comment on reddit is interesting.
But I'd do it a different way so anyone can can connect any eSATA DAS and have it appear to DSM as if it is a Synology expansion unit.
The would require a script that:
- Edits whatever files check if the eSATA device is a Synology expansion unit or not so DSM thinks all eSATA devices are Synology expansion units.
- Gets the vendor id, brand, device id and model name from the eSATA DAS.
- Adds them as supported in model.dtb and any other related files.
We may be limited to DAS with up to 5 drive bays.
It would probably only required editing synoinfo.conf, model.dtb and storage_panel.js
The part I'm not sure of how to do is step 3, which I'd need to edit model.dtb like this:
DX517 {
compatible = "Synology";
model = "synology_dx517";
I actually have a DX213 so I should be able to experiment on getting the information from the DX213.
No. Don't try that. You may brick the Synology.
EDIT Actually you would not brick the Synology, But you may lose your storage pool and volume(s).
I found 2 spare SATA drives for testing and pulled out all my production drives (internal M2, internal and external SATA) so that not to loose any data/settings in case I bricked the device. Then I connected the first SATA drive to an internal slot and installed a clean version of DSM 7.2.1-69057 Update 4. Next I logged in it via SSH, used your dtc binary to modify /etc.defaults/model.dtb and edited /etc.defaults/synoinfo.conf. After that my device booted successfully, no warnings were shown by DSM. Finally, I connected the second SATA drive to one of eSATA ports, and DSM reported it as the sixth drive.
The procedure step by step for myself and anyone else desiring to experiment
# 1. Do everything as root
sudo su
# 2. Modify device tree
mkdir /volume1/test && cd /volume1/test
wget https://raw.githubusercontent.com/007revad/Synology_enable_eunit/main/bin/dtc && chmod +x ./dtc
./dtc -I dtb -O dts -o ./model.dts /etc.defaults/model.dtb
vi ./model.dts
## make changes to esata_port items, be careful with syntax and indents
./dtc -I dts -O dtb -o ./model.dtb.new ./model.dts
mv /etc.defaults/model.dtb /etc.defaults/model.dtb.backup
cp ./model.dtb.new /etc.defaults/model.dtb && chmod 644 /etc.defaults/model.dtb
# 3. Modify configuration file
cp /etc.defaults/synoinfo.conf /etc.defaults/synoinfo.conf.backup && chmod 755 /etc.defaults/synoinfo.conf.backup
vi /etc.defaults/synoinfo.conf
## make maxdisks equal to the number of internal_slot items in the changed model.dts
# 4. RebootI repeated the procedure with my production drives. So far my DS1522+ has been working flawlessly for 12+ hours with 7 SATA drives all being considered internal. The only extra thing I had to do for my production drives is grow /dev/md0 and /dev/md1 partitions to include system volumes on my 2 eSATA-connected drives since they had already been initialised.
Storage Manager HDD/SSD page before the procedure
Storage Manager HDD/SSD page after the procedure
Despite having maxdisks="7", DSM still displays only 5 slots on the overview page, but it doesn't matter. At least I haven't found any negative implications of it yet. EDIT: I found a solution, see below.
Storage Manager Overview page unchanged
I have also looked through some Synology DSM binaries (synodiskport, libsynodiskmap.so.1, libhwcontrol.so, libsynosata.so.1) with IDA. I think it is also possible to patch one of them so that any esata_port items in /etc.defaults/model.dtb were treated as internal drives. But it is a more complicated way compared to the device tree modification technique. For now I would prefer not to experiment with binary patching.
I actually have a DX213 so I should be able to experiment on getting the information from the DX213.
The ideal way would be to have a Synology and a non-Synology expantion units with similar hardware together with a disk station having 2 eSATA ports so that both eSATA multi-drive enclosures could be connected simultaneously.
Edits whatever files check if the eSATA device is a Synology expansion unit or not
As mentioned above I suppose it would require binary patching since the friend-or-foe logic is hidden somewhere inside. There is also a terminology issue in the function names of the Synology binaries: there are EBox, Esata, Eunit, PCIeEunit, PMP terms and I don't fully understand the logic.
The libhwcontrol.so binary contains dozens of functions related to expansion units so it might be of primary interest for patching. However there are some relevant functions in the following libraries:
libsynosata.so.1 exported functions
libsynodiskmap.so.1 exported functions
Very nice.
The only extra thing I had to do for my production drives is grow /dev/md0 and /dev/md1 partitions to include system volumes on my 2 eSATA-connected drives since they had already been initialised.
As the eSATA drives are now showing in storage manager as internal SATA drives I wonder if you could have created the storage pool in storage manager (which would have mirrored md0 and md1 to the SSDs for you).
We can probably get those drives to show up in the storage manager overview page. Xpenology devs can make it show as many drives are installed. But as long as the storage page shows the drives and their health you don't really need the overview page.
I definitely prefer to avoid binary patching if possible.
As mentioned above I suppose it would require binary patching since the friend-or-foe logic is hidden somewhere inside. There is also a terminology issue in the function names of the Synology binaries: there are
EBox,Esata,Eunit,PCIeEunit,PMPterms and I don't fully understand the logic.
Synology uses both ebox and eunit as short names for expansion units. I assume ebox may be left over from a time when Synology might have referred to them as expansion boxes. I've not seen PCIeEunit before but it may be for expansion units that use the infiniband port or the minisas port. I have no idea what PMP is for, but it reminds me of portable media players from before mobile phones were invented.
For C and javascript I usually paste the function into chatGPT and ask it what is that code doing and then I can work out what needs changing :o)
The libhwcontrol.so binary contains dozens of functions related to expansion units so it might be of primary interest for patching.
2 of my other scripts patch libhwcontrol.so.1 to make DSM think 3rd party NVMe drives are Synology NVMe drives. It's definitely the 1st file I'd look at.
As the eSATA drives are now showing in storage manager as internal SATA drives I wonder if you could have created the storage pool in storage manager (which would have mirrored md0 and md1 to the SSDs for you).
At the final step of my experiment with the spare drives the Storage Manager suggested that it created a storage pool on the sixth drive. What was more important for me is the storage pool I had already had before the procedure was automatically recognised (same as before) and assembled (crucial thing as it should have been done manually for eSATA-connected drives) on boot. I mirrored md0 and md1 manually but I suppose the Storage Manager could have done it for me.
PMP stands for port multiplier, as far as I understood.
storage_panel.js
Dave, thank you for advice about /var/packages/StorageManager/target/ui/storage_panel.js. It contains its own device classification logic which assigns my DS1522+ named as synology_r1000_1522+ to TOWER_5_Bay group. There are similar groups for bigger devices (6-, 8- and 12-bay). I decided not to introduce a new 7-bay group (too complicated) and transferred my device into the 8-bay group.
This is where to adjust storage_panel.js
Storage Manager Overview page now looks like this
Storage Manager Overview page now looks like this
Nice find.
Searching through the 42,000 lines of storage_panel.js can be overwhelming (especially when you're not sure what string to search for). At least you only had to search for 1019+.
This is where to adjust storage_panel.js
What are you using to view storage_panel.js?
At least you only had to search for 1019+.
No, actually I started from the 5-bay tower image loaded by the page, then I found the HTML element class name in style.css. Finally, it was storage_panel.js where the TOWER_5_Bay group had this class name in one place and a list of devices in the other one.
What are you using to view storage_panel.js?
It is Chrome Developer Tools