Seagate SMART parsing for smartmontools
Opened this issue · 8 comments
openSeaChest (obviously) correctly parses the SMART for Seagate drives with -smartAttributes analyzed
Many open-source tools still use smartctl (e.g. smartd, Prometheus / Node Exporter, TrueNAS, UnRAID)
Seagate SMART requires extra parsing on Raw_Read_Error_Rate and Seek_Error_Rate. They report the total bytes read and errors that needs to be parsed, which causes people to think that their drives have failed. I recommend the openSeaChest folks port the parsing over to smartmontools
https://www.smartmontools.org/browser
Hi @jmhands,
Thanks for reporting this to us. We will look into how we can submit a patch for this.
After some internal review, we should make this update for attributes 1, 7, and 195. Raw 6:4 should be the displayed counter like we have in the analyzed smart attributes output in openSeaChest.
Attribute 188 should also be updated to show raw 1:0 for the counter value.
I just tagged a couple updates to this issue that I thought were important and related even though they belong in openSeaChest rather than smartmontools.
These updates help clarify the output for these attributes in the openSeaChest_SMART -d <handle> --smartAttributes hybrid output since it is possible to have non-warrantied attributes at/below threshold to indicate warnings that are not necessarily failures. This also fixes the counters for the above attributes within openSeaChest to match the error counts that users seem most interested in.
Exos X20 20TB also affected, users running smartctl could run the command with the following addition which will show 0 for the current error Rates:
sudo smartctl /dev/sdX -a -v 1,raw48:54 -v 7,raw48:54
Or add a similar block like the Exos X16 for the Exos X20 in /var/lib/smartmontools/drivedb/drivedb.h and run smartctl normally:
{ "Seagate Exos X20", // tested with ST20000NM007D-3DJ103/SN03
// ST20000NM007D-3DJ103/SC03
"ST2[0]000NM00[7]D-.*",
"", "",
"-v 1,raw48:54,Read_Error_Rate "
"-v 7,raw48:54,Seek_Error_Rate "
"-v 18,raw48,Head_Health "
"-v 200,raw48,Pressure_Limit "
"-v 240,msec24hour32"
},
@vonericsen could you show -v 188 and -v 195 or how more important ones needs to be interpreted in above format?
@walterav1984,
That might work. I was looking into finding a way to do this for any Seagate drive, that way each drive does not need to be added as another entry in the database.
This issue has unfortunately been on hold due to resource constraints lately, but I have left it open as this is still important and should get updated when we get the time again.
Could you hint what the interpretation of the 188 and 195 value's are for the EXOS X20, since 4 out of 8 new disks show timeout counts/errors between 1-12 counting with openSeaChest_SMART.
Its maybe a non worrying value as a left over result from 512 > 4K sector changing which disconnected and reconnected the drive itself when it finished.
@walterav1984,
As far as I know, these are still interpreted the same way as past products.
If you do openSeaChest_SMART -d <handle> --smartAttributes hybrid, this gives a somewhat simplified count, similar to smartmontools.
If you do openSeaChest_SMART -d <handle> --smartAttributes analyzed this breaks the raw data out into the multiple fields that Seagate is tracking in the raw data.
For 188, this will give you a total count (matching hybrid output), number of timeouts > 5 seconds and number of timeouts > 7.5s.
All a command timeout is tracking is how many COMRESETs the host has issued between the drive received the command and before it was able to send a response.
If the OS or some software has a really short timeout on a read and cancels it, the OS will issue a reset as part of error recovery.
I don't recall all that 195 tracks, but by its name it tracks how many times the ECC algorithm has triggered a correction of the data from the medium. On its own, it doesn't really tell much about what is going on and you would need some other attributes showing some other information to better correlate an issue.
The attributes with the pre-fail/warranty flag are those that are most likely to indicate a failure. Other attributes can range from purely informational (like temperature) to tracking a wear out item (like start-stop count) to attributes like these two where on their own, they don't indicate a failure or an issue and may increment for many other reasons, but if a drive is having a problem, there is generally counts in multiple attributes, including a pre-fail attribute that are increasing in a way that would say something is happening. Correlating multiple counts to a single issue/event is very difficult to do and generally takes a lot of failure analysis across a lot of failed drives to figure out the issue.
SMART attributes use the "current" or "nominal" value to indicate a percentage of health, usually from 1-100% with 100% being perfectly healthy. Some attributes may use a different scale, but generally this is what they are based on. A threshold indicates the point at which the drive is at a point the manufacturer thinks it can indicate a failure (pre-fail attributes) or is indicating that a non-failure, possibly old-age attribute is wearing out (anything without the pre-fail flag that has a threshold reported).
The worst ever tracks the lowest that scaled value has ever been so you can determine what the lowest point the drive saw was since it is possible to have an attribute fail in the past, but recalculate more recently and not indicate a current failure. Whether or not a vendor will honor a past failure or not will depend on the vendor's policy. To the best of my knowledge, Seagate will honor that past failure event as that is how SeaChest, SeaTools, and openSeaChest are all assessing the drive in their source code.
If you recently switched from 512B to 4KB sectors, the drive does begin a lot of background processing which can take a long time (a couple of days). If you read through #117, I was able to provide some information about this and the other users who were involved in that issue also responded with how much time they saw the background activity took to complete.
It is possible that if the drive is processing this background activity that it took a little longer to respond to a read or write request than normal and it was just slightly long enough for the OS to think it was a command timeout. I don't have a way to say this is the case for sure or not, but it may be why that counter incremented.
Thanks for the comprehensive answer, although I'm a little bit shocked by the so called "background" processing time of disks that might have introduced the "timeout counter" to go up.
The disks were directly set to 4k sectors on their first boot, it took a couple of minutes before it mentions succeed with a bit of a "shaddy" warning message which hints that atleast 1 hour to do some background stuff and that it may take longer. But reading your comment it looks like it can be like multiple 30 hours(whole read/write disk cycle)...
The machine was rebooted after an hour of the success message that it was set to 4k, but if its very important not to interrupt should it be a better idea to make this 4k sector switching a "online" process on which the terminal command won't quit or won't go to background? Or make an option to view the actual background process?
I will figure out a better way to rewrite that message. The purpose of it was to inform the user not to worry if the utility appears to "hang" for an hour while the sector size change starts and not to interrupt it during this time. Interrupting during this time that it is busy can make the drive fail to work properly and require rerunning the format or some other vendor unique recovery process.
I will try to make it clear that background processing will occur once the drive has finished this first part of the sector size change.
In the standards for this command, it is described as issuing the command, the drive holds the bus in a "busy" state until the sector size change is done. There is no requirement on background processing as it is vendor unique what happens once the drive has returned status.
The busy state is the only part of this that is "online" and once it is done the drive is ready to read and write.
The background processing the drive is doing cannot be forced to the foreground unless you are writing data to the drive. There is no way to view the status of background processes the drive is performing.
The intention of this was to allow the user to write their own data rather than waiting for a complete write of the entire drive (possibly in a number of days with today's and the future's HDD capacities) and allow the user to quickly set it up and begin using the drive with their own data.
If you were to do a complete overwrite of the entire drive, this would essentially force the background processing the disk is doing into the foreground (this only applies to changing sector size...other background tasks may not be able to be forced to foreground). You can do this with whatever data you want, but it is not necessary to do before using the disk.