Dependency heavy device explorer
Hertz642 opened this issue · 15 comments
So I found the problem that was making Solid not work for me. It turns out I made a mistake with defining the USE flag (these things should probably make compilation fail if the cmake flag is on but the packaging or something isn't working right).
I now get a new failure:
stderr: [warning] Failed enumerating UDisks2 objects: "org.freedesktop.DBus.Error.Disconnected"
stderr: "Not connected to D-Bus server"
stderr: [warning] Failed enumerating UDisks2 objects: "org.freedesktop.DBus.Error.Disconnected"
stderr: "Not connected to D-Bus server"
[warning] virtual QStringList Solid::Backends::UPower::UPowerManager::allDevices() error: "org.freedesktop.DBus.Error.Disconnected"
[debug] Creating "deviceexplorer" in 114 ms.
I looked up stuff on udisks and figured out how to fix it, but it looks like I need to configure the DBus daemon, and then configure the udisks daemon, which requires polkit, which requires ConsoleKit. Argh! I suppose these dependencies are understandable if your target user group is going to be running a more heavyweight desktop system like Gnome, but when you are trying to build a system from scratch it makes things difficult (especially if you want a bloat free system). I'm not a neckbeard (rather I am somewhat of a novice), but I have found that I very much prefer Unix philosophy CLI to APIs that seem to cover up stuff from the user (the reason why I originally quit Windows).
I am not trying to gripe about your software (I think its great!), but what is your intention with regard to depending on these sorts of complex desktop-style systems? Would it be troublesome to be able to have similar functionality without requiring them?
Yeah, Solid requires some dependencies, but I think they're quite standard in a modern Linux system. I wouldn't know how to have the same functionality without them. That's why the solid dependency is optional, though.
Well, that is true of a Linux system with a Freedesktop.org style desktop system, but there are a lot of setups that don't have those dependencies. There is also the other Unix-like systems like the BSDs. The Solid dependency is optional, but there is currently no way to get the device explorer to work without it. What functionality do you need for it to work? Devices can be displayed from the directory contents of /media (usually), and there are a number of daemons for mounting and unmounting devices for users. Here are a few examples:
Autofs/automount:
https://wiki.gentoo.org/wiki/AutoFS
https://wiki.archlinux.org/index.php/autofs
http://linux.die.net/man/8/automount
Pmount:
http://pmount.alioth.debian.org/
http://linux.die.net/man/1/pmount
Udevil:
http://ignorantguru.github.io/udevil/
It looks like automount is the most standard. It says in the manual page that you can make it unmount all drives by giving it the USR1 signal, and it will force the unmount if given the USR2 signal.
You might consider providing generic commands for every system task that might require a dependency, and then you can use options in the configuration file to allow the user (or admin) to change them to their own commands. If the configuration file was made easier for an admin or third party developer to manage it (e.g. ini instead of xml, seperate file for GUI layout or launched applications, separate system and user configs, etc.), it might be beneficial to expose other configuration options as well.
These commands could possibly provide fallbacks for all (AFAIK) of the non-trivial things that Orbital has to do to interact with the OS:
List devices (outputs a multiline string with one filename per line):
ls -1 /media
Unmount all devices:
kill -USR1 "$(cat /var/run/automount)"
Force unmounting all devices:
kill -USR2 "$(cat /var/run/automount)"
Get current volume (gives a string containing a number between 0 and 100):
amixer sget Master | grep -o -m 1 -E '[0-9]+%' | sed 's/%$//'
Set the volume to 40:
amixer -q sset Master 40%
Add 10% to the current volume:
amixer -q sset Master 10%+
Subtract 10% from the current volume:
amixer -q sset Master 10%-
Test to see if the audio output is enabled (gives the string 'on' for enabled, 'off' for muted):
amixer sget Master | grep -o -m 1 -E '\[on\]|\[off\]' | sed -e 's/\[//' -e 's/\]//'
Mute:
amixer -q sset Master mute
Unmute:
amixer -q sset Master unmute
Logout:
logout
Reboot:
shutdown -r now
Power off:
shutdown -hP now
The reason I put the ALSA commands here is for a default that can be switch to something else. For example, someone might want to use OSS4 (Linux) or the native OSS fork for BSD.
I think on most systems you could simply pipe the stdout to /dev/null on commands you aren't capturing output from. I think that all of these would exit with a nonzero value if there was an error, and capturing stderr could allow you to pass the error along to the user. This would provide a fairly generic means to replace any of the dependencies, since you could swap in anything from the CLI to replace it. Aside from providing the Freedesktop.org style optional dependencies, and perhaps a sane CLI default, you wouldn't have to care about how to interface with the many other possible setups.
Well, that is true of a Linux system with a Freedesktop.org style desktop system, but there are a lot of setups that don't have those dependencies. There is also the other Unix-like systems like the BSDs. The Solid dependency is optional, but there is currently no way to get the device explorer to work without it. What functionality do you need for it to work?
Mounting and unmounting volumes wouldn't be a big problem since there are many options, but what I really need Solid for is to get a list of the devices, with their names too, and to be notified when a device is added or removed. ls /media works for mounted volumes only, and anyway Solid lists not only volumes, but any device.
I don't think I will change the configuration file format. Qt doesn't provide a ini file parser, and anyway I don't think it is hard to manually modify the current format.
Where there's a will there's a way:
Notify of block device changes, giving the event type and device path: /etc/udev/rules.d/90-orbital-notify.rules
# Floppy drives:
KERNEL=="fd*", SUBSYSTEM=="block", ACTION=="add", RUN+="/path/to/orbital/notify/command add $devpath"
KERNEL=="fd*", SUBSYSTEM=="block", ACTION=="remove", RUN+="/path/to/orbital/notify/command remove $devpath"
# IDE hard drives:
KERNEL=="hd*", ACTION=="add", RUN+="/path/to/orbital/notify/command add $devpath"
KERNEL=="hd*", ACTION=="remove", RUN+="/path/to/orbital/notify/command remove $devpath"
# SCSI hard drives:
KERNEL=="sd*", ACTION=="add", RUN+="/path/to/orbital/notify/command add $devpath"
KERNEL=="sd*", ACTION=="remove", RUN+="/path/to/orbital/notify/command remove $devpath"
# Metadisk (raid) drives:
KERNEL=="md*", ACTION=="add", RUN+="/path/to/orbital/notify/command add $devpath"
KERNEL=="md*", ACTION=="remove", RUN+="/path/to/orbital/notify/command remove $devpath"
# SCSI CD-ROM drives:
KERNEL=="scd*", ACTION=="add", RUN+="/path/to/orbital/notify/command add $devpath"
KERNEL=="scd*", ACTION=="remove", RUN+="/path/to/orbital/notify/command remove $devpath"
# MMC/SD cards:
KERNEL=="mmcblk*", ACTION=="add", RUN+="/path/to/orbital/notify/command add $devpath"
KERNEL=="mmcblk*", ACTION=="remove", RUN+="/path/to/orbital/notify/command remove $devpath"
# USB block devices:
KERNEL=="ub*", ACTION=="add", RUN+="/path/to/orbital/notify/command add $devpath"
KERNEL=="ub*", ACTION=="remove", RUN+="/path/to/orbital/notify/command remove $devpath"
List currently attached devices that can be partitioned (no devpath):
cat /proc/partitions \
| grep -iE '[0-9]+ +[a-z0-9:_-]+' \
| grep -oiE '[a-z0-9:_-]+$'
Check to see if the device has a partition on it (will return non-empty if it does, empty if it does not):
# Use full path in case /sbin is not in PATH:
/sbin/blkid -o value -s TYPE "$devpath"
Given a devpath, return the user-friendly name if the device has one, or the device name if it does not:
#!/bin/sh
devpath="$1"
label="$( /sbin/blkid -o value -s LABEL "$devpath" )"
if [ -z "$label" ]; then
label="$( basename "$devpath" )"
fi
echo "$label"
I can maintain the commands/scripts for a default CLI backend if you like. I have wanted to help out more, but I don't really have much experience with C++/Qt. My main experience is in sysadmin/CLI type stuff.
Very interesting. I'll work out a way to use these. Thanks for the research.
I've started writing the cli backend for the hardware service. I can get the list of the devices from /proc/partitions, but i see now blkid requires root access. Isn't there some other way to get the type of a partition? Also I'd like to find a way to get the type of a device, e.g. internal disk vs. usb vs. optical disc.
blkid requires root access? I just tested it on Gentoo and Ubuntu and it does not require root access on either of them, at least not for getting labels. It only looks like it does because it is in /sbin, which is not in the default PATH for non-root users on most distros. For system scripts it is usually a good idea to supply full paths for any command that it is not a builtin, if it is certain that they will always be in the same place. There may also be something in one of the pseudo-filesystems to help us, so I will look into that.
Keep in mind while writing the backend that it needs to be sufficiently generic. People using Orbital on other OSes, like *BSD, may need to swap out the shell commands for something else.
Types of devices by /dev name, from Documentation/devices.txt in the Linux kernel source:
/dev/fd* - Floppy disk.
/dev/hd* - IDE drives, HDD and optical.
/dev/sd* - SCSI HDDs.
/dev/md* - RAID devices.
/dev/scd* - SCSI optical disks.
/dev/sonycd - Sony CDU-31A/CDU-33A CD-ROM.
/dev/gscd - GoldStar CD-ROM.
/dev/optcd - Optics Storage CD-ROM.
/dev/sjcd - Sanyo CD-ROM.
/dev/double* - "Double" compressed disk (I think this is a floppy successor?).
/dev/hitcd - Hitachi CD-ROM.
/dev/mfm* - Acorn MFM hard drive interface.
/dev/mcd - Mitsumi proprietary CD-ROM.
/dev/cdu535 - Sonly CDU-535 CD-ROM.
/dev/sbpcd* - Matsushita CD-ROM.
/dev/aztcd - Aztech/Orchid/Okano Wearnes CD-ROM.
/dev/cm205cd - Philips LMS CM-205 CD-ROM.
/dev/rom* - ROM/flash memory card.
/dev/c06cd - Philips LMS CM-206 CD-ROM.
/dev/slram - Slow memory ramdisk (?).
/dev/z2ram - Zoro II ramdisk.
/dev/ftl* - Flash Translation Layer (FTL) filesystems.
/dev/pd* - Parallel port IDE disks.
/dev/pcd* - Parallel port ATAPI CD-ROMs.
/dev/pf* - Parallel port ATAPI disk devices.
/dev/rd/cd*d* - Mylex DAC960 PCI RAID.
/dev/pda* - Generic PDA filesystem device.
/dev/scramdisk/* - Scramdisk/DriveCrypt encrypted drive.
/dev/c*d* - Compaq Intelligent Drive Array.
/dev/i2o/hd* - I2O hard disk.
/dev/ppdd* - PPDD encrypted disk driver.
/dev/nftl* - NAND Flash Translation Layer filesystem.
/dev/dasd* - IBM S/390 DASD block storage.
/dev/inftl* - Inverse NAND Flash Translation Layer.
/dev/u* - User-mode virtual block device.
/dev/jsfd - JavaStation flash disk.
/dev/amiraid/ar* - AMI HyperDisk RAID controller.
/dev/cpd/* - Compressed block device.
/dev/cciss/c*d* - Compaq Next Generation Drive Array
/dev/iseries/vd* - IBM iSeries virtual disk.
/dev/iseries/vcd* - IBM iSeries virtual CD-ROM.
/dev/ataraid/d* - IDE BIOS powered software RAID.
/dev/nwfs/v* - Netware Devices.
/dev/umem/d* - MicroMemory batter backed RAM adapter (NVRAM).
/dev/evms/* - Enterprise Volume Management System (EVMS).
/dev/drbd* - Distributed Replicated Block Device (DRBD).
/dev/etherd/* - EtherDrive Block Devices.
/dev/emd/* - Enhanced Metadisk RAID (EMD) storage units.
/dev/carmel/* - Carmel 8-port SATA Disks.
/dev/mmcblk* - MMC block devices (also includes SD cards).
/dev/ub* - USB block devices.
/dev/vx/dsk/*/* - Veritas volume manager (VxVM) volumes.
/dev/vx/dmp/* - Veritas VxVM dynamic multipathing driver.
/dev/xvd* - Xen Virtual Block Device.
/dev/rfd* - Rodent Flash Disk Flash Translation Layer
/dev/ssfdc* - SSFDC Flash Translation Layer filesystem.
/dev/blockrom* - ROM/Flash read-only translation layer.
This should be an exhaustive list of all block devices that can appear in Linux, except for some that I left out because they would not have filesystems on them. My wildcards are a little sloppy though. I could clean this up and turn it into a shell case statement giving device type.
I'm guessing you need the type of device for determining what icon to use? In that case I have the following device icons when using Oxygen:
drive-harddisk
drive-optical
drive-removable-media-usb-pendrive
drive-removable-media-usb
drive-removable-media
media-flash-memory-stick
media-flash-sd-mmc
media-flash-smart-media
media-flash
media-floppy
media-optical-audio
media-optical-blu-ray
media-optical-data
media-optical-dvd-data
media-optical-dvd-video
media-optical-dvd
media-optical-mixed-cd
media-optical-recordable
media-optical-video
media-optical
media-tape
Or, leaving out the more specific ones:
drive-harddisk
drive-optical
drive-removable-media
media-flash
media-floppy
media-optical
media-tape
The script that reports device discovery/update could give one of these types to Orbital via some command or something.
Two commands of interest: lsblk and findmnt.
They do not require root, and are in /bin instead of /sbin.
In my Arch system blkid runs as normal user, but it only outputs data it cached from a previous run with root. So if i run blkid without having run sudo blkid before it shows nothing. sudo blkid outputs and caches everything, so running blkid later works fine, until something changes. Then it will be out of sync.
That list of device names won't work. In my arch i have the cdrom drive as /dev/sr0, and both the internal sata disks and a usb external drive shows up as /dev/sd*. I need that for the icon, yeah.
After doing some experimenting, I have determined that while there are various commands that will list the filesystem label, none of them will do it when run as an unprivileged user (they will simply report the label as being blank). There is also, so far as I can tell, no file in /sys or /proc that gives the label. I checked this by doing a recursive grep of all the files in those directories, which is stupid but I got what I needed (it crashed my computer the first time I tried it when I messed up the command, but I did it on a system that didn't have anything important on it). You can sometimes get a label using 'mount -l', but it doesn't work with unmounted partitions and doesn't always work with mounted ones. I suspect the label reading commands are reading from the raw block devices in /dev to determine the labels, which would require privileged access. Udisks does this for dbus clients.
I suppose there are three options: use a privileged background process, set one of those commands to be suid, or have udev do it when a device is plugged in before dropping privileges and notifying Orbital. (Oops, looks like I forgot to drop privileges above with the udev rules, I'll look into how to fix that). I think the last option is the best, though with a generic CLI hook it might not matter to Orbital how it is done and we can just pick one as a good default. Or there is a fourth option which is the simplest: set the default CLI command/script to just give Orbital the device name as the pretty title (this isn't great, but most users of lightweight systems are going to know the device names better than the typical user of Freedesktop-style systems).
About that list of device names: when I was looking at the documentation for the 3.x, it mentioned /dev/sr* as a deprecated label, so I skipped over it. Maybe you are using 2.x? I can just merge the list from 2.x in to the above list. As for the USB external drives: it is showing up as /dev/sd* because it is using USB SCSI. I don't know what /dev/ub* is used for since I have never had a device appear as that. Most of those minor device names could probably be dropped without much trouble (users of weird devices probably know how to set them up better than we do).
About that list of device names: when I was looking at the documentation for the 3.x, it mentioned /dev/sr* as a deprecated label, so I skipped over it. Maybe you are using 2.x?
No, I'm running 3.12.6.
However 'lsblk -o NAME,FSTYPE,RM,MODEL,TYPE,SIZE,LABEL' seems to give everything i need, even with a normal user, and it's not setuid. I just need some sh-fu to parse the output. I could do that in c++ but then it would not allow to change the command.
This is what i get on my system:
NAME FSTYPE RM MODEL TYPE SIZE LABEL
sda 0 Maxtor 6G160P0 disk 149,1G
├─sda1 ntfs 0 part 24,4G
├─sda2 0 part 1K
├─sda3 swap 0 part 957M
├─sda4 ext4 0 part 105,1G
└─sda5 ntfs 0 part 18,6G Volume
sdb 0 ST3500418AS disk 465,8G
├─sdb1 ext4 0 part 28G
├─sdb2 swap 0 part 1,9G
└─sdb3 ext4 0 part 419,1G
sdc 0 ST3500418AS disk 465,8G
└─sdc1 ntfs 0 part 465,8G Volume
sdd 1 256MB Portable disk 250M
└─sdd1 vfat 1 part 250M
sr0 1 CDDVDW SH-S203D rom 1024M
I'd need a way, given a partition, to get the TYPE of its disk, so sda1 should give 'disk', while /dev/sr0 should give 'rom'. Getting the same for the MODEL would be nice. By looking at the value of RM i can distinguish between internal and external devices.
Ah, MODEL is doing what I was trying to get LABEL to do. I see that it actually gives you a value for an NTFS mount, but I couldn't get it to give me the LABELs that you can use to name your device. A couple of comments about your shell statements: my lsblk doesn't have a 'p' option (lower case); and you can use this statement to get rid of the pipes and do everything with only one subshell:
export $( lsblk -Po MODEL /dev/sda ) && eval "echo $MODEL"
Export avoids using the Bashism "declare", and eval gets rid of the quotes that lsblk puts on the variable.
Your statement doesn't work for me. eval lsblk -Pdo MODEL /dev/sda ;echo $MODEL does.
The -p option is for showing the full devpath, i.e. the "/dev/" in front of the sd*.
Also, i saw udevadm info --name=/dev/sda gives a lot of useful informations on the devices.
Btw, I'm struggling on the udev rule. I got it working, and i made it echo to a file when a device is added or removed. I'd like to use a unix socket instead of a file though, and let orbital connect to it, but i couldn't find a way to do so in bash. Do you have any idea?
When I try to run lsblk with -p, I get a unknown option error.
I will take a look at what udevadm can do.
You can communicate with Unix sockets using Netcat:
echo "message" | nc -U /path/to/socket
It has to be installed, since it isn't as low-level to Linux as the udev tools are, but most Unix-like systems have it installed. An alternative is to use a FIFO:
mkfifo /tmp/foobar
( read line < /tmp/foobar; [ "$line" = test ] && echo "Success!" ) &
echo "test" > /tmp/foobar
However, a process attempting to read or write with a FIFO will block until there is something at the other end to send/receive, which probably isn't what you want. However, it is lighter than a socket, and requires only the base utilities.