Error: Failed reading command from client: Resource temporarily unavailable <> pimctl !!??
Closed this issue · 8 comments
Log message is:
Oct 16 19:23:17 pfSense pimd[45774]: Failed reading command from client: Resource temporarily unavailable
Strange things there:
- "Failed reading command from client" comes from ipc.c where ..... ipc.h is not referred to not even existing any more
- No idea where "Resource temporarily unavailable" is coming from
Apart from pimctl --plain, pimctl is not working (no output). Also not a command like "pimctl --plain "
If you do not know where the problem is, I might try to tackle the pimctl next week.
It's either a read() error from the socket when the client tries to send a command,
or the wrong command (see below).
I've just pushed a change to return EBADMSG error code when the wrong command
is sent from pimctl so we can distinguish better between the two errors.
Note: after the first refactor of pimctl.c/ipc.c (to be able to share code with pimd-dense)
we currently only support exact commands. Partial commands is coming soon (I hope).
I.e., pimctl sh ig gr does not work anymore, you have to type the full command.
Joachim, I trace the main problem. It where details in ipc.c See pull request.
Two issues left:
- sometimes there is a space character in front of the entered command. As a consequence that command is not recognised
- each command is passing the if-statement within "ipc_read for-statement" twice. First with a command called help and then the real command.
After I fixed the communication I did test the commands (not all sub options). The Majority works. What does not seems to work is:
- pimctl debug ?
- pimctl log ?
- pimctl restart => No pimd running, no commands available. NOT TRUE / not working
- show pim neighbor => no response
- show pim compat
All tests and the build where executed using the brand new FreeBSD 12.2-RELEASE, in combination with the pfSense development release date Tue Oct 13.
Louis
Note that I did not update to today's pfSense release, since there might be another issue (not sure). This one to solve first :)
See response in pull reqeust #175.
On the first of your two remaining issues, what GIT hash of pimd have you been using when you run into the problem with space char before commands? The latest master, with last commit on Oct 22, works for me and I know I fixed both partial matching problems and extra space removal.
On the second, this is not an issue, this is by design. The new pimctl is agnostic of what PIM daemon it talks to, and to be agnostic if first queries available commands by sending "help" over the IPC channel. This returns the list of commands and arguments, which pimctl then uses to validate the user input. If user input matches the full commands (possibly expanded if short-forms of commands are used) is sent to the PIM daemon.
Here's my output from the commands you report to have problems with. As a reminder: if you get "No pimd running", then your pimctl tool cannot find the socket file that pimd creates. I've mentioned this before, the path to the socket file is specified using existing configure flags. On Linux systems I use --localstatedir=/var since those systems often use /var/run as the path for .pid and .sock files. This differs between systems, in particular on embedded systems like pfSense.
jocke@luthien:~$ sudo pimd -n # start daemon
[sudo] password for jocke:
pimd: 23:01:30.748 pimd version 3.0-beta1 starting ...
pimd: 23:01:30.748 Ignoring lo, has invalid address 127.0.0.1 and/or netmask 255.0.0.0
pimd: 23:01:30.761 Cannot open configuration file /etc/pimd.conf: No such file or directory
pimd: 23:01:30.761 Using built-in defaults, including RP/BSR candidate.
^Z
[1]+ Stopped sudo pimd -n
jocke@luthien:~$ bg # move to background
[1]+ sudo ./src/pimd -n &
jocke@luthien:~$ sudo pimctl # test IPC communication, default command
VIF Local Address Subnet Thresh Flags Neighbors
0 192.168.1.195 192.168.1 1 DR NO-NBR
1 192.168.122.1 192.168.122 1 DR NO-NBR
2 10.0.196.136 10.0.196/24 1 DR NO-NBR
3 172.17.0.1 172.17 1 DR NO-NBR
4 192.168.1.195 register_vif0 1
Number of Groups: 0
Number of Cache MIRRORs: 0
RP address Incoming Group Prefix Priority Holdtime
169.254.0.1 1 Static
232/8 1 Forever
Current BSR address: 192.168.122.1
jocke@luthien:~$ sudo pimctl help # ask PIM daemon what commands it has
Usage: pimctl [OPTIONS] [COMMAND]
Options:
-i, --ident=NAME Connect to named pimd instance
-p, --plain Use plain table headings, no ctrl chars
-t, --no-heading Skip table headings
-h, --help This help text
Commands:
debug [? | none | SYS] Debug subystem(s), separate multiple with comma
help This help text
kill Kill running pimd daemon, like SIGTERM
log [? | none | LEVEL] Set pimd log level: none, err, notice*, info, debug
restart Restart pimd and reload the .conf file, like SIGHUP
version Show pimd version
show status Show pimd status, default
show igmp groups Show IGMP group memberships
show igmp interface Show IGMP interface status
show pim interface Show PIM interface table
show pim neighbor Show PIM neighbor table
show pim routes [detail] Show PIM routing table
show pim rp Show PIM Rendezvous-Point (RP) set
show pim crp Show PIM Candidate Rendezvous-Point (CRP) from BSR
show pim compat [detail] Show PIM status, compat mode, previously `pimd -r`
jocke@luthien:~$ sudo pimctl show sta # example of partial completion and extra spaces
Elected BSR
Address : 192.168.122.1
Expiry Time : 0h0m50s
Priority : 5
Hash Mask Length : 30
Candidate BSR
State : Enabled
Address : 192.168.122.1
Priority : 5
Candidate RP
State : Enabled
Address : 192.168.122.1
Priority : 20
Holdtime : 75 sec
Join/Prune Interval : 60 sec
Hello Interval : 30 sec
Hello Holdtime : 105 sec
IGMP query interval : 125 sec
IGMP querier timeout : 380 sec
SPT Threshold : Enabled
SPT Interval : 100 sec
jocke@luthien:~$ sudo pimctl debug ? # listing available sub-systems for debug
all, asserts, bsr, crp, detail, igmp, interfaces, jp, kernel, mfc, mrt, neighbors, packets, pim, registers, rpf, rsrr, timers, traceroute
jocke@luthien:~$ sudo pimctl log ? # listing available log levels
alert, crit, debug, emerg, err, error, info, none, notice, panic, warn, warning
jocke@luthien:~$ sudo pimctl restart # restarting daemon (like SIGHUP)
pimd: 23:10:47.383 pimd version 3.0-beta1 restarting ...
pimd: 23:10:47.383 Ignoring lo, has invalid address 127.0.0.1 and/or netmask 255.0.0.0
pimd: 23:10:47.392 Cannot open configuration file /etc/pimd.conf: No such file or directory
pimd: 23:10:47.392 Using built-in defaults, including RP/BSR candidate.
jocke@luthien:~$ sudo pimctl show pim neigh # I have no neihgbors so no output (UNIX standard)
jocke@luthien:~$ sudo pimctl show pim compat # Show old SIGUSR1 dump file
VIF Local Address Subnet Thresh Flags Neighbors
0 192.168.1.195 192.168.1 1 DR NO-NBR
1 192.168.122.1 192.168.122 1 DR NO-NBR
2 10.0.196.136 10.0.196/24 1 DR NO-NBR
3 172.17.0.1 172.17 1 DR NO-NBR
4 192.168.1.195 register_vif0 1
Source Group RP Address Flags (*,G)
* 224.0.1.60 192.168.122.1 WC RP
Joined oifs: .....
Pruned oifs: .....
Leaves oifs: l....
Asserted oifs: .....
Outgoing oifs: o....
Incoming : ....I
TIMERS: Entry JP RS Assert VIFS: 0 1 2 3 4
0 50 0 0 0 0 0 0 0
Source Group RP Address Flags (*,G)
* 239.255.255.250 192.168.122.1 WC RP
Joined oifs: .....
Pruned oifs: .....
Leaves oifs: l....
Asserted oifs: .....
Outgoing oifs: o....
Incoming : ....I
TIMERS: Entry JP RS Assert VIFS: 0 1 2 3 4
0 50 0 0 0 0 0 0 0
Source Group RP Address Flags (S,G)
192.168.1.133 239.255.255.250 192.168.122.1 SPT CACHE SG
Joined oifs: .....
Pruned oifs: .....
Leaves oifs: l....
Asserted oifs: .....
Outgoing oifs: .....
Incoming : I....
TIMERS: Entry JP RS Assert VIFS: 0 1 2 3 4
195 50 0 0 0 0 0 0 0
Source Group RP Address Flags (S,G)
192.168.1.169 239.255.255.250 192.168.122.1 SPT CACHE SG
Joined oifs: .....
Pruned oifs: .....
Leaves oifs: l....
Asserted oifs: .....
Outgoing oifs: .....
Incoming : I....
TIMERS: Entry JP RS Assert VIFS: 0 1 2 3 4
200 55 0 0 0 0 0 0 0
Number of Groups: 2
Number of Cache MIRRORs: 2
RP address Incoming Group Prefix Priority Holdtime
192.168.122.1 4 75
224/4 20 70
169.254.0.1 1 Static
232/8 1 Forever
Current BSR address: 192.168.122.1
Joachim,
I did use your latest build. To be 100% sure of that, I compared the files you recently changes (config.c, debug.c, debug.h, defs.h and Makefile.am)
I did change the path, and that is working IMHO. Otherwise the majority of pimctl commands could not work. However they do work ...
Related to your versus my programming style, yep those are different. I would never write a function or subroutine without describing, function and in- and output- parameters on top of the function. You see that as terrible.
I do not like the use of global s. I would never(!) change an input-parameter during a function (unless it is also output parameter), so that it stays unchanged for the calling level.
I say this because you obviously find that way of working disgusting, ... That is your opinion, but perhaps I have the right to have mine.
Louis
I did something stupid! I did not test your latest changes, before doing my investigations!
Bottom line, you did already fix the issue, I noticed a few weeks ago!!
Based on your comments, I just realized that that could have fixed it yourself. So I did rebuild and retest based on your master branch without changes. Same results :)
I had been crabbing my had why it was working ...... given the minimal changes I made ..
Does not change my test results of course
Sincerely,
Louis
OK, it's becoming hard to follow here what's still is the issue, so I've now installed a FreeBSD 12.1 virtual machine to try things out myself. From what I can see so far, most of the pimctl commands work, only the debug and log commands don't work, so I'm creating separate issues for them. This was sort of expected since the IPC interface between pimd and pimctl was completely rewritten.
Also, for consistency with the other commands, I'll add a header to the show pim neighbor command, even if there are no neighbors. I guess that's what you meant earlier by "no response"?
...
I did not say 'terrible' or 'disgusting', it's just not the style of this project. When contributing minor patches to a project it is customary to adhere to the style of that project, in particular keeping with the style in the same file and surrounding lines, this is not unique to me or my projects. Not adhering to the project's coding style and attempting to add your own is just being plain rude since it just adds to the overhead of the maintainer to clean up after you.
I've now debugged the case with debug ? and log ? for you as well. For details, see issue #176, which I just closed since it's not a problem with pimd/pimctl but a limitation of the FreeBSD C shell, /bin/csh.
I believe that concludes the outstanding issues you've tagged on to this bug report and we should be able to close it. Do you concur?
Yep,
Problem is solved. Personally I would:
- give errno in the "len < 0" an value, since it is an output parameter (even if it is a global).
- consider adding my "log_notice" messages, since they are help full to trace problems (you have to put len ==0) first
Not new for you of course, but global parameters are in conflict with either multiple instances of re-entrant functions :)
I will upgrade to latest pfSense release now, to check if that works ok.
Louis