aaronwmorris/indi-allsky

Transition from Day to Night - Image capture doesn't resume

Closed this issue · 22 comments

I have an issue that the image capture won't restart as night falls. I have seen this issue occur over many nights recently, and I have had to manually intervene.

The log shows the nightfall transition from

Mar 3 18:06:28 PiAllSky [INFO] Capture-1-2098/MainThread capture.saferun() [265]: Daytime capture is disabled
Mar 3 18:07:27 PiAllSky [INFO] Capture-1-2098/MainThread capture.saferun() [209]: Exposure state: BUSY

to

Mar 3 18:07:38 PiAllSky [INFO] Capture-1-2098/MainThread capture.saferun() [209]: Exposure state: ALERT
Mar 3 18:07:38 PiAllSky [INFO] Capture-1-2098/MainThread capture.saferun() [208]: Camera last ready: 44466.5s
Mar 3 18:07:30 PiAllSky [INFO] MainProcess-2094/MainThread allsky._queueManualTasks() [998]: Checking for manually submitted tasks
Mar 3 18:07:27 PiAllSky [WARNING] Capture-1-2098/MainThread indi.abortCcdExposure() [961]: Aborting exposure
Mar 3 18:07:27 PiAllSky [INFO] Capture-1-2098/MainThread miscDb.addNotification() [871]: Added Camera notification: 611
Mar 3 18:07:27 PiAllSky [INFO] Capture-1-2098/MainThread indi.getTelescopeRaDec() [853]: Telescope Coord: RA 5.18, Dec 41.74

and reloading the config doesn't help. I continue to see a log entry of the type below.

Mar 3 19:53:30 PiAllSky [INFO] Capture-1-2098/MainThread capture.saferun() [209]: Exposure state: ALERT

If I manually stop and start the indi-allsky the data taking restarts as normal.

Mar 3 20:08:44 PiAllSky [INFO] Capture-1-5701/MainThread capture.saferun() [209]: Exposure state: OK
Mar 3 20:08:44 PiAllSky [INFO] Capture-1-5701/MainThread capture.saferun() [208]: Camera last ready: 2.4s

I am running the version shown at bottom of this note.

What can I do to further help debug this issue?

Thanks, Rick

Merge: c567a42 07c11df
Author: Aaron W Morris aaron@aarmor.net
Date: Thu Feb 22 21:12:57 2024 -0500
Merge pull request #1142 from aaronwmorris/dev

Is it a correct assumption you are using an ZWO ASI120?

CAMERA:

Name
SX CCD UltraStar

Driver
indi_sx_ccd

Resolution
1392 x 1040 (1737 diag)

Dimensions
9.0mm x 6.7mm (11.2mm diag)
Pixels
6.45µm
CFA
None
Bits
16
Gain
-1--1
Exposure
0.000100-3600s

That is a massive sensor (relatively speaking). Starlight Xpress is the one vendor I have not had much experience.

What is your maximum exposure time? In order to have indi-allsky try to abort an exposure, the exposure must have exceeded 180s hard timeout. Sometimes cameras misbehave and exposures exceed the configured value. I picked 180 because I did not foresee people using exposures much longer than 60s. I can extend that if that is a problem.

Aborting exposures could leave the camera in an Alert state.

I have 60 seconds as the maximum exposure - longer than that and the star trails become too noticeable. This works well on moonless dark nights in suburbia.
I have set 4 secs as the minimum, 5 secs as default. I did this because if I allow much shorter times the autoexposure adjust at the start of the night will not work. It remains stuck at very short exposures, and all images are completely black. The logic doesn't increase the exposure times for some reason.

Are there different default times I should try? Can I expand the log file level to get more diagnostic data for the exposure logic and error case?

This is probably a problem with my bit depth scaling auto-detection magic.

indi-allsky accepts RAW image data by default, which is usually uint16 data. There is some variability in the INDI drivers as to the data range used for the FITS data. For instance, if you have a 12-bit camera, some drivers auto-scale the data to the full 16-bit space and the maximum data value will be 65535. Unfortunately, some drivers will leave the data in the 12-bit space, so the maximum value is 4095. indi-allsky needs to know this to properly scale the data to 8-bits for the resulting JPEG image.

indi-allsky has a simple algorithm that assumes the data is 8-bits by default, and based on the maximum value in the data, detects the likely data range. Sometimes, there is not enough noise (hot pixels actually help here) and the algorithm will assume 12-bit data is 10-bits, scale the data incorrectly, and the image will be "gray" at the target brightness value. The exposure is never scaled up because the algorithm thinks it achieved the target exposure.

One of the fixes is what you already discovered, setting the exposure to a sufficiently high value so there is enough data to properly detect the data range.

The other option is to override the auto-detection algorithm and explicitly define the bit depth of the data. This setting is on the Camera Tab at Camera Bit Depth

I changed the Bit Depth to manual value - 16 bit (correct for CCD)
I also put Min Exposure back on Auto values. This seems to be working when manually started.

However, the issue with failed re-start at the beginning of the night after being suspended all day is still happening.

The Notifcation window says "Camera last ready 45288s ago. Camera might be hung." (TIme is half a day)

Is it possible the code is misinterpreting the daytime suspension as a non-responsive camera?

Thanks, Rick

indi-allsky-error

No, it should not misinterpret that. That is a problem I fixed long ago.

Can you post the output of ./misc/support_info.sh ?

admin@PiAllSky:~/indi-allsky $ ./misc/support_info.sh 
./misc/support_info.sh: line 49: warning: command substitution: ignored null byte in input
#################################
### indi-allsky support info  ###
#################################

Distribution: Debian
Release: 11
Arch: aarch64
Bits: 64

CPUs: 4
Memory: 3885396 kB

System: Raspberry Pi 4 Model B Rev 1.5

Uptime
 22:38:16 up 4 days,  4:56,  3 users,  load average: 0.39, 0.29, 0.21

Filesystems
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/root      491715136 76201368 395476884  17% /
devtmpfs         1678472        0   1678472   0% /dev
tmpfs            1942696      180   1942516   1% /dev/shm
tmpfs             777080     1252    775828   1% /run
tmpfs               5120        4      5116   1% /run/lock
/dev/mmcblk0p1    261108    32674    228434  13% /boot
tmpfs             388536       20    388516   1% /run/user/1000

system python: Python 3.9.2

indiserver: /usr/local/bin/indiserver


User info
uid=1000(admin) gid=1000(admin) groups=1000(admin),4(adm),20(dialout),24(cdrom),27(sudo),29(audio),44(video),46(plugdev),60(games),100(users),101(systemd-journal),104(input),106(render),108(netdev),117(lpadmin),997(gpio),998(i2c),999(spi)

Process info
admin        645  0.2  0.0  83096  3688 ?        Ss   Mar09  12:10 /usr/local/bin/indiserver -p 7624 indi_simulator_telescope indi_sx_ccd
admin        709  0.1  0.2  34688 10664 ?        S    Mar09   6:09 indi_simulator_telescope
admin        710  0.1  0.4 185344 16256 ?        Sl   Mar09   7:11 indi_sx_ccd
admin       2956  0.0  0.5  27320 19440 ?        SNs  Mar09   1:26 /home/admin/indi-allsky/virtualenv/indi-allsky/bin/python /home/admin/indi-allsky/virtualenv/indi-allsky/bin/gunicorn --config /etc/indi-allsky/gunicorn.conf.py indi_allsky.wsgi
admin       2958  0.0  5.3 829340 205952 ?       SNl  Mar09   3:11 /home/admin/indi-allsky/virtualenv/indi-allsky/bin/python /home/admin/indi-allsky/virtualenv/indi-allsky/bin/gunicorn --config /etc/indi-allsky/gunicorn.conf.py indi_allsky.wsgi
admin      21000  3.8  5.4 573764 211812 ?       SNsl 21:23   2:50 /home/admin/indi-allsky/virtualenv/indi-allsky/bin/python3 allsky.py --log syslog run
admin      21005  0.6  2.8 495404 108900 ?       Sl   21:23   0:28 /home/admin/indi-allsky/virtualenv/indi-allsky/bin/python3 allsky.py --log syslog run
admin      21007  2.3  4.7 708272 185380 ?       Sl   21:23   1:47 /home/admin/indi-allsky/virtualenv/indi-allsky/bin/python3 allsky.py --log syslog run
admin      21008  0.0  2.6 374052 103056 ?       SN   21:23   0:04 /home/admin/indi-allsky/virtualenv/indi-allsky/bin/python3 allsky.py --log syslog run

USB info
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 007: ID 0bda:8176 Realtek Semiconductor Corp. RTL8188CUS 802.11n WLAN Adapter
Bus 001 Device 003: ID 1278:0525 Starlight Xpress ECHO2
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

USB Permissions
      123      0 drwxr-xr-x   4 root     root           80 Dec 31  1969 /dev/bus/usb
      126      0 drwxr-xr-x   2 root     root           60 Dec 31  1969 /dev/bus/usb/002
      127      0 crw-rw-r--   1 root     root     189, 128 Mar  9 16:17 /dev/bus/usb/002/001
      124      0 drwxr-xr-x   2 root     root          120 Mar 13 18:43 /dev/bus/usb/001
      460      0 crw-rw-r--   1 root     root     189,   6 Mar 13 18:43 /dev/bus/usb/001/007
      139      0 crw-rw-rw-   1 root     root     189,   2 Mar 13 22:38 /dev/bus/usb/001/003
      138      0 crw-rw-r--   1 root     root     189,   1 Mar  9 16:17 /dev/bus/usb/001/002
      125      0 crw-rw-r--   1 root     root     189,   0 Mar  9 16:17 /dev/bus/usb/001/001

Module info
Module                  Size  Used by
rfcomm                 53248  4
cmac                   16384  3
algif_hash             16384  1
aes_arm64              16384  3
aes_generic            36864  1 aes_arm64
algif_skcipher         16384  1
af_alg                 28672  6 algif_hash,algif_skcipher
bnep                   24576  2
hci_uart               53248  1
btbcm                  24576  1 hci_uart
bluetooth             565248  31 hci_uart,btbcm,bnep,rfcomm
ecdh_generic           16384  2 bluetooth
ecc                    36864  1 ecdh_generic
libaes                 16384  3 aes_arm64,bluetooth,aes_generic
rtl8192cu              90112  0
rtl_usb                20480  1 rtl8192cu
rtl8192c_common        65536  1 rtl8192cu
rtlwifi               118784  3 rtl8192c_common,rtl_usb,rtl8192cu
binfmt_misc            20480  1
mac80211              974848  3 rtl_usb,rtl8192cu,rtlwifi
libarc4                16384  1 mac80211
vc4                   331776  11
snd_soc_hdmi_codec     20480  2
drm_display_helper     16384  1 vc4
cec                    53248  1 vc4
brcmfmac              331776  0
drm_dma_helper         20480  1 vc4
drm_kms_helper        204800  3 drm_dma_helper,vc4
bcm2835_codec          49152  0
rpivid_hevc            53248  0
brcmutil               24576  1 brcmfmac
bcm2835_isp            28672  0
snd_soc_core          274432  2 vc4,snd_soc_hdmi_codec
bcm2835_v4l2           45056  0
v4l2_mem2mem           45056  2 bcm2835_codec,rpivid_hevc
cfg80211              925696  3 rtlwifi,brcmfmac,mac80211
bcm2835_mmal_vchiq     36864  3 bcm2835_codec,bcm2835_v4l2,bcm2835_isp
videobuf2_dma_contig    20480  3 bcm2835_codec,rpivid_hevc,bcm2835_isp
videobuf2_vmalloc      16384  1 bcm2835_v4l2
videobuf2_memops       16384  2 videobuf2_vmalloc,videobuf2_dma_contig
videobuf2_v4l2         32768  5 bcm2835_codec,bcm2835_v4l2,rpivid_hevc,v4l2_mem2mem,bcm2835_isp
videobuf2_common       69632  9 bcm2835_codec,videobuf2_vmalloc,videobuf2_dma_contig,videobuf2_v4l2,bcm2835_v4l2,rpivid_hevc,v4l2_mem2mem,videobuf2_memops,bcm2835_isp
snd_compress           20480  1 snd_soc_core
snd_bcm2835            28672  1
snd_pcm_dmaengine      20480  1 snd_soc_core
rfkill                 32768  8 rtlwifi,bluetooth,cfg80211
videodev              274432  7 bcm2835_codec,videobuf2_v4l2,bcm2835_v4l2,videobuf2_common,rpivid_hevc,v4l2_mem2mem,bcm2835_isp
syscopyarea            16384  1 drm_kms_helper
sysfillrect            16384  1 drm_kms_helper
snd_pcm               139264  5 snd_bcm2835,snd_soc_hdmi_codec,snd_compress,snd_soc_core,snd_pcm_dmaengine
v3d                    90112  8
sysimgblt              16384  1 drm_kms_helper
snd_timer              36864  1 snd_pcm
gpu_sched              49152  1 v3d
mc                     61440  7 videodev,bcm2835_codec,videobuf2_v4l2,videobuf2_common,rpivid_hevc,v4l2_mem2mem,bcm2835_isp
vc_sm_cma              36864  2 bcm2835_mmal_vchiq,bcm2835_isp
fb_sys_fops            16384  1 drm_kms_helper
snd                   106496  8 snd_bcm2835,snd_soc_hdmi_codec,snd_timer,snd_compress,snd_soc_core,snd_pcm
raspberrypi_hwmon      16384  0
drm_shmem_helper       24576  1 v3d
i2c_brcmstb            16384  0
uio_pdrv_genirq        16384  0
uio                    24576  1 uio_pdrv_genirq
nvmem_rmem             16384  0
i2c_dev                20480  0
drm                   581632  17 gpu_sched,drm_kms_helper,drm_dma_helper,v3d,vc4,drm_shmem_helper,drm_display_helper
fuse                  135168  3
drm_panel_orientation_quirks    28672  1 drm
backlight              24576  1 drm
ip_tables              32768  0
x_tables               53248  1 ip_tables
ipv6                  557056  42

git status
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean

git log
commit f0a294ab4eb4e0180d9b968d53794951cb315c2e
Merge: c567a429 07c11dfa
Author: Aaron W Morris <aaron@aarmor.net>
Date:   Thu Feb 22 21:12:57 2024 -0500

    Merge pull request #1142 from aaronwmorris/dev
    
    pass list instead of tuple for fish2pano

indi version: 2.0.3

indi packages

Detected indi properties
looking for *.*.*
Connected to localhost on port 7624
Queried properties from *
SX CCD UltraStar.CONNECTION.CONNECT=On
SX CCD UltraStar.CONNECTION.DISCONNECT=Off
SX CCD UltraStar.DRIVER_INFO.DRIVER_NAME=SX CCD
SX CCD UltraStar.DRIVER_INFO.DRIVER_EXEC=indi_sx_ccd
SX CCD UltraStar.DRIVER_INFO.DRIVER_VERSION=1.15
SX CCD UltraStar.DRIVER_INFO.DRIVER_INTERFACE=6
SX CCD UltraStar.POLLING_PERIOD.PERIOD_MS=1000
SX CCD UltraStar.DEBUG.ENABLE=Off
SX CCD UltraStar.DEBUG.DISABLE=On
SX CCD UltraStar.CONFIG_PROCESS.CONFIG_LOAD=Off
SX CCD UltraStar.CONFIG_PROCESS.CONFIG_SAVE=Off
SX CCD UltraStar.CONFIG_PROCESS.CONFIG_DEFAULT=Off
SX CCD UltraStar.CONFIG_PROCESS.CONFIG_PURGE=Off
SX CCD UltraStar.ACTIVE_DEVICES.ACTIVE_TELESCOPE=Telescope Simulator
SX CCD UltraStar.ACTIVE_DEVICES.ACTIVE_ROTATOR=Rotator Simulator
SX CCD UltraStar.ACTIVE_DEVICES.ACTIVE_FOCUSER=Focuser Simulator
SX CCD UltraStar.ACTIVE_DEVICES.ACTIVE_FILTER=CCD Simulator
SX CCD UltraStar.ACTIVE_DEVICES.ACTIVE_SKYQUALITY=SQM
SX CCD UltraStar.CCD_EXPOSURE.CCD_EXPOSURE_VALUE=58
SX CCD UltraStar.CCD_ABORT_EXPOSURE.ABORT=Off
SX CCD UltraStar.CCD_FRAME.X=0
SX CCD UltraStar.CCD_FRAME.Y=0
SX CCD UltraStar.CCD_FRAME.WIDTH=1392
SX CCD UltraStar.CCD_FRAME.HEIGHT=1040
SX CCD UltraStar.CCD_FRAME_RESET is write-only
SX CCD UltraStar.CCD_BINNING.HOR_BIN=1
SX CCD UltraStar.CCD_BINNING.VER_BIN=1
SX CCD UltraStar.FITS_HEADER is write-only
SX CCD UltraStar.CCD_CAPTURE_FORMAT.INDI_MONO=On
SX CCD UltraStar.CCD_TRANSFER_FORMAT.FORMAT_FITS=On
SX CCD UltraStar.CCD_TRANSFER_FORMAT.FORMAT_NATIVE=Off
SX CCD UltraStar.CCD_TRANSFER_FORMAT.=Off
SX CCD UltraStar.CCD_INFO.CCD_MAX_X=1392
SX CCD UltraStar.CCD_INFO.CCD_MAX_Y=1040
SX CCD UltraStar.CCD_INFO.CCD_PIXEL_SIZE=6.44921875
SX CCD UltraStar.CCD_INFO.CCD_PIXEL_SIZE_X=6.44921875
SX CCD UltraStar.CCD_INFO.CCD_PIXEL_SIZE_Y=6.44921875
SX CCD UltraStar.CCD_INFO.CCD_BITSPERPIXEL=16
SX CCD UltraStar.CCD_COMPRESSION.INDI_ENABLED=Off
SX CCD UltraStar.CCD_COMPRESSION.INDI_DISABLED=On
sending enableBLOB SX CCD UltraStar.CCD1
SX CCD UltraStar.TELESCOPE_TIMED_GUIDE_NS.TIMED_GUIDE_N=0
SX CCD UltraStar.TELESCOPE_TIMED_GUIDE_NS.TIMED_GUIDE_S=0
SX CCD UltraStar.TELESCOPE_TIMED_GUIDE_WE.TIMED_GUIDE_W=0
SX CCD UltraStar.TELESCOPE_TIMED_GUIDE_WE.TIMED_GUIDE_E=0
SX CCD UltraStar.CCD_FRAME_TYPE.FRAME_LIGHT=On
SX CCD UltraStar.CCD_FRAME_TYPE.FRAME_BIAS=Off
SX CCD UltraStar.CCD_FRAME_TYPE.FRAME_DARK=Off
SX CCD UltraStar.CCD_FRAME_TYPE.FRAME_FLAT=Off
SX CCD UltraStar.SCOPE_INFO.FOCAL_LENGTH=0
SX CCD UltraStar.SCOPE_INFO.APERTURE=0
SX CCD UltraStar.WCS_CONTROL.WCS_ENABLE=Off
SX CCD UltraStar.WCS_CONTROL.WCS_DISABLE=On
SX CCD UltraStar.UPLOAD_MODE.UPLOAD_CLIENT=On
SX CCD UltraStar.UPLOAD_MODE.UPLOAD_LOCAL=Off
SX CCD UltraStar.UPLOAD_MODE.UPLOAD_BOTH=Off
SX CCD UltraStar.UPLOAD_SETTINGS.UPLOAD_DIR=/home/admin
SX CCD UltraStar.UPLOAD_SETTINGS.UPLOAD_PREFIX=IMAGE_XXX
SX CCD UltraStar.CCD_FAST_TOGGLE.INDI_ENABLED=Off
SX CCD UltraStar.CCD_FAST_TOGGLE.INDI_DISABLED=On
SX CCD UltraStar.CCD_FAST_COUNT.FRAMES=1
SX CCD UltraStar.ACTIVE_DEVICES.ACTIVE_TELESCOPE=Telescope Simulator
SX CCD UltraStar.ACTIVE_DEVICES.ACTIVE_ROTATOR=Rotator Simulator
SX CCD UltraStar.ACTIVE_DEVICES.ACTIVE_FOCUSER=Focuser Simulator
SX CCD UltraStar.ACTIVE_DEVICES.ACTIVE_FILTER=CCD Simulator
SX CCD UltraStar.ACTIVE_DEVICES.ACTIVE_SKYQUALITY=SQM
Telescope Simulator.CONNECTION.CONNECT=On
Telescope Simulator.CONNECTION.DISCONNECT=Off
Telescope Simulator.DRIVER_INFO.DRIVER_NAME=Telescope Simulator
Telescope Simulator.DRIVER_INFO.DRIVER_EXEC=indi_simulator_telescope
Telescope Simulator.DRIVER_INFO.DRIVER_VERSION=1.0
Telescope Simulator.DRIVER_INFO.DRIVER_INTERFACE=5
Telescope Simulator.POLLING_PERIOD.PERIOD_MS=250
Telescope Simulator.DEBUG.ENABLE=Off
Telescope Simulator.DEBUG.DISABLE=On
Telescope Simulator.CONFIG_PROCESS.CONFIG_LOAD=Off
Telescope Simulator.CONFIG_PROCESS.CONFIG_SAVE=Off
Telescope Simulator.CONFIG_PROCESS.CONFIG_DEFAULT=Off
Telescope Simulator.CONFIG_PROCESS.CONFIG_PURGE=Off
Telescope Simulator.CONNECTION_MODE.CONNECTION_SERIAL=On
Telescope Simulator.CONNECTION_MODE.CONNECTION_TCP=Off
Telescope Simulator.DEVICE_PORT.PORT=/dev/ttyUSB0
Telescope Simulator.DEVICE_BAUD_RATE.9600=On
Telescope Simulator.DEVICE_BAUD_RATE.19200=Off
Telescope Simulator.DEVICE_BAUD_RATE.38400=Off
Telescope Simulator.DEVICE_BAUD_RATE.57600=Off
Telescope Simulator.DEVICE_BAUD_RATE.115200=Off
Telescope Simulator.DEVICE_BAUD_RATE.230400=Off
Telescope Simulator.DEVICE_AUTO_SEARCH.INDI_ENABLED=On
Telescope Simulator.DEVICE_AUTO_SEARCH.INDI_DISABLED=Off
Telescope Simulator.DEVICE_PORT_SCAN.Scan Ports=Off
Telescope Simulator.ACTIVE_DEVICES.ACTIVE_GPS=GPS Simulator
Telescope Simulator.ACTIVE_DEVICES.ACTIVE_DOME=Dome Simulator
Telescope Simulator.DOME_POLICY.DOME_IGNORED=On
Telescope Simulator.DOME_POLICY.DOME_LOCKS=Off
Telescope Simulator.TELESCOPE_INFO.TELESCOPE_APERTURE=10
Telescope Simulator.TELESCOPE_INFO.TELESCOPE_FOCAL_LENGTH=10
Telescope Simulator.TELESCOPE_INFO.GUIDER_APERTURE=120
Telescope Simulator.TELESCOPE_INFO.GUIDER_FOCAL_LENGTH=900
Telescope Simulator.SCOPE_CONFIG_NAME.SCOPE_CONFIG_NAME=indi-allsky
Telescope Simulator.MOUNT_TYPE is write-only
Telescope Simulator.SIM_PIER_SIDE is write-only
Telescope Simulator.MOUNT_MODEL is write-only
Telescope Simulator.MOUNT_AXES.PRIMARY=0
Telescope Simulator.MOUNT_AXES.SECONDARY=41.74069976806640625
Telescope Simulator.FLIP_HA is write-only
Telescope Simulator.ON_COORD_SET.TRACK=On
Telescope Simulator.ON_COORD_SET.SLEW=Off
Telescope Simulator.ON_COORD_SET.SYNC=Off
Telescope Simulator.EQUATORIAL_EOD_COORD.RA=9.3650381921692815723
Telescope Simulator.EQUATORIAL_EOD_COORD.DEC=41.74069976806640625
Telescope Simulator.TELESCOPE_ABORT_MOTION.ABORT=Off
Telescope Simulator.TELESCOPE_TRACK_MODE.TRACK_SIDEREAL=On
Telescope Simulator.TELESCOPE_TRACK_MODE.TRACK_SOLAR=Off
Telescope Simulator.TELESCOPE_TRACK_MODE.TRACK_LUNAR=Off
Telescope Simulator.TELESCOPE_TRACK_MODE.TRACK_CUSTOM=Off
Telescope Simulator.TELESCOPE_TRACK_STATE.TRACK_ON=Off
Telescope Simulator.TELESCOPE_TRACK_STATE.TRACK_OFF=On
Telescope Simulator.TELESCOPE_TRACK_RATE.TRACK_RATE_RA=15.04106717867020393
Telescope Simulator.TELESCOPE_TRACK_RATE.TRACK_RATE_DE=0
Telescope Simulator.TELESCOPE_MOTION_NS.MOTION_NORTH=Off
Telescope Simulator.TELESCOPE_MOTION_NS.MOTION_SOUTH=Off
Telescope Simulator.TELESCOPE_MOTION_WE.MOTION_WEST=Off
Telescope Simulator.TELESCOPE_MOTION_WE.MOTION_EAST=Off
Telescope Simulator.TELESCOPE_REVERSE_MOTION.REVERSE_NS=Off
Telescope Simulator.TELESCOPE_REVERSE_MOTION.REVERSE_WE=Off
Telescope Simulator.TELESCOPE_SLEW_RATE.1x=Off
Telescope Simulator.TELESCOPE_SLEW_RATE.2x=Off
Telescope Simulator.TELESCOPE_SLEW_RATE.3x=Off
Telescope Simulator.TELESCOPE_SLEW_RATE.4x=On
Telescope Simulator.TARGET_EOD_COORD.RA=0
Telescope Simulator.TARGET_EOD_COORD.DEC=0
Telescope Simulator.TIME_UTC.UTC=2024-03-09T21:44:16
Telescope Simulator.TIME_UTC.OFFSET=-5.00
Telescope Simulator.GEOGRAPHIC_COORD.LAT=41.74069976806640625
Telescope Simulator.GEOGRAPHIC_COORD.LONG=-71.30860137939453125
Telescope Simulator.GEOGRAPHIC_COORD.ELEV=0
Telescope Simulator.TELESCOPE_PARK.PARK=On
Telescope Simulator.TELESCOPE_PARK.UNPARK=Off
Telescope Simulator.TELESCOPE_PARK_POSITION.PARK_HA=0
Telescope Simulator.TELESCOPE_PARK_POSITION.PARK_DEC=41.74069976806640625
Telescope Simulator.TELESCOPE_PARK_OPTION.PARK_CURRENT=Off
Telescope Simulator.TELESCOPE_PARK_OPTION.PARK_DEFAULT=Off
Telescope Simulator.TELESCOPE_PARK_OPTION.PARK_WRITE_DATA=Off
Telescope Simulator.TELESCOPE_PARK_OPTION.PARK_PURGE_DATA=Off
Telescope Simulator.TELESCOPE_PIER_SIDE.PIER_WEST=Off
Telescope Simulator.TELESCOPE_PIER_SIDE.PIER_EAST=On
Telescope Simulator.APPLY_SCOPE_CONFIG.SCOPE_CONFIG1=On
Telescope Simulator.APPLY_SCOPE_CONFIG.SCOPE_CONFIG2=Off
Telescope Simulator.APPLY_SCOPE_CONFIG.SCOPE_CONFIG3=Off
Telescope Simulator.APPLY_SCOPE_CONFIG.SCOPE_CONFIG4=Off
Telescope Simulator.APPLY_SCOPE_CONFIG.SCOPE_CONFIG5=Off
Telescope Simulator.APPLY_SCOPE_CONFIG.SCOPE_CONFIG6=Off
Telescope Simulator.USEJOYSTICK.ENABLE=Off
Telescope Simulator.USEJOYSTICK.DISABLE=On
Telescope Simulator.SNOOP_JOYSTICK.SNOOP_JOYSTICK_DEVICE=Joystick
Telescope Simulator.TELESCOPE_TIMED_GUIDE_NS.TIMED_GUIDE_N=0
Telescope Simulator.TELESCOPE_TIMED_GUIDE_NS.TIMED_GUIDE_S=0
Telescope Simulator.TELESCOPE_TIMED_GUIDE_WE.TIMED_GUIDE_W=0
Telescope Simulator.TELESCOPE_TIMED_GUIDE_WE.TIMED_GUIDE_E=0
Telescope Simulator.GUIDE_RATE.GUIDE_RATE_WE=0.5
Telescope Simulator.GUIDE_RATE.GUIDE_RATE_NS=0.5
Telescope Simulator.ACTIVE_DEVICES.ACTIVE_GPS=GPS Simulator
Telescope Simulator.ACTIVE_DEVICES.ACTIVE_DOME=Dome Simulator
Telescope Simulator.DOME_POLICY.DOME_IGNORED=On
Telescope Simulator.DOME_POLICY.DOME_LOCKS=Off
Telescope Simulator.TELESCOPE_INFO.TELESCOPE_APERTURE=10
Telescope Simulator.TELESCOPE_INFO.TELESCOPE_FOCAL_LENGTH=10
Telescope Simulator.TELESCOPE_INFO.GUIDER_APERTURE=120
Telescope Simulator.TELESCOPE_INFO.GUIDER_FOCAL_LENGTH=900
Telescope Simulator.SCOPE_CONFIG_NAME.SCOPE_CONFIG_NAME=indi-allsky
Telescope Simulator.USEJOYSTICK.ENABLE=Off
Telescope Simulator.USEJOYSTICK.DISABLE=On
Telescope Simulator.SNOOP_JOYSTICK.SNOOP_JOYSTICK_DEVICE=Joystick
Telescope Simulator.MOUNT_TYPE is write-only
Telescope Simulator.SIM_PIER_SIDE is write-only
Telescope Simulator.MOUNT_MODEL is write-only
Telescope Simulator.MOUNT_AXES.PRIMARY=0
Telescope Simulator.MOUNT_AXES.SECONDARY=41.74069976806640625
Telescope Simulator.FLIP_HA is write-only

libcamera: not detected

libcamera packages
ii  libcamera-apps                       1.2.1-1                          arm64        libcamera-apps
ii  libcamera-tools                      0~git20230720+bde9b04f-1         arm64        complex camera support library (tools)
ii  libcamera0:arm64                     0~git20230720+bde9b04f-1         arm64        complex camera support library
ii  python3-libcamera                    0~git20230720+bde9b04f-1         arm64        complex camera support library (Python 3 bindings)

libcamera cameras
libcamera-hello: /usr/bin/libcamera-hello
No cameras available!

python packages
ii  libpython3-dev:arm64                 3.9.2-3                          arm64        header files and a static library for Python (default)
ii  libpython3-stdlib:arm64              3.9.2-3                          arm64        interactive high-level object-oriented language (default python3 version)
ii  libpython3.9:arm64                   3.9.2-1                          arm64        Shared Python runtime library (version 3.9)
ii  libpython3.9-dev:arm64               3.9.2-1                          arm64        Header files and a static library for Python (v3.9)
ii  libpython3.9-minimal:arm64           3.9.2-1                          arm64        Minimal subset of the Python language (version 3.9)
ii  libpython3.9-stdlib:arm64            3.9.2-1                          arm64        Interactive high-level object-oriented language (standard library, version 3.9)
ii  python-apt-common                    2.2.1                            all          Python interface to libapt-pkg (locales)
ii  python-is-python3                    3.9.2-1                          all          symlinks /usr/bin/python to python3
ii  python-pip-whl                       20.3.4-4+rpt1+deb11u1            all          Python package installer (pip wheels)
ii  python3                              3.9.2-3                          arm64        interactive high-level object-oriented language (default python3 version)
ii  python3-appdirs                      1.4.4-1                          all          determining appropriate platform-specific directories (Python 3)
ii  python3-apt                          2.2.1                            arm64        Python 3 interface to libapt-pkg
ii  python3-astroid                      2.5.1-1                          all          rebuild a new abstract syntax tree from Python's AST (Python3)
ii  python3-asttokens                    2.0.4-1                          all          annotate Python asbtract syntax trees with code references (Python 3)
ii  python3-automationhat                0.2.0                            all          Python library for the Pimoroni Automation HAT
ii  python3-blinker                      1.4+dfsg1-0.3                    all          fast, simple object-to-object and broadcast signaling library
ii  python3-blinkt                       0.1.2                            all          Python library for the Pimoroni Blinkt!
ii  python3-bs4                          4.9.3-1                          all          error-tolerant HTML parser for Python 3
ii  python3-buttonshim                   0.0.2                            all          Python library for the Pimoroni Button SHIM
ii  python3-cairo:arm64                  1.16.2-4+b2                      arm64        Python3 bindings for the Cairo vector graphics library
ii  python3-cap1xxx                      0.1.3                            all          python library designed to drive various Microchip CAP1xxx touch ICs.
ii  python3-certifi                      2020.6.20-1                      all          root certificates for validating SSL certs and verifying TLS hosts (python3)
ii  python3-cffi-backend:arm64           1.14.5-1                         arm64        Foreign Function Interface for Python 3 calling C code - runtime
ii  python3-chardet                      4.0.0-1                          all          universal character encoding detector for Python3
ii  python3-click                        7.1.2-1                          all          Wrapper around optparse for command line utilities - Python 3.x
ii  python3-colorama                     0.4.4-1                          all          Cross-platform colored terminal text in Python - Python 3.x
ii  python3-colorzero                    1.1                              all          Construct, convert, and manipulate colors in a Pythonic manner.
ii  python3-cryptography                 3.3.2-1                          arm64        Python library exposing cryptographic recipes and primitives (Python 3)
ii  python3-cups:arm64                   2.0.1-4+b1                       arm64        Python3 bindings for CUPS
ii  python3-cupshelpers                  1.5.14-1                         all          Python utility modules around the CUPS printing system
ii  python3-dbus                         1.2.16-5                         arm64        simple interprocess messaging system (Python 3 interface)
ii  python3-debconf                      1.5.77                           all          interact with debconf from Python 3
ii  python3-dev                          3.9.2-3                          arm64        header files and a static library for Python (default)
ii  python3-distlib                      0.3.2+really+0.3.1-0.1           all          low-level components of python distutils2/packaging
ii  python3-distro                       1.5.0-1                          all          Linux OS platform information API
ii  python3-distutils                    3.9.2-1                          all          distutils package for Python 3.x
ii  python3-docutils                     0.16+dfsg-4                      all          text processing system for reStructuredText (implemented in Python 3)
ii  python3-drumhat                      0.1.0                            all          Python library for the Pimoroni Drum HAT.
ii  python3-envirophat                   1.0.0                            all          Python library for the Pimoroni Enviro pHAT.
ii  python3-explorerhat                  0.4.2                            all          python library designed to control the Explorer HAT and pHAT.
ii  python3-filelock                     3.0.12-2                         all          platform independent file locking module
ii  python3-flask                        1.1.2-2+deb11u1                  all          micro web framework based on Werkzeug and Jinja2 - Python 3.x
ii  python3-fourletterphat               0.1.0                            all          python library for the Pimoroni Four Letter pHAT
ii  python3-gi                           3.38.0-2                         arm64        Python 3 bindings for gobject-introspection libraries
ii  python3-gi-cairo                     3.38.0-2                         arm64        Python 3 Cairo bindings for the GObject library
ii  python3-gpiozero                     1.6.2-1                          all          Simple API for controlling devices attached to a Pi's GPIO pins.
ii  python3-html5lib                     1.1-3                            all          HTML parser/tokenizer based on the WHATWG HTML5 specification
ii  python3-idna                         2.10-1                           all          Python IDNA2008 (RFC 5891) handling (Python 3)
ii  python3-importlib-metadata           1.6.0-2                          all          library to access the metadata for a Python package - Python 3.x
ii  python3-isort                        5.6.4-1                          all          library for sorting Python imports (Python 3)
ii  python3-itsdangerous                 1.1.0-3                          all          Various helpers to pass trusted data to untrusted environment - Python 3.x
ii  python3-jedi                         0.18.0-1                         all          autocompletion tool for Python 3
ii  python3-jinja2                       2.11.3-1                         all          small but fast and easy to use stand-alone template engine
ii  python3-jwt                          1.7.1-2                          all          Python 3 implementation of JSON Web Token
ii  python3-kms++                        0~git20221104~171243+adc05b6-1   arm64        C++ library for kernel mode setting - python3 bindings
ii  python3-lazy-object-proxy            1.5.2-1                          arm64        Python 3 fast and thorough lazy object proxy
ii  python3-ldb                          2:2.2.3-2~deb11u2                arm64        Python 3 bindings for LDB
ii  python3-lib2to3                      3.9.2-1                          all          Interactive high-level object-oriented language (lib2to3)
ii  python3-libcamera                    0~git20230720+bde9b04f-1         arm64        complex camera support library (Python 3 bindings)
ii  python3-logilab-common               1.8.1-1                          all          useful miscellaneous modules used by Logilab projects (Python3)
ii  python3-lxml:arm64                   4.6.3+dfsg-0.1+deb11u1           arm64        pythonic binding for the libxml2 and libxslt libraries
ii  python3-markupsafe                   1.1.1-1+b3                       arm64        HTML/XHTML/XML string library for Python 3
ii  python3-mccabe                       0.6.1-3                          all          Python code complexity checker (Python 3)
ii  python3-microdotphat                 0.2.1                            all          Python library for the Pimoroni Micro Dot pHAT.
ii  python3-minimal                      3.9.2-3                          arm64        minimal subset of the Python language (default python3 version)
ii  python3-more-itertools               4.2.0-3                          all          library with routines for operating on iterables, beyond itertools (Python 3)
ii  python3-mote                         0.0.4                            all          python library designed to control Mote.
ii  python3-motephat                     0.0.3                            all          Python library for the Pimoroni Mote pHAT
ii  python3-mypy                         0.812-1                          arm64        public modules for mypy (Python 3)
ii  python3-mypy-extensions              0.4.3-2                          all          Experimental type system extensions for mypy typechecker (Python 3)
ii  python3-numpy                        1:1.19.5-1                       arm64        Fast array facility to the Python 3 language
ii  python3-oauthlib                     3.1.0-2                          all          generic, spec-compliant implementation of OAuth for Python3
ii  python3-opengl                       3.1.5+dfsg-1                     all          Python bindings to OpenGL (Python 3)
ii  python3-openssl                      20.0.1-1                         all          Python 3 wrapper around the OpenSSL library
ii  python3-pantilthat                   0.0.7                            all          Python library for the Pimoroni Pan-Tilt HAT
ii  python3-parso                        0.8.1-1                          all          Python parser that supports error recovery - Python 3.X
ii  python3-pexpect                      4.8.0-2                          all          Python 3 module for automating interactive applications
ii  python3-pgzero                       1.2.post4+dfsg-2                 all          Zero-boilerplate games programming framework based on Pygame (Python 3)
ii  python3-phatbeat                     0.1.1                            all          Python library for the Pimoroni pHAT BEAT.
ii  python3-pianohat                     0.1.0                            all          Python library for the Pimoroni Piano HAT.
ii  python3-picamera2                    0.3.12-2                         all          Libcamera based python library
ii  python3-pidng                        4.0.9-1                          arm64        Create Adobe DNG RAW files using Python (Python 3)
ii  python3-piexif                       1.1.3-1                          all          Exif manipulation with pure python script (Python 3)
ii  python3-piglow                       1.2.5                            all          Python library for the Pimoroni PiGlow.
ii  python3-pigpio                       1.79-1+rpt1                      all          Python module which talks to the pigpio daemon (Python 3)
ii  python3-pil:arm64                    8.1.2+dfsg-0.3+deb11u1           arm64        Python Imaging Library (Python3)
ii  python3-pip                          20.3.4-4+rpt1+deb11u1            all          Python package installer
ii  python3-pkg-resources                52.0.0-4                         all          Package Discovery and Resource Access using pkg_resources
ii  python3-prctl                        1.7-2+b4                         arm64        Python interface to the prctl() syscall (Python 3)
ii  python3-psutil                       5.8.0-1                          arm64        module providing convenience functions for managing processes (Python3)
ii  python3-ptyprocess                   0.7.0-1                          all          Run a subprocess in a pseudo terminal from Python 3
ii  python3-pygame                       1.9.6+dfsg-4+b1                  arm64        SDL bindings for games development (Python 3)
ii  python3-pygments                     2.7.1+dfsg-2.1                   all          syntax highlighting package written in Python 3
ii  python3-pyinotify                    0.9.6-1.3                        all          simple Linux inotify Python bindings
ii  python3-pyqt5                        5.15.2+dfsg-3                    arm64        Python 3 bindings for Qt5
ii  python3-pyqt5.sip                    12.8.1-1+b2                      arm64        runtime module for Python extensions using SIP
ii  python3-rainbowhat                   0.1.0                            all          Python library for the Pimoroni Rainbow HAT
ii  python3-renderpm:arm64               3.5.59-2                         arm64        python low level render interface
ii  python3-reportlab                    3.5.59-2                         all          ReportLab library to create PDF documents using Python3
ii  python3-reportlab-accel:arm64        3.5.59-2                         arm64        C coded extension accelerator for the ReportLab Toolkit
ii  python3-requests                     2.25.1+dfsg-2                    all          elegant and simple HTTP library for Python3, built for human beings
ii  python3-requests-oauthlib            1.0.0-1.1                        all          module providing OAuthlib auth support for requests (Python 3)
ii  python3-responses                    0.12.1-1                         all          Utility library for mocking out the requests Python 3 library
ii  python3-roman                        2.0.0-5                          all          module for generating/analyzing Roman numerals for Python 3
ii  python3-rpi.gpio                     0.7.0-0.2+b2                     arm64        Module to control Raspberry Pi GPIO channels (Python 3)
ii  python3-rtimulib                     7.2.1-6                          arm64        Versatile C++ and Python 9-dof, 10-dof and 11-dof IMU library (Python 3)
ii  python3-scrollphat                   0.0.7                            all          python library designed to control Scroll pHAT.
ii  python3-scrollphathd                 1.2.1                            all          Python library for the Pimoroni Scroll pHAT HD.
ii  python3-send2trash                   1.6.0~b1+git20210122.2eb3242-1   all          Python module for sending file to trash natively
ii  python3-sense-hat                    2.6.0-1                          all          Sense HAT python library (Python 3)
ii  python3-serial                       3.5~b0-1                         all          pyserial - module encapsulating access for the serial port
ii  python3-setuptools                   52.0.0-4                         all          Python3 Distutils Enhancements
ii  python3-simplejpeg                   1.6.4-1                          arm64        Simple package for fast JPEG encoding and decoding
ii  python3-simplejson                   3.17.2-1                         arm64        simple, fast, extensible JSON encoder/decoder for Python 3.x
ii  python3-six                          1.16.0-2                         all          Python 2 and 3 compatibility library (Python 3 interface)
ii  python3-skywriter                    0.0.7                            all          python library designed to control Skywriter.
ii  python3-smbc                         1.0.23-1+b1                      arm64        Python 3 bindings for the Samba client library
ii  python3-smbus:arm64                  4.2-1+b1                         arm64        Python 3 bindings for Linux SMBus access through i2c-dev
ii  python3-sn3218                       1.2.7                            all          python library to help control the SN3218 18-channel PWM LED driver.
ii  python3-soupsieve                    2.2.1-1                          all          modern CSS selector implementation for BeautifulSoup (Python 3)
ii  python3-spidev                       20200602~200721-1                arm64        Bindings for Linux SPI access through spidev (Python 3)
ii  python3-talloc:arm64                 2.3.1-2+b1                       arm64        hierarchical pool based memory allocator - Python3 bindings
ii  python3-tk:arm64                     3.9.2-1                          arm64        Tkinter - Writing Tk applications with Python 3.x
ii  python3-toml                         0.10.1-1                         all          library for Tom's Obvious, Minimal Language - Python 3.x
ii  python3-touchphat                    0.0.1                            all          python library designed to control Touch pHAT
ii  python3-twython                      3.8.2+dfsg-2                     all          Pure Python3 wrapper for the Twitter API
ii  python3-typed-ast                    1.4.2-1                          arm64        AST with PEP 484 type comments support
ii  python3-typing-extensions            3.7.4.3-1                        all          Backported and Experimental Type Hints for Python
ii  python3-unicornhathd                 0.0.4                            all          Python library for the Pimoroni Unicorn HAT HD.
ii  python3-urllib3                      1.26.5-1~exp1                    all          HTTP library with thread-safe connection pooling for Python3
ii  python3-v4l2                         0.3.2-1                          all          Python bindings for the v4l2 userspace api
ii  python3-venv                         3.9.2-3                          arm64        venv module for python3 (default python3 version)
ii  python3-virtualenv                   20.4.0+ds-2+deb11u1              all          Python virtual environment creator
ii  python3-webencodings                 0.5.1-2                          all          Python implementation of the WHATWG Encoding standard
ii  python3-werkzeug                     1.0.1+dfsg1-2+deb11u1            all          collection of utilities for WSGI applications (Python 3.x)
ii  python3-wheel                        0.34.2-1                         all          built-package format for Python
ii  python3-wrapt                        1.12.1-4+b1                      arm64        decorators, wrappers and monkey patching. - Python 3.x
ii  python3-yaml                         5.3.1-5                          arm64        YAML parser and emitter for Python3
ii  python3-zipp                         1.0.0-3                          all          pathlib-compatible Zipfile object wrapper - Python 3.x
ii  python3.9                            3.9.2-1                          arm64        Interactive high-level object-oriented language (version 3.9)
ii  python3.9-dev                        3.9.2-1                          arm64        Header files and a static library for Python (v3.9)
ii  python3.9-minimal                    3.9.2-1                          arm64        Minimal subset of the Python language (version 3.9)
ii  python3.9-venv                       3.9.2-1                          arm64        Interactive high-level object-oriented language (pyvenv binary, version 3.9)

Detected indi-allsky virtualenv
virtualenv python: Python 3.9.2
virtualenv PATH: /home/admin/indi-allsky/virtualenv/indi-allsky/bin:/usr/local/bin:/usr/bin:/bin
flask command: /home/admin/indi-allsky/virtualenv/indi-allsky/bin/flask

virtualenv python modules
alembic==1.12.0
apache-libcloud==3.8.0
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
asciitree==0.3.3
astroalign==2.4.2
astropy==5.3.4
astropy-healpix==1.0.0
astroscrappy==1.1.0
bcrypt==4.0.1
blinker==1.6.3
boto3==1.28.62
botocore==1.31.62
Bottleneck==1.3.7
cachetools==5.3.2
ccdproc==2.4.1
certifi==2023.7.22
cffi==1.16.0
charset-normalizer==3.3.0
circuitbreaker==1.4.0
click==8.1.7
cloudpickle==2.2.1
cryptography==42.0.4
Cython==3.0.3
dask==2023.9.3
dbus-python==1.3.2
ephem==4.1.4
fasteners==0.19
fish2pano==0.4.0
Flask==3.0.2
Flask-Login==0.6.3
Flask-Migrate==4.0.5
Flask-SQLAlchemy==3.1.1
Flask-WTF==1.2.1
fsspec==2023.9.2
google-api-core==2.12.0
google-api-python-client==2.117.0
google-auth==2.23.3
google-auth-httplib2==0.2.0
google-auth-oauthlib==1.2.0
google-cloud-core==2.3.3
google-cloud-storage==2.12.0
google-crc32c==1.5.0
google-resumable-media==2.6.0
googleapis-common-protos==1.61.0
greenlet==3.0.0
gunicorn==21.2.0
httplib2==0.22.0
idna==3.4
imageio==2.31.5
imageio-ffmpeg==0.4.9
importlib-metadata==6.8.0
inotify==0.2.10
is-safe-url==1.0
itsdangerous==2.1.2
Jinja2==3.1.2
jmespath==1.0.1
lazy_loader==0.3
locket==1.0.0
lxml==4.9.3
Mako==1.2.4
MarkupSafe==2.1.3
mysql-connector-python==8.1.0
networkx==3.1
nose==1.3.7
numcodecs==0.12.0
numpy==1.26.0
oauthlib==3.2.2
oci==2.121.0
opencv-python-headless==4.8.1.78
packaging==23.2
paho-mqtt==1.6.1
paramiko==3.3.1
partd==1.4.1
passlib==1.7.4
piexif==1.1.3
Pillow==10.0.1
pkg_resources==0.0.0
prettytable==3.9.0
protobuf==4.21.12
psutil==5.9.5
pyasn1==0.5.0
pyasn1-modules==0.3.0
pycparser==2.21
pycurl==7.45.2
pyerfa==2.0.0.3
pygifsicle==1.0.7
pyindi-client @ git+https://github.com/indilib/pyindi-client.git@674706fc4caca8c28ab60ada121b8554027a56c7
PyMySQL==1.1.0
PyNaCl==1.5.0
pyOpenSSL==23.3.0
pyparsing==3.1.1
python-dateutil==2.8.2
pytz==2024.1
PyYAML==6.0.1
rawpy==0.18.1
reproject==0.12.0
requests==2.31.0
requests-oauthlib==1.3.1
requests-toolbelt==1.0.0
rsa==4.9
s3transfer==0.7.0
scikit-image==0.22.0
scipy==1.11.3
semantic-version==2.10.0
sep==1.2.1
setuptools-rust==1.7.0
shapely==2.0.1
six==1.16.0
SQLAlchemy==2.0.21
support-developer==1.0.5
tifffile==2023.9.26
tomli==2.0.1
toolz==0.12.0
typing_extensions==4.8.0
uritemplate==4.1.1
urllib3==1.26.17
wcwidth==0.2.8
Werkzeug==3.0.1
WTForms==3.0.1
zarr==2.16.1
zipp==3.17.0

#################################
###     end support info      ###
#################################

Please note that the Starlight Xpress (SX) Oculus Pro uses the ICX267AL Sony SuperHAD interline CCD
https://www.sxccd.com/product/oculus-pro-all-sky-camera-150o/
Full spec below

I select INDI_SX_CCD when setting up the software. Should I have selected the INDI_SONY_CCD ??

  • CCD type: ICX267AL Sony SuperHAD interline CCD with low dark current and vertical anti-blooming.
  • CCD Full resolution pixel data: Pixel size: 4.65uM x 4.65uM, Image format: 1392 x 1040 pixels
  • CCD Image area: 6.4mm (Horizontal) x 4.75mm (Vertical).
  • CCD quality: Grade 1
  • Spectral Response: QE max at 520nM (~50%), 30% at 420nM and 670nM.
  • Readout Noise: Less than 10 electrons RMS – typically only 7 electrons.
  • Full-well capacity: Greater than 15,000 e- (unbinned)
  • Anti-blooming: Overload margin greater than 1000x.
  • Dark current: Less than 0.1 electrons/second @ + 10C ambient.
  • Data format: 16 bits.
  • System gain: 0.3 electrons per ADU
  • Lens details: 2.5mm f/1.2 150 degree ‘Fish Eye’
  • Computer Interface: Built-in USB 2.0 compatible interface.
  • Image download time: Typically 0.6 seconds at full resolution using USB 2.0.
  • Power requirements: Camera is USB powered, dew heater requires 12v at 1 amp max.
  • Input connection: ‘Mini B’ USB socket for USB2.0 and a 2.1mm jack for +12v heater supply.
  • Cooling system: Ambient air cooling.
  • Viewing dome: High quality polycarbonate hemisphere (replaceable)
  • Overall size: 150mm tall x 95mm diameter

I tried a new setup with the INDI_SONY_CCD selected and the hardware camera was not recognized. No connection possible.

So I went back to the original setup with INDI_SX_CCD.

Let me know if you see anything critical in the diagnostic printout associated with the previous INDI_SX_CCD setup shown in the above support_info.sh.

The indi_sony_ccd driver is only for Sony DSLRs.

Everything looks fairly normal. I doubt the issue is related to the transition between day and night. Most cameras that have issues with the transition are caused by the gain being changed. Most (if not all) Starlight Xpress cameras have a fixed gain, so that is not changed. The only other "change" between day and night is the bin mode is set again, but if both modes are bin1, there is no change (between day and night).

As a guess, you might be running into a physical problem with the USB cable. With temperature changes, it can exacerbate problems with connectors in the USB connector to the camera. Have you tried changing the USB cable?

The USB cable on SX Oculus comes through a weather seal so I can't replace the cable. I have tried reseating it at the Pi. Will report any change in pattern.

I could also run some simple code to periodically access camera at wide intervals to look for communications issues?

I could try running daytime camera at a much lower cadence? So I won't acquire many pictures but it would keep camera alive?

Note that simply stopping and starting indiallsky service, after it hangs at the beginning of the night, always results in immediate successful camera acquisition without any reported misreads at beginning.

I have reviewed the indilib docs a little more closely. I have always considered when the exposure control is in IPS_ALERT state, that the camera was effectively in a failed status. However, that is not really true. It just means the previous command/action failed.

Let me think about this to see if I can integrate this concept.

Merged #1193 as an attempt to continue taking images after the camera exposure reports ALERT.

thanks for feedback on the way ALERTS are handled. Should I monitor issue here or on the other 1193 issue chain for updates on that?

On the hardware side; Tonight I have determined that the SX Oculus Camera USB connection management may behave differently depending on the condition of a separate 12V DC line that is also fed into the Oculus unit. I am going to start running with the 12V DC connection improved (I made a better connector) to see if that influences the initial ALERT state at nightfall. The DC supply is for the dew heater, but it appears it could possibly affect the USB behavior based on my tests plugging/unplugging the 12V line.

(Before working on the 12V line I checked the USB connector on the Oculus. The USB connector showed no issues with mechanical flexing/slipping causing electrical intermittency or poor connections.)

We'll keep the discussion here.

The potential issue that I saw with 12V heater supply possibly not being connected properly and a cross interaction with USB was a red herring. Everything is connected properly. I still see the Camera logic hang at the start of the night’s observation.

I tried saving images once an hour during the day. This didn’t change an ALERT being issues at the start of the night (smaller number of seconds in message box), and the acquisition hanging at that point.

I will try a regular daytime schedule, images once a minute and see how this works when the night transition occurs.

Thanks, Rick

Did you upgrade the code yet for the fix I attempted? Its fine if it did not work, I have another idea if that fix did not work.

Aaron, apologies was away from the lab. Will test new code now (assume main branch) and report after a couple of cycles. Assume I should go back to having it suspended during day and only run at night.

(My tests with older code this week where it was running at 1 minute intervals both day and night were stable. As discussed the issue was occurring when I have larger steps in day or if camera is fully suspended during day)

@aaronwmorris the new code logic is working. Image capture slept during day and restarted successfully two days running. Thank you so much for working on this issue.
I really like the new changes to status reporting, menus and state change count down.

(A separate question I should ask in a new thread, if it has an interesting answer… did you design any of the databases you use inside your code to also be potentially read-only/accessed externally? At present I create a new external database with summary characteristics of the FITS files that have been copied to the GCP based on my own processing. I realize you have already calculated/recorded this information in your own internal database files (e.g. Exposure time, number of stars identified, moon position, etc.…). Is there a logical way to serve this data externally to people analysing the FITS files? No big deal if this is not part of your design, but I would hate to think I have simply misunderstood your intended data export interface.

Another question which I should post in discussion?? Has anyone had success creating a pipeline which automatically determines the full sky coordinates for each all sky image so any desired object can then be located within the images? Where is the right place to have that discussion?)

The database is a standard Sqlite database. The database is configured with the Write Ahead Logging [WAL] journal so it may be accessed concurrently, for both reads and writes, with almost no impact. Since it is Sqlite, it is not really capable of being accessed remotely. I have a page that has some basic query info: https://github.com/aaronwmorris/indi-allsky/wiki/SQL-queries

It is also possible to configure the database as MariaDB/MySQL which could be accessed remotely. I have a procedure for converting an existing Sqlite DB to MariaDB. https://github.com/aaronwmorris/indi-allsky/wiki/Database-conversion

As for the FITS files, they also have most of the standard FITS headers. Exposure, latitude, longitude, etc so it should be easy enough to calculate some standard astrometric info. I could look into adding things like stars if it would be useful.