Queue sending to a custom URL and send every x minutes
Closed this issue · 8 comments
The system already, from what I understand, will queue if offline. Instead of sending every point to a custom URL, queue points and send every X number of minutes. This is similar to the concept of the Auto-Send.
This would essentially be a way to record an entire journey, but not to be constantly sending it.
And this appears to be not working presently. Today, for instance, I had the app up on the screens with functioning GPS but data communication offline and no data reports were ever uploaded upon return to data communication.
For my understanding of the goal - is it basically to have more reliable sending of Custom URL points?
Taking the auto send concept, you could have Custom URL logging enabled if you wanted, or Custom URL auto-send, or both enabled. Then every 60 minutes, or when you press stop (the auto send options), or when you use the upload menu, the Custom URL points are re-sent to the target server. This way, if some of the previously queued points got lost or never sent, at least the auto-send will re-send it to the server. Does that sound like the goal?
This is somewhat similar to how the OpenGTS feature operates right now. OpenGTS logging will queue it up right away. Then every 60 minutes, or when pressing stop, or when the user uses the 'upload' menu, the code reads the GPX file and reconstructs the OpenGTS URLs, then re-sends it to the server.
I could think of something similar to do with Custom URL 'auto send'. There are a few things different with a Custom URL 'auto send'.
First one is, on the target server, you would need to be able to handle 'duplicates'. For example, use the timestamp or epoch sent when you receive the URL. In other words on the server side the write operation needs to be idempotent.
Now the second and main one, it's the storage of the Custom URL data. Right now, the URL is constructed on t he fly, and queued up right away. There's a lot of types of data that could be sent via Custom URL.
Here's the list so far
%lat
%lon
%sat
%desc
%alt
%acc
%dir
%prov
%spd
%timestamp
%timeoffset
%time
%date
%starttimestamp
%batt
%aid
%ser
%act
%filename
%profile
%hdop
%vdop
%pdop
%dist
Unlike OpenGTS, it's not possible to reconstruct a Custom URL from a GPX file, because most of this data types shown above is simply not stored in GPX format. I would definitely need to store all these values in a separate data 'store'. So that when 60 minutes are up, or when user presses stop, or when the user chooses to send a file manually, these values are available.
So... thinking about this, what about CSV. The CSV file format is pretty much a data store. And its columns look very close to Custom URL already. Here are its columns
time,lat,lon,elevation,accuracy,bearing,speed,satellites,provider,hdop,vdop,pdop,geoidheight,ageofdgpsdata,dgpsid,activity,battery,annotation
What I could do is when auto-sending, just re-read the CSV file and reconstruct the URL and send its values out. I would need to add a few more columns to the existing CSV format, it should be possible.
That will mean, the third thing - if a user wants Custom URL auto-sending, I will need to force CSV file writing to also be enabled. And when they choose to 'upload' a file to Custom URL, I'd present the CSV files.
That's not a problem. It means a local backup that I could replay into the server
Agreed, the goal is more reliable sending of Custom URL points when the device is frequently offline but recording.
What you describe sounds reasonable with csv on the device and with the remote recipient responsible for handling duplicates.
I wonder why queuing up used to work more reliably in the past and now less so recently; before fdroid, I had sent my gpslogger in areas frequently out of data service, and it did an admirable job of reporting the route when it reached data service again.
I think the unreliability is due to the increasing throttling that Android has been applying in each new version of the OS. Each new release there are additional restrictions on background processing, and location data. Both of which this app makes use of. Their aim is to improve battery life as well as unrestricted data siphoning, so they do this by throttling or restricting background tasks from running, which actually is understandable.
That in turn affects GPSLogger, because the background tasks used as part of the queueing and sending can be killed off and there isn't a recovery from that.
I'm also using quite an old job queueing library, and I ought to be moving onto the newer libraries for this, called JobScheduler, but even on that page are big red warnings:
Note: Beginning with API 30 (Build.VERSION_CODES.R), JobScheduler will throttle runaway applications. Calling schedule(android.app.job.JobInfo) and other such methods with very high frequency can have a high cost and so, to make sure the system doesn't get overwhelmed, JobScheduler will begin to throttle apps, regardless of target SDK version.
Note: Scheduling a job can have a high cost, even if it's rescheduling the same job and the job didn't execute, especially on platform versions before version Build.VERSION_CODES.Q. As such, the system may throttle calls to this API if calls are made too frequently in a short amount of time.
I will spend some time exploring the CSV file aspect since there's more data to manage now. It's likely a library will be needed, probably the Apache Common CSV one.
Custom URL can now be used as an auto sender, such as every 60 minutes or when stop is pressed.
It works by storing the values in the CSV file, so as a result the CSV file has gotten a bunch of new fields added.
You can also 'upload' the CSV file to the Custom URL sender, and the values will be read and converted to URLs and sent to the server. This should serve as a way to mitigate any lost Custom URLs or do away with offline queueing problems.
You will need to ensure that the Custom URL server can receive the same log lines multiple times.
I've put a test APK here: https://github.com/mendhak/gpslogger/releases/tag/v119-rc1
As usual, this APK is not compatible with the F-Droid version.
If you install this APK you will lose all GPSLogger data from the F-Droid version.
v119 is now on F-Droid and in the releases. I'm going around and closing some issues.