cpfair/tapiriik

Stuck on "Queueing" screen for the last few days

Opened this issue Β· 60 comments

Hello there πŸ‘‹ ,

I tried to use tapiriik for the very first time to migrate my workouts from endomondo to strava.

I started 3 days ago and UI stuck here:
Screenshot 2020-11-02 at 23 07 43

I had empty strava account before starting the migration and it's still empty after 3 days. Any ideas what's going on? Any place where I could check some logs?

Exact same use case, exact same problem here.

Same here. I've added also dropbox to check out if it can download my trainings from endomondo, a new folder in my dropbox have been added but no gpx files there.
Not sure if tapiriik is working at all.

Same issue here too.

Same.

It's worth noting that Endomondo has just announced that it's shutting down, so their servers might be very busy these days - I also tried their "native" sync to mapmyrun.com 2 days ago & there's still nothing there. Moreover, I requested a download of all my data yesterday - still no update - both actions are said to take a few days though.

So, it makes sense to have things slow, although I admit that a better status reflection would be nice (so far, the last message on tapiriik's twitter is from 1 year ago).

Also, I guess it doesn't help that Strava is said to force some rate limits themselves.
FWIW though, a simple (1 activity) sync from Samsung Health (via their app) to Strava went fine earlier today.

Thanks nevertheless for this great service! First-time user here, would love to see it works :)

Same here - Synching from Strava to Sporttracks, which have been working several weeks, Last week automatic synch was disabled for no obvious reason. Sync account is now restored, but nothing has synchronized between accounts since 29th of October.

Same. I disabled Strava and the synchronization between the rest of the services started without problem.

Also same. Trying to move data from soon-to-be-killed Endomondo to Strava like thousands of other people I imagine.
Not sure if queueing is down to load on Endomondo, Strava or tapiriik side?
Was going to attempt to run my own instance of tapiriik but need API keys which Endomondo are no longer giving out from what I've read, so that's a dead end too.

Happy to get my hands dirty if there's any way I can help to get this working?

Same issue here. I just created as Strava account and now am trying to move a few years of workout data from Endomondo to Strava.
I'm also stuck at the "Queuing" stage since yesterday. Tapiriik's activity dashboard shows no activities.

Is there anything else I need to do? Specific settings I need to enable in Endomondo and/or Strava?
Or can I just wait this out and my data will be transferred eventually?

It's been very slow lately, but it does work eventually

Same here, tried to import from Endomondo to both Dropbox and Strava. Is stuck on queueing for 2 days now.
I'll give it some more time ... but is anyone aware of possible alternatives?

You can request all data from endomondo. In my case it took 3 days before package was available. Unfortunately a lot of the files are corrupted; after importing them to starva I see activities with 0h duration, or with some impossible paces.

frycu commented

Right now I'm waiting over 6 days to sync between Endomondo > Strava > Dropbox and nothing happened - I even paid for 1 year subscription and nothing changed.

Quick update: I opened this issue 7 days ago and unfortunately still no luck. Stuck on the same screen.

I have the same problem.

Right now I'm waiting over 6 days to sync between Endomondo > Strava > Dropbox and nothing happened - I even paid for 1 year subscription and nothing changed.

Same thing here :-(

Same here. Setup everything from scratch and it was not working. There is no reaction via mail/twitter.

Same.

@fishion You don't need API key, just proper auth. I created dirty demo code for Endomondo auth - https://gist.github.com/jangaraj/d5dffbabe692aeb6c5bbc09eb9ee538a It should be improved and merged with https://github.com/cpfair/tapiriik/blob/master/tapiriik/services/Endomondo/endomondo.py Let's make hands dirty - I will be happy if I can use dirty local python script for Endomondo->Strava migration.

syre commented

@fishion You don't need API key, just proper auth. I created dirty demo code for Endomondo auth - https://gist.github.com/jangaraj/d5dffbabe692aeb6c5bbc09eb9ee538a It should be improved and merged with https://github.com/cpfair/tapiriik/blob/master/tapiriik/services/Endomondo/endomondo.py Let's make hands dirty - I will be happy if I can use dirty local python script for Endomondo->Strava migration.

Is there a solution for using Google OAuth?

@fishion You don't need API key, just proper auth. I created dirty demo code for Endomondo auth - https://gist.github.com/jangaraj/d5dffbabe692aeb6c5bbc09eb9ee538a It should be improved and merged with https://github.com/cpfair/tapiriik/blob/master/tapiriik/services/Endomondo/endomondo.py Let's make hands dirty - I will be happy if I can use dirty local python script for Endomondo->Strava migration.

You make a good point. In fact, since I have downloaded an archive of all my Endomondo activity (requested through the Endomondo interface - takes a few days for them to produce it, but does appear to work), I guess all I actually need is a script to upload those to Strava. I might take a look at that soon when I get some minutes. Will update here if I get anywhere with it.

Great suggestion to request your data from Endomondo and get it to Strava in another way.
At least is a 'good startingpoint. I doubt Tapiriik will do the job before Endomondo shuts down ...

@pavoxbv MapMyFitness can import directly from Endomondo. Also, you can try fitnesssyncer.com to sync from Endomondo to Strava

Same deal here. I am somewhat annoyed with UA that they are closing endomondo, so I'd don't particularly want to downgrade to MapMyFitness. But this sync does not seem to work at all. I am tempted to run it locally, if I can find the time. For now, I'm trying removing Strava, and syncing with Runkeeper instead.

No - does not appear to work with Runkeeper either. I've had a look at the procedure for running locally - if I can find the time at some point, I'll give it a go. But as far as I can tell, this service appears to be dead, at least in the sense of being able to sync endomondo.

Tried to sync from endomondo to both strava and runkeeper for days now. Still just queing.
I'm gonna try fitnesssyncer.com as mentioned by @racuna
I realize my fitness/health data is gonna be all over the internet soon, not my dream exactly...

Just signed up for fitnessyncer. I can report after a test that it DOES work, although you will need to sign up for a 3.99 per month plan to get the historic data synced. As for this - if it's dead, it would be good for the developer to update the website, as people (including me) are making contributions that now presumably won't be seen again.

Thanks for letting us know. So apparently it's not a capacity issue on Endomondo's or Strava's side.

Agreed - definitely seems to be a tapiriik issue. With fitnesssyncer I've got four runs from the last month into Strava so far - I've kicked off a historic sync. I started with Endomondo in 2012, so three's a lot of data to pull in - I'll let you know if it all works.

I hope someone will come up with a solution to mass upload activites to Strava.
Until that at least the archive download from Endomondo (started from Endomondo account page) worked after a few days waiting. And the donation money was returned by PayPal after opening a refund request. (it worked surprisingly fast, maybe they received a lot if similar requests recently?)

lipis commented

I'm willing to pay if that works.. but from what I read it doesn't help even if you pay!

Any news?

fitnesssyncer is gradually importing my history from Endomondo, and syncing to Strava. It's going extremely slowly - I have to assume that Endomondo's API has been throttled to within an inch of its life. But in fairness it is moving.

lipis commented

Did anyone try run it locally? Maybe some clues there?

tapiriik
I'm syncing from Endomondo to Runkeeper and it works sometimes (I mean two hours a week) but for new workouts that are sinc'ed almost instantly. I've noticed that all my workouts have been read from tapiirik but most of them are not uploaded to RunKeeper. I've also paid a subcription but it didn't made any change

I managed to get Endomondo synced to Strava using www.fitnesssyncer.com instead. The free version seems to work only for up to 2 months of history, but I suppose buying a month of subscription to sync everything, and then cancel, should work.

I'm making slow progress on a script that will upload everything from an Endomondo export archive. It's not too difficult but just short on tine to get it done! Will have something soon I hope. Will be a node-based solution you can run yourselves, and I'll provide (hopefully) good-enough instructions for using it.

Starting today tapiriik seems to started syncing again!
C3E87C54-4DAB-4F15-802E-06A86BE817A8

Starting today tapiriik seems to started syncing again!
C3E87C54-4DAB-4F15-802E-06A86BE817A8

Wow, just wondering: Did it go through ? I have had no luck yet.

Starting today tapiriik seems to started syncing again!
C3E87C54-4DAB-4F15-802E-06A86BE817A8

it looks like only you were lucky one

OK, have a script which seems to be working to bulk import tcx files to Strava as downloaded from Endomondo
https://github.com/fishion/StravaImporter
I'm currently using it to upload all my tracks into a test Strava account to make sure it behaves on large number of files. Strava has some fairly strict rate limiting, so it's going to take a couple of days to test properly, but sharing here in case anyone wants to test it themselves. Any patches (including on the docs side) also welcome.

OK, have a script which seems to be working to bulk import tcx files to Strava as downloaded from Endomondo
https://github.com/fishion/StravaImporter
I'm currently using it to upload all my tracks into a test Strava account to make sure it behaves on large number of files. Strava has some fairly strict rate limiting, so it's going to take a couple of days to test properly, but sharing here in case anyone wants to test it themselves. Any patches (including on the docs side) also welcome.

Hi is there's a chance I can contact you to help me with your import tool?

OK, have a script which seems to be working to bulk import tcx files to Strava as downloaded from Endomondo
https://github.com/fishion/StravaImporter
I'm currently using it to upload all my tracks into a test Strava account to make sure it behaves on large number of files. Strava has some fairly strict rate limiting, so it's going to take a couple of days to test properly, but sharing here in case anyone wants to test it themselves. Any patches (including on the docs side) also welcome.

Hi is there's a chance I can contact you to help me with your import tool?

Sure, If there's specific things you're having trouble with, let me know and I will try to make the instructions better. Probably better to open up issues in that project (https://github.com/fishion/StravaImporter/issues) rather than poluting the conversation here further. I expect I'll do some further work on it to try to get the correct activity types across (see the notes section in README) so may be worth holding off for the time being before using outside of a test Strava account. I would definitely recommend exporting all your data from Endomondo in the meantime though.

The same here (Garmin Connect -> Strava).

OK, have a script which seems to be working to bulk import tcx files to Strava as downloaded from Endomondo
https://github.com/fishion/StravaImporter
I'm currently using it to upload all my tracks into a test Strava account to make sure it behaves on large number of files. Strava has some fairly strict rate limiting, so it's going to take a couple of days to test properly, but sharing here in case anyone wants to test it themselves. Any patches (including on the docs side) also welcome.

In anyone still needs it, the script is working properly now, and I've used it to import my full Endomondo export (~400 activities) into Strava without any issues. It also now fixes the activity types after initial upload using the data from the exported json file. Let me know if you use it and run into any issues.

FittnessSyncer works perfect !!!
FitnessSyncer

Did anyone try run it locally? Maybe some clues there?

I spent 3 days trying to get it to run locally, no luck so far. It needs very old versions of libs that are hard to find and randomly crashes on newer ones. Dropbox libs simply won't load and crash the whoel thing; I removed al ldropbox references from my local copy and got to the point where the server actually starts showing a web interface - only to crash again on django pipelines. I'm loath to set up a vagrant box just for this but will try as option of last resort now

Did anyone try run it locally? Maybe some clues there?

I spent 3 days trying to get it to run locally, no luck so far. It needs very old versions of libs that are hard to find and randomly crashes on newer ones. Dropbox libs simply won't load and crash the whoel thing; I removed al ldropbox references from my local copy and got to the point where the server actually starts showing a web interface - only to crash again on django pipelines. I'm loath to set up a vagrant box just for this but will try as option of last resort now

I forked it and got it running with Docker and Kubernetes: https://github.com/neilboyd/tapiriik
You can see it in real life at https://www.siiink.com/
I only managed to connect a few services so far, including Dropbox and Strava.
It's not fully tested, but it seems to work πŸ˜ƒ

Strava has rate limiting on their API. The fact that this issue seems to arise at a time where a lot of people are trying to push a ton of data from Endomondo to Strava makes me think the following is happening:

  • Lots of people want to push a ton of data to Strava all of a sudden through Tapiriik
  • Strava API rate limiting kicks in at the Tapiriik level (rate limits are typically per API key / app)
  • This makes it so that anyone trying to sync anything to Strava experiences queuing because the Tapiriik core loop probably is serial (eg. until it has synced an activity to all places it should push it to (here including Strava) it won't go on to the next activity).

The project itself seems unmaintained so nobody is there to talk to Strava about fixing it. Some of us could try and go contact the Strava support to try to ask them to lift the rate-limiting on Tapiriik for a few months (until march 31st, Endomondo's deadline) to avoid this issue. Once the bulk of this will be over Tapiriik should essentially start working magically again since it won't be trying to push a ton of data at once to Strava's API.

I'll give it a try to contact Strava's support although they're typically quite unhelpful like most of those freemium fitness apps.

I forked it and got it running with Docker and Kubernetes: https://github.com/neilboyd/tapiriik
You can see it in real life at https://www.siiink.com/
I only managed to connect a few services so far, including Dropbox and Strava.
It's not fully tested, but it seems to work πŸ˜ƒ

Hmm, my main goal was to get Endomondo-data out of there before they shut down. Couldn't get that to work with tapiriik as a base though; so now I went down a different route and built a Node-based container that downloads the data for me. Problem solved :-)

Hmm, my main goal was to get Endomondo-data out of there before they shut down.

You can do that just by downloading from Endomondo. Then if you like you can save to Dropbox and sync from there.

@svenjambor Any reason you found Endomondo's own data export/archive unsuitable for getting your data out?

Using @fishion's tool to then get that exported data into Strava via the API at ones leisure seemed to work pretty well in my own testing.

Did anyone try run it locally? Maybe some clues there?

I spent 3 days trying to get it to run locally, no luck so far. It needs very old versions of libs that are hard to find and randomly crashes on newer ones. Dropbox libs simply won't load and crash the whoel thing; I removed al ldropbox references from my local copy and got to the point where the server actually starts showing a web interface - only to crash again on django pipelines. I'm loath to set up a vagrant box just for this but will try as option of last resort now

I forked it and got it running with Docker and Kubernetes: https://github.com/neilboyd/tapiriik
You can see it in real life at https://www.siiink.com/
I only managed to connect a few services so far, including Dropbox and Strava.
It's not fully tested, but it seems to work πŸ˜ƒ

Thanks, I tried – but it got slightly stuck on. Strava β€“ sat there trying to sync, but to no avail.

I did that about two weeks back and same again this week. Never got the notification.

@svenjambor Oh, that's odd. I've done it for 3 different endomondo accounts (helping others upload their data to Strava too) and always got the link within an hour or so for all 3. (1200, 800 and 500 workouts in the accounts, including hundreds of photos on one of them)

@tdubourg What feedback did you get from Strava?

@neilboyd Thanks for creating the fork. I can't help with the programming, but good luck keeping it working.

I investigated a little bit yesterday. The request to get activities from Strava takes about 90 seconds. If there's a lot of activities it may need several requests. This will cause the queue to build up very big very quickly. This sounds to me like Strava is broken.
That was my experience, but I can't say if that's what's happening to tapiriik.

I downloaded my data from Endomondo and uploaded it to Dropbox. From dropbox I synched tot strava. It worked! Took a few days of waiting byt finally all activities were automatically synched. Happy!

@RobDickson Strava actually only got back to me today.... Sort-of: "Sorry for the wait. We are continuing to work through our backlog as we deal with the unexpected volume of support requests."

If it's a rate limiting issue with the API, then slow the queues to the API down a bit. I would say this is a Tapiriik issue as much as it is a Strava issue. Paid 5 bucks (which isn't a lot) for the use of a server that will get used for something (a queuing screen) that doesn't benefit me.

Today is Feb 3, 2021 the final Solution. to go from Endomondo to Starva
endomodo had been shut down and their data server is some what not accessible.
you can still export it from their website.

The Solution is as follow,
use the Endomondo build in method to transfer to mapymyrun
use https://www.fitnesssyncer.com/ to syn mapmyrun to Starva

all data from 2013 to 2020 is syned free of charge.