SamR1/FitTrackee

Import from CSV

Closed this issue · 37 comments

I've a exported a CSV from tracking app in order to import it to FitTrackee.

Here's some CSV line :

id,uuid,user_id,sport_id,title,gpx,creation_date,modification_date,workout_date,duration,pauses,moving,distance,min_alt,max_alt,descent,ascent,max_speed,ave_speed,bounds,map,map_id,weather_start,weather_end,notes
39,3958a0ec-f780-4579-b7a4-28deffdd8fe5,3,14,Mont Mc,,2018-08-13,,2018-08-13,7:30:00,,7:30:00,20,,3169,,,,,,,,,,-Salomon
postgres@debian-12:~$ bash -x fittrackee-import-csv.sh
+ psql -c '\c fittrackee' -c 'COPY workouts FROM '\''/var/lib/postgresql/sports.csv'\'' DELIMITER '\'','\'' CSV HEADER'
Vous êtes maintenant connecté à la base de données « fittrackee » en tant qu'utilisateur « postgres ».
COPY 1

On the FitTrackee http://fittrackee//workouts/4N6Ak9JcXMrvop5d4DhCK7 screen I've got (seems well formed) :

durée: 7:30:00
distance: 20 km

Nothing on the http://fittrackee/statistics

On the http://fittrackee/workouts/add, and Erreur, veuillez réessayer ou contacter l'administrateur.

Hi,

Some data are missing, preventing from getting statistical data.

On workout creation, the average and maximum speeds are calculated by the API. These are the missing data.

Thanks for answer I've inserted values that have average and maximum speed, statistics are done 👍 .
I'm gonna think about how can I import entries that don't have max and average speed.
Do you have suggestions ?

One solution is to use the API to add workouts. This way, all the values will be correctly calculated, as the records.

More information in the documentation:

Another question , I've imported 1999 entries but I've Pas de records. on the left side.

I'd like to use FitTrackee for sports that don't have distance , avg_speed, max_speed. I mean musculation.
How can I proceed if : distance , avg_speed and max_speed are unavoidable ?

Another question , I've imported 1999 entries but I've Pas de records. on the left side.

Some values are calculated by the API (like average speed, max speed, records), so importing data directly into the database generates incomplete workouts and may lead to errors.
The best way is to use the API to import workouts.

I'd like to use FitTrackee for sports that don't have distance , avg_speed, max_speed. I mean musculation.
How can I proceed if : distance , avg_speed and max_speed are unavoid

FitTrackee is not designed to handle this type of activity. It is built around gpx files for outdoor activities.
It's possible to add a workout without a GPX file as a workaround (for instance when the gps device fails to save the file) and distance is mandatory.

Another question , I've imported 1999 entries but I've Pas de records. on the left side.

Some values are calculated by the API (like average speed, max speed, records), so importing data directly into the database generates incomplete workouts and may lead to errors. The best way is to use the API to import workouts.

I don't have found any errors in fittrackee.log but I don't know how to import my csv export by using API.
I've read your documentation.

I'd like to use FitTrackee for sports that don't have distance , avg_speed, max_speed. I mean musculation.
How can I proceed if : distance , avg_speed and max_speed are unavoid

FitTrackee is not designed to handle this type of activity. It is built around gpx files for outdoor activities. It's possible to add a workout without a GPX file as a workaround (for instance when the gps device fails to save the file) and distance is mandatory.

Yes, I've already import workouts without GPX , which i've done with CSV 👍
Strava permits to use activities (Yoga, musculation) which are not outdoor activities only you can see lots of sports https://support.strava.com/hc/en-us/articles/216919407-Supported-Sport-Types-on-Strava

The distance field is not fixed to not null

fittrackee=# \d workouts
                                              Table « public.workouts »
      Colonne      |            Type             | Collationnement | NULL-able |              Par défaut
-------------------+-----------------------------+-----------------+-----------+--------------------------------------
 id                | integer                     |                 | not null  | nextval('workouts_id_seq'::regclass)
 uuid              | uuid                        |                 | not null  |
 user_id           | integer                     |                 | not null  |
 sport_id          | integer                     |                 | not null  |
 title             | character varying(255)      |                 |           |
 gpx               | character varying(255)      |                 |           |
 creation_date     | timestamp without time zone |                 |           |
 modification_date | timestamp without time zone |                 |           |
 workout_date      | timestamp without time zone |                 | not null  |
 duration          | interval                    |                 | not null  |
 pauses            | interval                    |                 |           |
 moving            | interval                    |                 |           |
 distance          | numeric(6,3)                |                 |           |
 min_alt           | numeric(6,2)                |                 |           |
 max_alt           | numeric(6,2)                |                 |           |
 descent           | numeric(8,3)                |                 |           |
 ascent            | numeric(8,3)                |                 |           |
 max_speed         | numeric(6,2)                |                 |           |
 ave_speed         | numeric(6,2)                |                 |           |
 bounds            | double precision[]          |                 |           |
 map               | character varying(255)      |                 |           |
 map_id            | character varying(50)       |                 |           |
 weather_start     | json                        |                 |           |
 weather_end       | json                        |                 |           |
 notes             | character varying(500)      |                 |           |
Index :
    "workouts_pkey" PRIMARY KEY, btree (id)
    "ix_workouts_map_id" btree (map_id)
    "ix_workouts_sport_id" btree (sport_id)
    "ix_workouts_user_id" btree (user_id)
    "ix_workouts_workout_date" btree (workout_date)
    "workouts_uuid_key" UNIQUE CONSTRAINT, btree (uuid)
Contraintes de clés étrangères :
    "workouts_sport_id_fkey" FOREIGN KEY (sport_id) REFERENCES sports(id)
    "workouts_user_id_fkey" FOREIGN KEY (user_id) REFERENCES users(id)
Référencé par :
    TABLE "records" CONSTRAINT "records_workout_id_fkey" FOREIGN KEY (workout_id) REFERENCES workouts(id)
    TABLE "workout_segments" CONSTRAINT "workout_segments_workout_id_fkey" FOREIGN KEY (workout_id) REFERENCES workouts(id)

Yes, some constraints are missing in the database, but are present in the code.
For example, you can't add a workout without the distance with the User Interface.

This is why it's recommended to use the API to import workouts. Otherwise, any missing values must be calculated outside the application and also imported into the database.

There are some tools using FitTrackee API endpoints here (example code for importing without .gpx file).

Here's a very simplified example with a CSV file (untested, may require adaptation/correction depending on CSV file format and authentication method):

with open("<path to file (csv without header)>", newline="") as csvfile:
    workouts_reader = csv.reader(csvfile, delimiter=",")
    for index, row in enumerate(workouts_reader):
        duration = datetime.datetime.strptime(row[9], "%H:%M:%S")
        delta = datetime.timedelta(
            hours=duration.hour,
            minutes=duration.minute,
            seconds=duration.second,
        )
        workout_date = f"{row[8]} 00:00"

        # fittrackee_session is an OAuth2Session from requests_oauthlib
        r = fittrackee_session.post(
            f"<fittrackee url>/api/workouts/no_gpx",
            json=dict(
                distance=row[12],
                duration=delta.total_seconds(),
                notes=row[24],
                sport_id=row[3],
                title=row[4],
                workout_date=workout_date,
            ),
            headers=dict(content_type="application/json"),
        )
        if r.status_code == 201:
            print(f" > row {index} (id: {row[0]}) imported")
        else:
            error_message = r.json().get("message", "unknown")
            print(
                f" > an error has occurred when importing row {index} "
                f"(id: {row[0]}): {error_message} "
                "(checks FitTrackee logs)"
            )

The position and content of the cells in each row correspond to the example provided.
Note: there may be an API rate limit depending on FitTrackee configuration.

Indeed Strava supports many more sports than FitTrackee, which focuses on outdoor activities (since it's mainly built around .gpx files, it offers far fewer features than the other applications).

Thanks for pasting code example, which i've tried to modify :

#!/usr/bin/python3

import csv
import datetime

with open("/root/ft/sports_no_head.csv", newline="") as csvfile:
    workouts_reader = csv.reader(csvfile, delimiter=",")
    for index, row in enumerate(workouts_reader):
        duration = datetime.datetime.strptime(row[9], "%H:%M:%S")
        delta = datetime.timedelta(
            hours=duration.hour,
            minutes=duration.minute,
            seconds=duration.second,
        )
        workout_date = f"{row[8]} 00:00"

        # fittrackee_session is an OAuth2Session from requests_oauthlib
        r = fittrackee_session.post(
            f"http://ft.my.tld/api/workouts/no_gpx",
            json=dict(
                distance=row[12],
                duration=delta.total_seconds(),
                notes=row[24],
                sport_id=row[3],
                title=row[4],
                workout_date=workout_date,
            ),
            headers=dict(content_type="application/json"),
        )
        if r.status_code == 201:
            print(f" > row {index} (id: {row[0]}) imported")
        else:
            error_message = r.json().get("message", "unknown")
            print(
                f" > an error has occurred when importing row {index} "
                f"(id: {row[0]}): {error_message} "
                "(checks FitTrackee logs)"
            )
root@debian-12:~/ft# /usr/bin/python3 import.py
Traceback (most recent call last):
  File "/root/ft/import.py", line 18, in <module>
    r = fittrackee_session.post(
        ^^^^^^^^^^^^^^^^^^
NameError: name 'fittrackee_session' is not defined

Hi,

The example is incomplete. The fittrackee_session initialisation depends on authentication method (an example is given in the code linked above).

Yes, but I've never use authentication method, I don't understand how to fit with it.

OK. Here's an minimal example adapted from the code linked above: https://codeberg.org/SamR1/csv-importer.
It allows to import a CSV file (after creating an app in FitTrackee to get client id and secret).

I've followed the documentation, I've removed header from CSV I'm getting :

(.venv) root@debian:~/sport/csv-importer# python3 main.py --csv-file ../sports.csv

Please open the following URL to authorize 'CSV importer':

https://ft.my.tld/profile/apps/authorize?response_type=code&client_id=gmdiXXXXXXXXXXXXXn4&scope=workouts%3Awrite&state=n6bnS6XXXXXXXXXXiuO8BXYNl

and enter the URL displayed after application authorization and redirection to get authorization code:
Autoriser [CSV importer](https://ft.my.tld/profile/apps/3) à utiliser votre compte ?

L'application demande les accès suivants :

workouts:write
    donne les droits en écriture aux routes workouts.

I've clicked Autoriser but nothing appends.

I've clicked Autoriser but nothing appends.

After clicking on Autoriser, you should be redirected to a URL that doesn't exist.
You need to copy this URL, paste it into the terminal and press Enter to complete the Authorization flow.

Ok thanks for that precision.

After pasting that URL :

(At the moment I'm testing that functionality so that's why I've generated a self-signed CA and certificate I've added the CA in /usr/local/share/ca-certificates and update-ca-certificates.

  File "/root/sport/csv-importer/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/root/sport/csv-importer/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
    conn.connect()
  File "/root/sport/csv-importer/.venv/lib/python3.11/site-packages/urllib3/connection.py", line 653, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/sport/csv-importer/.venv/lib/python3.11/site-packages/urllib3/connection.py", line 806, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
               ^^^^^^^^^^^^^^^^
  File "/root/sport/csv-importer/.venv/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 465, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/sport/csv-importer/.venv/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 509, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ssl.py", line 1075, in _create
    self.do_handshake()
  File "/usr/lib/python3.11/ssl.py", line 1346, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:992)

Does your program reads /usr/local/share/ca-certificates ?

It uses Requests-OAuthlib for OAuth2 (which works through SSL layer).

You can test without HTTPS. To do so, you have to disable checks by setting an environment variable OAUTHLIB_INSECURE_TRANSPORT:

export OAUTHLIB_INSECURE_TRANSPORT=1

documentation: https://requests-oauthlib.readthedocs.io/en/latest/examples/real_world_example.html#web-app-example-of-oauth-2-web-application-flow

I've export that variable, enable nginx without https, change .venv, source .venv, export OAUTH variable.
But I'm getting :

Autoriser [CSV importer](http://ft.dmd.zld/profile/apps/4) à utiliser votre compte ?
Erreur. Veuillez réessayer ou contacter l'administrateur.

L'application demande les accès suivants :

workouts:write
    donne les droits en écriture aux routes workouts.

I can't enter Autoriser

docker logs -f fittrackee
authlib.oauth2.rfc6749.errors.InsecureTransportError: insecure_transport: OAuth 2 MUST utilize https.
(.venv) root@debian:~/sport/csv-importer# echo $OAUTHLIB_INSECURE_TRANSPORT
1

Hi,

I tried with a fresh install with docker (but without nginx) and added export OAUTHLIB_INSECURE_TRANSPORT=1 in .env before sourcing it, the script imported the csv file:

$ python main.py --csv-file test.csv

Please open the following URL to authorize 'CSV importer':

http://localhost:5000/profile/apps/authorize?response_type=code&client_id=<CLIENT_ID>&scope=workouts%3Awrite&state=6cUDd0xJMCDlTUXnCRcmgsTcoLS6Ah

and enter the URL displayed after application authorization and redirection to get authorization code:
http://localhost/?code=CnL8ieYzMplVW5qqxCSbmXkM7cnjCrnXgIQOzGMmBfnC9G6F&state=6cUDd0xJMCDlTUXnCRcmgsTcoLS6Ah


Starting csv file import...
 > row 0 (id: 39) imported

Without OAUTHLIB_INSECURE_TRANSPORT, I have the following error

$ python main.py --csv-file test.csv
[...]
oauthlib.oauth2.rfc6749.errors.InsecureTransportError: (insecure_transport) OAuth 2 MUST utilize https.

I've added export OAUTHLIB_INSECURE_TRANSPORT=1 before as you've done.

(.venv) root@debian:~/sport/csv-importer# echo $OAUTHLIB_INSECURE_TRANSPORT
1
(.venv) root@debian:~/sport/csv-importer#  source .env
(.venv) root@debian:~/sport/csv-importer# python3 main.py --csv-file ../sports.csv

Please open the following URL to authorize 'CSV importer':

http://192.168.122.17:5000/profile/apps/authorize?response_type=code&client_id=5khBLni2IAreBgV2VVZhCHfS&scope=workouts%3Awrite&state=qioq9XuJK6YuvfSJPil3yYKBWc7MBA

and enter the URL displayed after application authorization and redirection to get authorization code:

But I still have :

Autoriser [CSV](http://192.168.122.17:5000/profile/apps/1) à utiliser votre compte ?
Erreur. Veuillez réessayer ou contacter l'administrateur.

L'application demande les accès suivants :

workouts:write
    donne les droits en écriture aux routes workouts.

In logs :

authlib.oauth2.rfc6749.errors.InsecureTransportError: insecure_transport: OAuth 2 MUST utilize https.

I don't understand what differs from your installation ?

Is that possible to add in the csv-importer the ability to import equipment ?

If the default equipments are set on FitTrackee, there is no need to change the importer. The default equipment will be automatically associated with the workout based on the sport.

Right but I'd like to import my csv workouts with right equipment at the same time.

OK, I updated the script to add equipement, with an additional column.

Thanks for adding equipment do you have a csv example ?
I still have mistakes when importing.



Autoriser [CSV](http://192.168.122.17:5000/profile/apps/1) à utiliser votre compte ?
Erreur. Veuillez réessayer ou contacter l'administrateur.

L'application demande les accès suivants :

workouts:write
    donne les droits en écriture aux routes workouts.

Hi,

I used the same file as provided in the first comment, but with an additional column containing equipment_id of the equipment to associate.

1/ Ok with real certificate it's ok , no error, thanks

2/ I've read your documentation :
Other fields are needs, could you please add all columns in order to get all values during csv import ?

Only few columns are used:

equipment_id (if empty, no equipment will be associated, even if a default equipment exist for workout sport)
distance
duration
notes
sport_id
title
workout_date

Warning: some data are mandatory, see API documentation.

Hi,

I added ascent and descent to match all data handled by the API endpoint.

Hi, thanks a lot 👍

I'm trying to import equipment in the csv

I've created a pair of shoes :

 id |                 uuid                 | user_id |                label                 | description | equipment_type_id |       creation_date        | is_active | total_distance | total_duration | total_moving | total_workouts
----+--------------------------------------+---------+--------------------------------------+-------------+-------------------+----------------------------+-----------+----------------+----------------+--------------+----------------
  1 | e6d5f973-e95c-4e71-9fa8-90c5a87bb087 |       1 | Salomon XX             |             |                 1 | 2024-05-16 16:11:37.906669 | t         |          0.000 | 00:00:00       | 00:00:00     | 

The csv is formatted like that with equipment I've added the 1 in the end of the line wich is the Salomon XX

39,3958a0ec-f780-4579-b7a4-28deffdd8fe5,3,14,Mont Mc,,2018-08-13,,2018-08-13,7:30:00,,7:30:00,20,,3169,,,,,,,,,,Gourde Sac à dos,1

I can't import with equipment.
What's I've missed ?

Without equipment it works very well 👍

Hi,

Glad it works :).

For equipment, the value to add is the short id (string value). You can find it in the equipment detail URL.

Where can i found the string value ? I can't in the SQL query above
Can you give the SQL query to found ?

Hi,

short id is generated from the equipment uuid (it's not stored in database).

>>> import shortuuid
>>> from uuid import uuid4
>>> equipment_uuid = uuid4()
>>> equipment_uuid
UUID('10812809-d30f-444d-9be4-1470b3e7a0f5')
>>> shortuuid.encode(equipment_uuid)
'4wPhfUXCx9rwHfqrF6CSPR'

It's ok, thanks :)
It could be nice to store that value in the equipments table in order to directly get that value ?

I've imported a workout in the database the day and hour are : 2024-05-24 12:42:06.86368

  id  |                 uuid                 | user_id | sport_id |             title             | gpx |       creation_date       | modification_date |    workout_date     | duration | pauses |  moving  | distance | min_alt | max_alt | descent | ascent  | max_speed | ave_speed | bounds | map | map_id | weather_start | weather_end | notes
------+--------------------------------------+---------+----------+-------------------------------+-----+---------------------------+-------------------+---------------------+----------+--------+----------+----------+---------+---------+---------+---------+-----------+-----------+--------+-----+--------+---------------+-------------+-------
 1334 | a1702a53-5e7a-4f0e-a40d-a82ccd0ce2c1 |       1 |        8 | Pyrenees |     | 2024-05-24 12:42:06.86368 |                   | 2024-05-23 22:00:00 | 01:02:45 |        | 01:02:45 |    7.678 |         |         | 394.930 | 393.330 |      7.34 |      7.34 |        |     |        |               |             |
(1 row)

In the fittrackee interface the hour is set to 00:00 :

	Séance	Date	Distance	Durée	Vitesse moy.	Vitesse max	Dénivelé positif	Dénivelé négatif
	Pyrenees 24/05/2024 00:00	7.68 km	1:02:45	7.34 km/h	7.34 km/h	393.33 m	394.93 m

In the CSV :

1335,380d2017-3ab9-4470-b770-1c5b19a159ff,3,8,Pyrenees,,2024-05-24,,2024-05-24,01:02:45,,01:02:45,7.678,,,394.93,393.33,11.38,7.41,,,,,,,NXULvEHa69XmnB3Ew7DBjq

Why does the hour is different from db to interface ?

Hi,

2024-05-24 12:42:06.86368 is the workouts.creation_date (workout creation date on FitTrackee, not displayed on the interface).
The date displayed on the interface is workouts.workout_date, which is 2024-05-23 22:00:00 in UTC (24/05/2024 00:00 in Europe/Paris timezone).

Thanks for answer.

I've tried to set timezone in the csv file (the workouts.workout_date field) , but it does not :

69,158e0484-e75a-4783-a7fd-6ec8d1731f4c,3,1,Alps,,2019-01-11,,2019-01-11 07:00:00,0:30:00,,0:30:00,6,,,0,0,10.8,10.8,,,,,,La Sportiva sac,bPSViQ5RwJotxm23LUXgFZ
python main.py --csv-file ../sports.csv
...
Starting csv file import...
 > an error has occurred when importing row 0 (id: 69): Error during workout save. (checks FitTrackee logs)

cat fittrackee.log
2024/05/31 08:38:11 - fittrackee - ERROR - unconverted data remains: :00 00:00
2024/05/31 08:38:11 - werkzeug - INFO - 192.168.1.2 - - [31/May/2024 08:38:11] "POST /api/workouts/no_gpx HTTP/1.1" 500 -

Hi,

The script has been created for the CSV file provided in the first comment. The workout date has the following format: %Y-%m-%d.
I updated the script to handle the same format as the API: %Y-%m-%d %H:%M.

I modified the example to match the expected format for workout date (2019-01-11 07:00) and an existing equipment and imported the file:

69,158e0484-e75a-4783-a7fd-6ec8d1731f4c,3,1,Alps,,2019-01-11,,2019-01-11 07:00,0:30:00,,0:30:00,6,,,0,0,10.8,10.8,,,,,,La Sportiva sac,Dk4ftgGTosMCeeLbhemP2f

image

Values in database:

fittrackee=# SELECT * FROM workouts;
 id |                 uuid                 | user_id | sport_id | title | gpx |       creation_date        | modification_date |    workout_date     | duration | pauses |  moving  | distance | min_alt | max_alt | descent | ascent | max_speed | ave_speed | bounds | map | map_id | weather_start | weather_end |      notes      
----+--------------------------------------+---------+----------+-------+-----+----------------------------+-------------------+---------------------+----------+--------+----------+----------+---------+---------+---------+--------+-----------+-----------+--------+-----+--------+---------------+-------------+-----------------
  1 | b56260e9-bebc-4df8-a89b-923e73ac6b8a |       1 |        1 | Alps  |     | 2024-06-01 08:52:00.347091 |                   | 2019-01-11 06:00:00 | 00:30:00 |        | 00:30:00 |    6.000 |         |         |   0.000 |  0.000 |     12.00 |     12.00 |        |     |        |               |             | La Sportiva sac
(1 ligne)

fittrackee=# 

As the time zone in my case is Europe/Paris, the value stored in the database is 2019-01-11 06:00:00.

Feel free to modify the script to match the CSV file. The script is provided only as an example.

I've used the modified csv-importer , all is good !
Thanks again 👍