import script doesnt seem to work properly
hanscees opened this issue · 4 comments
hi,
the importscript doesnt seem to work properly.
see also ticket matomo-org/matomo-log-analytics#352
I am using tjhe docker version and in contrary what your github page sais the import script uses python3 now.
However it does not work in my case.
Can you please check if the docker image has the latest version?
I am using the script like so:
/usr/bin/python3 /var/lib/docker/volumes/matmoto_matomo/_data/misc/log-analytics/import_logs.py --url=http://192.168.0.61:8080 --user=hanscees@hanscees.con --password=seclet --idsite=1 --recorders=4 --debug --dryrun /root/bomengidsstats/access.log-20230515
The error is:
usage: import_logs.py [-h] [--auth-user AUTH_USER] [--auth-password AUTH_PASSWORD] [--debug] [--debug-tracker]
[--debug-request-limit DEBUG_REQUEST_LIMIT] --url MATOMO_URL [--api-url MATOMO_API_URL]
[--tracker-endpoint-path MATOMO_TRACKER_ENDPOINT_PATH] [--dry-run] [--show-progress]
[--show-progress-delay SHOW_PROGRESS_DELAY] [--add-sites-new-hosts] [--idsite SITE_ID]
[--idsite-fallback SITE_ID_FALLBACK] [--config CONFIG_FILE] [--login LOGIN] [--password PASSWORD]
[--token-auth MATOMO_TOKEN_AUTH] [--hostname HOSTNAMES] [--exclude-path EXCLUDED_PATHS]
[--exclude-path-from EXCLUDE_PATH_FROM] [--include-path INCLUDED_PATHS] [--include-path-from INCLUDE_PATH_FROM]
[--useragent-exclude EXCLUDED_USERAGENTS] [--enable-static] [--enable-bots] [--enable-http-errors]
[--enable-http-redirects] [--enable-reverse-dns] [--strip-query-string]
[--query-string-delimiter QUERY_STRING_DELIMITER] [--log-format-name LOG_FORMAT_NAME]
[--log-format-regex LOG_FORMAT_REGEX] [--log-date-format LOG_DATE_FORMAT] [--log-hostname LOG_HOSTNAME]
[--skip SKIP] [--recorders RECORDERS] [--recorder-max-payload-size RECORDER_MAX_PAYLOAD_SIZE] [--replay-tracking]
[--replay-tracking-expected-tracker-file REPLAY_TRACKING_EXPECTED_TRACKER_FILE] [--output OUTPUT]
[--encoding ENCODING] [--disable-bulk-tracking] [--debug-force-one-hit-every-Ns FORCE_ONE_ACTION_INTERVAL]
[--force-lowercase-path] [--enable-testmode] [--download-extensions DOWNLOAD_EXTENSIONS]
[--add-download-extensions EXTRA_DOWNLOAD_EXTENSIONS] [--w3c-map-field KEY=VAL] [--w3c-time-taken-millisecs]
[--w3c-fields W3C_FIELDS] [--w3c-field-regex KEY=VAL] [--title-category-delimiter TITLE_CATEGORY_DELIMITER]
[--dump-log-regex] [--ignore-groups REGEX_GROUPS_TO_IGNORE] [--regex-group-to-visit-cvar KEY=VAL]
[--regex-group-to-page-cvar KEY=VAL] [--track-http-method TRACK_HTTP_METHOD] [--retry-max-attempts MAX_ATTEMPTS]
[--retry-delay DELAY_AFTER_FAILURE] [--request-timeout REQUEST_TIMEOUT] [--include-host INCLUDE_HOST]
[--exclude-host EXCLUDE_HOST] [--exclude-older-than EXCLUDE_OLDER_THAN] [--exclude-newer-than EXCLUDE_NEWER_THAN]
[--add-to-date SECONDS_TO_ADD_TO_DATE] [--request-suffix REQUEST_SUFFIX] [--accept-invalid-ssl-certificate]
[--php-binary PHP_BINARY]
file [file ...]
import_logs.py: error: the following arguments are required: file
I have found the bug, && in the passwd crashes the script
python3 ../import_logs1.py --url http://192.168.0.61:8080 --auth-user hanscees@hanscees.com --auth-password wrongpaswd access.log-20230515.gz
Fatal error: the configuration file/home/config/config.ini.php could not be read. Please check permission. This file must be readable by the user running this script to get the authentication token
python3 ../import_logs1.py --url http://192.168.0.61:8080 --auth-user hanscees@hanscees.com --auth-password wrongpasswd&&5 access.log-20230515.gz
import_logs1.py: error: the following arguments are required: file
the maintainer sais he wont fix it and its all my fault for not escaping characters/ :1st_place_medal:
Hi @hanscees,
there's literally no way how this can be fixed within the importer since the remaining parameters don't even get passed to the import script if the CLI/shell you run them from interferes with the command/the values because of the very nature of how the shell works and what special characters it uses to chain commands etc.
We can update the documentation/guide mentioning to quote the values if necessary. Would that help?
Hi @hanscees,
there's literally no way how this can be fixed within the importer since the remaining parameters don't even get passed to the import script if the CLI/shell you run them from interferes with the command/the values because of the very nature of how the shell works and what special characters it uses to chain commands etc.
We can update the documentation/guide mentioning to quote the values if necessary. Would that help?
I think adding this in the documentation would have helped me. I would suggest you add directly below the example in the webpage :
remember to put your password between "" if you have a safe password with special characters or the script might brake
So for instance
python3 ../import_logs.py --url http://192.168.0.6 --auth-user hans@hans.con --auth-password "wrongpasswd&&5" access.log-20230515.gz
I used this python snippit to play with the problem:
#!/usr/bin/python3
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("square", help="display a square of a given number",
type=int)
parser.add_argument(
'--login', dest='login',
help="You can manually specify the Matomo Super User login"
)
parser.add_argument(
'--password', dest='password',
help="You can manually specify the Matomo Super User password"
)
args = parser.parse_args()
print(args.square**2)
if args.password:
print("password is :", args.password)