ValueError: semaphore or lock released too many times

Question

ValueError: semaphore or lock released too many times

Closed this issue 6 years ago · 25 comments

Hi oPromessa,

Install went smoothly on synology DS214play but when I try to run I get this error, please help.

root@DS:/dev# uploadr.py -a
Traceback (most recent call last):
File "/root/apps/Python/bin/uploadr.py", line 68, in
import lib.FlickrUploadr as FlickrUploadr
File "/root/apps/Python/lib/python2.7/site-packages/lib/FlickrUploadr.py", line 225, in
@rate_limited.rate_limited(5) # 5 calls per second
File "/root/apps/Python/lib/python2.7/site-packages/lib/rate_limited.py", line 146, in decorate
last_time.release()
File "/root/apps/Python/lib/python2.7/site-packages/lib/rate_limited.py", line 76, in release
self.ratelock.release()
ValueError: semaphore or lock released too many times

Answer 1 · 2018-08-26T00:09:18.000Z

Hello @rkilg

I looked at the source and it seems weird to me this error.

Can you help me out and:

Confirm which DSM version you have on synology.
set LOGGING_LEVEL to 10 on uploadr.ini file
And rerun and get me the uploadr.err and uploadr.out files obtained with the following command (try to clean up your private key from the file)
- uploadr.py -a 2> uploadr.err | tee uploadr.out

Answer 2 · 2018-08-26T02:03:08.000Z

Hi @oPromessa,
DSM 6.2-23739 Update 2

Same errors in uploadr.err:
Traceback (most recent call last):
File "/root/apps/Python/bin/uploadr.py", line 68, in
import lib.FlickrUploadr as FlickrUploadr
File "/root/apps/Python/lib/python2.7/site-packages/lib/FlickrUploadr.py", line 225, in
@rate_limited.rate_limited(5) # 5 calls per second
File "/root/apps/Python/lib/python2.7/site-packages/lib/rate_limited.py", line 146, in decorate
last_time.release()
File "/root/apps/Python/lib/python2.7/site-packages/lib/rate_limited.py", line 76, in release
self.ratelock.release()
ValueError: semaphore or lock released too many times

Nothing in uploadr.out?

uploadr.ini:

 cat ~/apps/Python/etc/uploadr.ini
[Config]
###############################################################################
#   Base folder where we keep the database and related control files.
#      Select either the current working directory or a specific folder
#   Other values use a reference to this base folder with %(FOLDER)s
###############################################################################
FOLDER = os.path.abspath(os.getcwd())
#FOLDER = "~/apps/flickr-uploader-2.8.6"

###############################################################################
#   Location to scan for new files
###############################################################################
FILES_DIR = "/volume2/photo/Photos"

###############################################################################
#   Flickr settings
#   Set your own API key and secret message
#   Apply for an API key at: http://www.flickr.com/services/apps/create/apply
#   Recommended to leave title empty to use filename without extension as title
###############################################################################
FLICKR = {
        "title"                 : "",
        "description"           : "",
        "tags"                  : "auto-upload",
        "is_public"             : "0",
        "is_friend"             : "0",
        "is_family"             : "0",
        "api_key"               : "XX",
        "secret"                : "XX"
        }

###############################################################################
#   How often to check for new files to upload (in seconds)
#   Only used with the -d (--daemon) argument option.
###############################################################################
SLEEP_TIME = 1 * 60

###############################################################################
#   Only with --drip-feed option:
#      How often to wait between uploading individual files (in seconds)
###############################################################################
DRIP_TIME = 1 * 60

###############################################################################
#   File we keep the history of uploaded files in.
###############################################################################
DB_PATH = os.path.join(%(FOLDER)s, "flickrdb")

###############################################################################
#   Location of file lock to ensure only one execution is active of uplaodr.py
###############################################################################
LOCK_PATH = os.path.join(%(FOLDER)s, ".flickrlock")

###############################################################################
#   File where we keep the flickr authentication information.
###############################################################################
TOKEN_CACHE = os.path.join(%(FOLDER)s, "token")

###############################################################################
#   List of folder names you don't want to parse
###############################################################################
EXCLUDED_FOLDERS = ["@eaDir","#recycle",".picasaoriginals","_ExcludeSync","Corel Auto-Preserve","Originals","Automatisch beibehalten von Corel"]

###############################################################################
#   List of filename regular expressions you wish to ignore
#   Regex is used to search the filename (as opposed to matching it completely)
#   Scan through a filename, looking for any location where any of these
#   regular expressions matches. Use ^ (start of filename) or $ (end of
#   filename) as appropriate.
#   Example:
#      IGNORED_REGEX = [ 'IMG_[0-8]', '^[\d\d\d]', '.+Ignore.+']
###############################################################################
IGNORED_REGEX = []

###############################################################################
#   List of file extensions you agree to upload
###############################################################################
ALLOWED_EXT = ["jpg","png","avi","mov","mpg","mp4","3gp"]

###############################################################################
#   RAW File Conversion (optional)
###############################################################################
CONVERT_RAW_FILES = True
RAW_EXT = ["3fr", "ari", "arw", "bay", "crw", "cr2", "cap", "dcs", "dcr", "dng", "drf", "eip", "erf", "fff", "iiq", "k25", "kdc", "mdc", "mef", "mos", "mrw", "nef", "nrw", "obm", "orf", "pef", "ptx", "pxn", "r3d", "raf", "raw", "rwl", "rw2", "rwz", "sr2", "srf", "srw", "x3f"]
RAW_TOOL_PATH = "/usr/bin/"

###############################################################################
#   Files greater than this value won't be uploaded (1Mo = 1000000)
###############################################################################
FILE_MAX_SIZE = 50000000

###############################################################################
#   Do you want to check each time if already uploaded files have been changed?
###############################################################################
MANAGE_CHANGES = True

###############################################################################
#   Full set name
#   Example:
#      FILES_DIR = /home/user/media
#      File to upload: /home/user/media/2014/05/05/photo.jpg
#   FULL_SET_NAME:
#      False: 05
#      True: 2014/05/05
###############################################################################
FULL_SET_NAME = False

###############################################################################
#   Counter for certain SQL operations attempts in case of 'DB Locked' error
###############################################################################
MAX_SQL_ATTEMPTS = 3

###############################################################################
#   Counter for uploading, replacing attempts
###############################################################################
MAX_UPLOAD_ATTEMPTS = 10

###############################################################################
#   Level to be used for Logging purposes. Select one of the following values:
#      Level		Numeric value
#      CRITICAL         50
#      ERROR            40
#      WARNING          30
#      INFO             20
#      DEBUG            10
#      NOT              0
###############################################################################
LOGGING_LEVEL = 10

###############################################################################
#   Output logging information into a rotating set of log file(s).
#      ROTATING_LOGGING to Enable (True) or Disable (False)
#      ROTATING_LOGGING_PATH location of folder/main logging filename
#         Create folder logs as required.
#      ROTATING_LOGGING_FILE_SIZE for maximum file size of each log file
#      ROTATING_LOGGING_FILE_COUNT for maximum count of old log files to keep
#      ROTATING_LOGGING_LEVEL Level Logging.
#         Check LOGGING_LEVEL setting for acceptable values.
#         Normally set ROTATING_LOGGING_LEVEL lower than LOGGING_LEVEL
###############################################################################
ROTATING_LOGGING = True
ROTATING_LOGGING_PATH = os.path.join(%(FOLDER)s, "logs", "uploadr.err")
ROTATING_LOGGING_FILE_SIZE = 25*1024*1024  # 25 MBytes
ROTATING_LOGGING_FILE_COUNT = 3
ROTATING_LOGGING_LEVEL = 30

Answer 3 · 2018-08-28T20:52:47.000Z

Hi @oPromessa

Any idea on the problem here?

Answer 4 · 2018-08-29T00:34:54.000Z

Sorry for the delay. I’ve been out. Will take a look at it tomorrow.

I’m struggling as the error seems to occur on initializing even before the main code starts to run. So there must be some compatibility issue.

I’m really sorry about that.

Can you run “python -V” to confirm the python version you have.
I assume you’re using version 2.8.6 from uploadr. You installed it from Pypi or you got the tar from the version repository?
Can you run the “env” command to confirm the PYTHONPATH and other python related variables. Send me the output.
Do you have any other python related tool/module installed?
Can you try running uploadr.py with other options just to double check:
- uploadr.py -v
- uploadr.py -vv -n
- uploadr.py -v -n -p 5

Answer 5 · 2018-08-29T08:48:09.000Z

root@DS:~ python -V
Python 2.7.14
root@DS:~ env
SHELL=/bin/ash
TERM=xterm-256color
LC_ALL=en_US.utf8
USER=root
FLICKR_UPLOADR_PY_SECRET= XXXX
SUDO_USER=admin
SUDO_UID=1024
USERNAME=root
PAGER=more
MAIL=/var/mail/root
PATH=/opt/bin:/opt/sbin:/root/apps/Python/bin:/opt/bin:/opt/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/syno/sbin:/usr/syno/bin:/usr/local/sbin:/usr/local/bin:/root:/var/packages/Java8/target/j2sdk-image/bin:/var/packages/Java8/target/j2sdk-image/jre/bin:/var/packages/Java8/target/j2sdk-image/bin:/var/packages/Java8/target/j2sdk-image/jre/bin
PWD=/root
JAVA_HOME=/var/packages/Java8/target/j2sdk-image/jre
LANG=en_US.utf8
TZ=/etc/TZ
SHLVL=1
SUDO_COMMAND=/bin/ash
HOME=/root
TERMINFO=/usr/share/terminfo
FLICKR_UPLOADR_PY_API_KEY= XXXX
PYTHONPATH=/root/apps/Python/lib/python2.7/site-packages
LOGNAME=root
CLASSPATH=.:/var/packages/Java8/target/j2sdk-image/jre/lib
PGDATA=/var/services/pgsql
SUDO_GID=100
_=/opt/bin/env
root@DS:~ uploadr.py -v
Traceback (most recent call last):
  File "/root/apps/Python/bin/uploadr.py", line 68, in <module>
    import lib.FlickrUploadr as FlickrUploadr
  File "/root/apps/Python/lib/python2.7/site-packages/lib/FlickrUploadr.py", line 225, in <module>
    @rate_limited.rate_limited(5)  # 5 calls per second
  File "/root/apps/Python/lib/python2.7/site-packages/lib/rate_limited.py", line 146, in decorate
    last_time.release()
  File "/root/apps/Python/lib/python2.7/site-packages/lib/rate_limited.py", line 76, in release
    self.ratelock.release()
ValueError: semaphore or lock released too many times
root@DS:~ pip install python
/usr/local/lib/python2.7/site-packages/cffi/model.py:526: UserWarning: 'point_conversion_form_t' has no values explicitly defined; next version will refuse to guess which integer type it is meant to be (unsigned/signed, int/long)
  % self._get_c_name())
Requirement already satisfied: python in /usr/lib/python2.7/lib-dynload (2.7.9)

root@DS:~ pip check
/usr/local/lib/python2.7/site-packages/cffi/model.py:526: UserWarning: 'point_conversion_form_t' has no values explicitly defined; next version will refuse to guess which integer type it is meant to be (unsigned/signed, int/long)
  % self._get_c_name())
zsi 2.0rc3 requires pyxml, which is not installed.
pyopenssl 16.0.0 has requirement cryptography>=1.3, but you have cryptography 0.9.3.
caldav 0.1.10 has requirement lxml==2.1.5, but you have lxml 3.6.0.
root@DS:~ uploadr.py -vv -n
Traceback (most recent call last):
  File "/root/apps/Python/bin/uploadr.py", line 68, in <module>
    import lib.FlickrUploadr as FlickrUploadr
  File "/root/apps/Python/lib/python2.7/site-packages/lib/FlickrUploadr.py", line 225, in <module>
    @rate_limited.rate_limited(5)  # 5 calls per second
  File "/root/apps/Python/lib/python2.7/site-packages/lib/rate_limited.py", line 146, in decorate
    last_time.release()
  File "/root/apps/Python/lib/python2.7/site-packages/lib/rate_limited.py", line 76, in release
    self.ratelock.release()
ValueError: semaphore or lock released too many times
root@DS:~ uploadr.py -v -n -p 5
Traceback (most recent call last):
  File "/root/apps/Python/bin/uploadr.py", line 68, in <module>
    import lib.FlickrUploadr as FlickrUploadr
  File "/root/apps/Python/lib/python2.7/site-packages/lib/FlickrUploadr.py", line 225, in <module>
    @rate_limited.rate_limited(5)  # 5 calls per second
  File "/root/apps/Python/lib/python2.7/site-packages/lib/rate_limited.py", line 146, in decorate
    last_time.release()
  File "/root/apps/Python/lib/python2.7/site-packages/lib/rate_limited.py", line 76, in release
    self.ratelock.release()
ValueError: semaphore or lock released too many times

Just removed command prompt hashes('#') due to strikeout in comment...

Answer 6 · 2018-08-29T18:07:59.000Z

What I've seen so far is... still checking... I'm on a hunt for a config error... bear with me!

we have both the same DSM version: DSM 6.2-23739 Update 2. 🆗
you have python 2.7.14 where as I have 2.7.12 ❗️
Question to you Have you installed the Python module from the Package Center or simply used the one pre-installed?
you are running as "root" whereas I've created a user with some other name ("super" for instance) with admin privileges which I use. ❗️
you have a DS214play and I have a DS415+ although I also test on a virtual system with DS3615xs (but in this case with a DSM 6.1 and python 2.7.12 also!)
Good of you to send me the output of pip check. Question to you Can you send the output of pip list also?
You have some issue on your system related with some python modules:
- cffi which I don't use (CFFI... it's probably not related. Also found this reference related to the issue reported
- zsi
- pyopenssl
- caldav
Question to you Could you uninstall such modules, at least cffi and pyopenssl? On my side I''try to install in a simulated environment such modules and see the result.
Question to you Could you run python -v /root/apps/Python/bin/uploadr.py -n 2>&1 | tee output.log and send me the output.log to see if some clue comes up.

Answer 7 · 2018-08-29T18:35:59.000Z

UPDATE: ON my test system, python 2.7.12 and DSM 6.1...

Was able to install cffi. Tested uploadr.py and still ran ok. 🆗
Was not able to install zsii as it reports it needs PyXML>0.8.3 and although version 0.8.4 seems to exist it does not install it. 😒
Installed pyopenssl and things started to get messy... on a module I use heavily (flickrapi)... I tried removing it and it seems it screws up the OpenSSL module which in turn, prevents pip from uninstalling it via pip uninstall pyopenssl. In fact pip is no longer operational... not even pip -V. Although, strangely enough it is working on your system.
- Question for you can you run also pip -V I'm using version 18 which I've upgraded via pip install --upgrade pip

ruler@VirtualNAS:~/apps/issue.73$ pip install pyopenssl  --prefix=~/apps/Python 
Collecting pyopenssl
  Using cached https://files.pythonhosted.org/packages/96/af/9d29e6bd40823061aea2e0574ccb2fcf72bfd6130ce53d32773ec375458c/pyOpenSSL-18.0.0-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in /volume1/homes/ruler/apps/Python/lib/python2.7/site-packages (from pyopenssl) (1.11.0)
Collecting cryptography>=2.2.1 (from pyopenssl)
  Using cached https://files.pythonhosted.org/packages/87/e6/915a482dbfef98bbdce6be1e31825f591fc67038d4ee09864c1d2c3db371/cryptography-2.3.1-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: idna>=2.1 in /volume1/homes/ruler/apps/Python/lib/python2.7/site-packages (from cryptography>=2.2.1->pyopenssl) (2.6)
Collecting enum34; python_version < "3" (from cryptography>=2.2.1->pyopenssl)
  Using cached https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl
Requirement already satisfied: cffi!=1.11.3,>=1.7 in /volume1/homes/ruler/apps/Python/lib/python2.7/site-packages (from cryptography>=2.2.1->pyopenssl) (1.11.5)
Collecting asn1crypto>=0.21.0 (from cryptography>=2.2.1->pyopenssl)
  Using cached https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl
Collecting ipaddress; python_version < "3" (from cryptography>=2.2.1->pyopenssl)
  Using cached https://files.pythonhosted.org/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl
Requirement already satisfied: pycparser in /volume1/homes/ruler/apps/Python/lib/python2.7/site-packages (from cffi!=1.11.3,>=1.7->cryptography>=2.2.1->pyopenssl) (2.18)
Installing collected packages: enum34, asn1crypto, ipaddress, cryptography, pyopenssl
Successfully installed asn1crypto-0.24.0 cryptography-2.3.1 enum34-1.1.6 ipaddress-1.0.22 pyopenssl-18.0.0
ruler@VirtualNAS:~/apps/issue.73$ uploadr.py -n
Traceback (most recent call last):
  File "/var/services/homes/ruler/apps/Python/bin/uploadr.py", line 68, in <module>
    import lib.FlickrUploadr as FlickrUploadr
  File "/var/services/homes/ruler/apps/Python/lib/python2.7/site-packages/lib/FlickrUploadr.py", line 45, in <module>
    import flickrapi
  File "/var/services/homes/ruler/apps/Python/lib/python2.7/site-packages/flickrapi/__init__.py", line 44, in <module>
    from flickrapi.core import FlickrAPI
  File "/var/services/homes/ruler/apps/Python/lib/python2.7/site-packages/flickrapi/core.py", line 13, in <module>
    from . import tokencache, auth
  File "/var/services/homes/ruler/apps/Python/lib/python2.7/site-packages/flickrapi/tokencache.py", line 8, in <module>
    from flickrapi.auth import FlickrAccessToken
  File "/var/services/homes/ruler/apps/Python/lib/python2.7/site-packages/flickrapi/auth.py", line 23, in <module>
    from requests_toolbelt import MultipartEncoder
  File "/var/services/homes/ruler/apps/Python/lib/python2.7/site-packages/requests_toolbelt/__init__.py", line 12, in <module>
    from .adapters import SSLAdapter, SourceAddressAdapter
  File "/var/services/homes/ruler/apps/Python/lib/python2.7/site-packages/requests_toolbelt/adapters/__init__.py", line 12, in <module>
    from .ssl import SSLAdapter
  File "/var/services/homes/ruler/apps/Python/lib/python2.7/site-packages/requests_toolbelt/adapters/ssl.py", line 12, in <module>
    import requests
  File "/var/services/homes/ruler/apps/Python/lib/python2.7/site-packages/requests/__init__.py", line 84, in <module>
    from urllib3.contrib import pyopenssl
  File "/var/services/homes/ruler/apps/Python/lib/python2.7/site-packages/urllib3/contrib/pyopenssl.py", line 46, in <module>
    import OpenSSL.SSL
  File "/var/services/homes/ruler/apps/Python/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
  File "/var/services/homes/ruler/apps/Python/lib/python2.7/site-packages/OpenSSL/crypto.py", line 12, in <module>
    from cryptography import x509
  File "/var/services/homes/ruler/apps/Python/lib/python2.7/site-packages/cryptography/x509/__init__.py", line 8, in <module>
    from cryptography.x509.base import (
  File "/var/services/homes/ruler/apps/Python/lib/python2.7/site-packages/cryptography/x509/base.py", line 16, in <module>
    from cryptography.x509.extensions import Extension, ExtensionType
  File "/var/services/homes/ruler/apps/Python/lib/python2.7/site-packages/cryptography/x509/extensions.py", line 13, in <module>
    from asn1crypto.keys import PublicKeyInfo
  File "/var/services/homes/ruler/apps/Python/lib/python2.7/site-packages/asn1crypto/keys.py", line 22, in <module>
    from ._elliptic_curve import (
  File "/var/services/homes/ruler/apps/Python/lib/python2.7/site-packages/asn1crypto/_elliptic_curve.py", line 51, in <module>
    from ._int import inverse_mod
  File "/var/services/homes/ruler/apps/Python/lib/python2.7/site-packages/asn1crypto/_int.py", line 56, in <module>
    from ._perf._big_num_ctypes import libcrypto
  File "/var/services/homes/ruler/apps/Python/lib/python2.7/site-packages/asn1crypto/_perf/_big_num_ctypes.py", line 35, in <module>
    libcrypto_path = find_library(b'crypto' if sys.version_info < (3,) else 'crypto')
  File "/usr/lib/python2.7/ctypes/util.py", line 242, in find_library
    raise RuntimeError("can not find library %s" % name)
RuntimeError: can not find library crypto

In conclusion... something fishy here... maybe offtrack but still. Can you uninstall pyopenssl or do you need it? I saw references on how to uninstall on this synology forum and then also this link (already a bit old I know) which makes references to cffi also on the GitHub issue for core related packages building on synology

Answer 8 · 2018-08-29T19:16:04.000Z

One more update... I've installed on a test system with DSM 6.1.6-15266 Update 1 the "Python Module"

And started getting similar warnings as in your system on pip check... So I guess you've installed the Python Module.

ruler@VirtualNAS:~/apps/issue.73$ pip check
/usr/local/lib/python2.7/site-packages/cffi/model.py:526: UserWarning: 'point_conversion_form_t' has no values explicitly defined; next version will refuse to guess which integer type it is meant to be (unsigned/signed, int/long)
  % self._get_c_name())
zsi 2.0rc3 requires pyxml, which is not installed.
pyopenssl 16.0.0 has requirement cryptography>=1.3, but you have cryptography 0.9.3.
caldav 0.1.10 has requirement lxml==2.1.5, but you have lxml 3.6.0.

My app (flickr-uploadr) still runs but brings up warnings on cffi.

I've uninstall it (Python Module) and things came back to normal.

So my MAIN suggestion at this time, in case you don't need it, is to REMOVE this python Module from the Package center and try it out with the basic python installation DSM comes with.

You may have to uninstall and reinstall flickr-uploader. Give it a try.

For your reference the output of my pip list is very small:

Package           Version  
----------------- ---------
certifi           2018.4.16
chardet           3.0.4    
flickr-uploader   2.8.6    
flickrapi         2.4.0    
idna              2.6      
oauthlib          2.1.0    
pip               18.0     
requests          2.18.4   
requests-oauthlib 0.8.0    
requests-toolbelt 0.8.0    
setuptools        39.2.0   
six               1.11.0   
urllib3           1.22     
wheel             0.31.1

Hope it works!! Let me know!

Answer 9 · 2018-08-30T09:20:41.000Z

Have removed python module but still same error
Shall I remove any of these?

Please note I have iPKGui and easy bootstrap installer as I was trying to load a security module (BASE) which required this as well as many others from various package sources, I had Debian chroot too which I have now uninstalled as well which didn’t make much difference do you know a way to produce a list of all packages installed from the dsm package centre?

root@DS:~# pip list
Package Version

certifi 2018.8.13
chardet 3.0.4
defusedxml 0.5.0
flickr-uploader 2.8.6
flickrapi 2.4.0
idna 2.7
netifaces 0.10.4
oauthlib 2.1.0
pip 18.0
pycurl 7.19.5.1
requests 2.19.1
requests-oauthlib 1.0.0
requests-toolbelt 0.8.0
setuptools 40.2.0
six 1.11.0
sortphotos 1.0.1
urllib3 1.23
wheel 0.31.1
WSDiscovery 0.2

Answer 10 · 2018-08-30T11:19:47.000Z

Bummer! I was hopeful.

Question to you Could you run python -v /root/apps/Python/bin/uploadr.py -n 2>&1 | tee output.log and send me the output.log to see if some clue comes up.
I do not use ipkg
to list installed DSM packages from the command line you can:
- run /usr/syno/bin/synopkg list
- use /usr/syno/bin/synopkg -h to check for more options

Answer 11 · 2018-08-30T15:13:49.000Z

One more though... I've uploaded a test version with more debug info on the particular area to see if we can get a grip on things.

Can you uninstall flickr-uploader with the following command

pip uninstall flickr-uploader

And install this alpha-version 2.8.7-alpha.31 with the following command-line:

pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple flickr-uploader==2.8.7a31 --prefix=~/apps/Python/ --no-cache-dir

And make a simple test and send me the uploadr.log file

uploadr.py --help 2>&1 | tee uploadr.log

If step#3 somehow works you can then try the -a option. Please use a different output log filename not to mix things up. You can attach files on the comment by dragging/dropping

uploadr.py -a 2>&1 | tee uploadr-a.log

Answer 12 · 2018-08-30T21:37:07.000Z

pkglist.txt
uploadr.log

Ok done hopefully this sheds some light?

Answer 13 · 2018-08-30T21:41:02.000Z

uploadr-a.log

Answer 14 · 2018-08-30T23:00:26.000Z

It finally went past that stupid and weird error. Still don’t know the exact cause but it should now be working.

Although you get some debugging info upon starting the app you can now try to configure your key/secret access to Flickr and test t out. I suggest you to create a dummy Flickr account just to be on the safe side.

Let me know.

Update
Made a new alpha-version with less debug. Try this one out.
One more though... I've uploaded a test version with more debug info on the particular area to see if we can get a grip on things.

Can you uninstall flickr-uploader with the following command

pip uninstall flickr-uploader

And install this alpha-version 2.8.7-alpha.33 with the following command-line:

pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple flickr-uploader==2.8.7a33 --prefix=~/apps/Python/ --no-cache-dir

Answer 15 · 2018-08-31T22:39:00.000Z

great stuff all working now thanks so much, new account was best idea as i've organised and de-duped my 50K photos, just another quick question, my videos didn't upload cause of this error
[uploadr] Skipping file due to size restriction:
even though I maxed out the MAX_FILE_SIZE ini parameter to 99999999999999900000?

But there were many mov and avi, does this error arise cause it can't convert, i've installed the exiftool.

Where would i find the log file if the exiftool doesn't convert properly or the flick-uploader can't execute the exif conversion tool?

Answer 16 · 2018-09-01T15:03:11.000Z

That's great news @rkilg !!! Super!

Video file size
Flickr itself has a maximum size limit for videos. flickr-uploader first checkouts the MAX_FILE_SIZE parameter and, if below that max_size, it uploads it. If you get an error from Flickr is the server itself refusing. As flickr is moving things around did not find the exact limit for Videos in effect nowadays but I recall reading somewhere it was 1GB.
Side Note: The exiftool is used to convert RAW pics into JPEGS.
The uploaded videos are converted/decoded/verified by Flickr itself.
1. I did find Flickr sometimes issues Error: 5: FileType was not recognised" for some MP4, MOV and MPG files. but it's not related with video file size.
Log files location depends on how you launch uploadr.py:
1. Via the DSM Synology Schedule by using the provided uploadr.cron file: you should have an upload.out (more generic messages) and a upload.err (more debugging info) which level of output is controlled via the LOGGING_ERROR setting in uploadr.ini
2. If you run it mannulayy I normally do: uploadr.py -v -b -p 10 > uploadr.out 2> uploadr.err to generate the uploadr.out and uploadr.err output files. Again the upload.err (more debugging info) which level of output is controlled via the LOGGING_ERROR setting in uploadr.ini.
The "Skipping file due to size restriction" message is based on the FILE_MAX_SIZE setting on uploadr.ini.
1. Maybe such a number is too big and screws things around. I need the Log files to check that.
2. Can you try with 999000000 (~1GB)?
3. Or maybe can you try with 1999000000 (~2GB)?

Answer 17 · 2018-09-02T10:41:08.000Z

@rkilg an update on item 5. above. As I suspected, 99999999999999900000 is too big so the app defaults it back to 50000000 which is 50MB.

So, you definitely have to go with 1GB or 2GB and I still think Flickr only accepts video up to 1GB... but give it a try.

Answer 18 · 2018-09-07T10:07:10.000Z

Hi @oPromessa

Can you help me with this error please?

[3058][2018.09.07 00:57:11]:[MainProcess][ERROR ]:[root] +++#40: Flickrapi exception on upload
[3058][2018.09.07 00:57:11]:[MainProcess][ERROR ]:[root] Error code:[4] message:[Error: 4: Filesize was zero]
[3058][2018.09.07 00:57:11]:[MainProcess][ERROR ]:[root] (<class 'flickrapi.exceptions.FlickrError'>, FlickrError(u'Error: 4: Filesize was zero',), <traceback object at 0xb651da04>)
[3058][2018.09.07 00:57:11]:[MainProcess][ERROR ]:[root] xxx#042: Sleep 10 and check if file is already uploaded.
[3058][20

Answer 19 · 2018-09-07T21:43:55.000Z

Well... as per fhe Flickr error codes flickr API errors it indicates the file being uploaded is of "zero" size.

A few lines above this error you should have an indication of which file was being processed.

To find all zero files in your system you can run any of the following options on the command find:

$ cd /volume1/photos
$ # find command to list all files of size 0
$ find . -type f -size 0
...
$ # find command to list all files *.jpg filenames of size 0. Note you have to use the backslash.
$ find . -type f -size 0 -name \*.jpg
...
$ # find command to list all files *.mp4 filenames of size 0. Note you have to use the backslash.
$ find . -type f -size 0 -name \*.mp4
...
$ # find command to list all files *.mp4 OR *.jpg filenames of size 0. Note you have to use the backslash.
$ find . -type f -size 0 -name \*.mp4 -o -name \*.jpg
...

Answer 20 · 2018-09-09T05:57:21.000Z

Hi @oPromessa

I deleted the 0 size files, strange they where there, now I'm getting this error and nothing is uploading?

[1311][2018.09.09 05:55:09]:[MainProcess][CRITICAL]:[root] FINAL ini_file:[/root/apps/Python/bin/uploadr.ini]
[1311][2018.09.09 05:55:09]:[MainProcess][WARNING ]:[root] CONSOLE_LOGGING.setLevel=[10] ROTATING_LOGGING.setLevel/enabled?=[30/False] MY_CFG.LOGGING_LEVEL=[10]
[1311][2018.09.09 05:55:09]:[MainProcess][WARNING ]:[root] FILES_DIR: [/volume2/photo/Photo/]
[1311][2018.09.09 05:55:09]:[MainProcess][WARNING ]:[root] Cached token: Token Non-Existant.
[1311][2018.09.09 05:55:09]:[MainProcess][WARNING ]:[root] Added mimetype "video/3gp"/".3gp" correctly.
[1311][2018.09.09 05:55:09]:[MainProcess][WARNING ]:[root] Checking token: nuflickr is None:[True] nuflickr.token_cache.token is None:[Not valid as nuflickr is None]

I already had ~15K photos uploaded not sure what to do, the oauth token can be seen in /volume2/photo/ directory?

Answer 21 · 2018-09-09T09:45:18.000Z

It seems it has found the INI file on /root/apps/Python/bin/uploadr.ini which may have wrong config.

The normal place for INI file should be /root/apps/Python/etc/uploadr.ini and from previous logs you sent I trust that where the correct config is.

After making sure the file on ETC folder is correct you can delete the file from the BIN folder and try it out.

Another important thing is that the token as well as the control database (flickrdb) are saved on the directory from where you launch the app:

that would be the current working directory if you are launching it manually
or the folder you set on the uploadr.cron file if you launch it via Task scheduler

Answer 22 · 2018-09-09T10:10:22.000Z

hi @oPromessa

It just seems to be stuck on authenticating, like it can't reach the flickr site?

root@DS:~/token# uploadr.py -a -vv > uploader.log
[1696][2018.09.09 09:54:04]:[MainProcess][CRITICAL]:[root] FINAL ini_file:[/root/apps/Python/bin/uploadr.ini]
[1696][2018.09.09 09:54:04]:[MainProcess][WARNING ]:[root] CONSOLE_LOGGING.setLevel=[20] ROTATING_LOGGING.setLevel/enabled?=[30/False] MY_CFG.LOGGING_LEVEL=[20]
[1696][2018.09.09 09:54:04]:[MainProcess][INFO ]:[root] Output for FLICKR Configuration:
{'api_key': '',
'description': '',
'is_family': '0',
'is_friend': '0',
'is_public': '0',
'secret': '',
'tags': 'auto-upload',
'title': ''}
[1696][2018.09.09 09:54:04]:[MainProcess][WARNING ]:[root] FILES_DIR: [/volume2/photo/Photo/]
[1696][2018.09.09 09:54:04]:[MainProcess][INFO ]:[root] Cached token: Connecting...
[1696][2018.09.09 09:54:04]:[MainProcess][INFO ]:[root] Cached token: Connected. Looking in TOKEN_CACHE:[/root/token/token]
[1696][2018.09.09 09:54:04]:[MainProcess][WARNING ]:[root] Cached token: Token Non-Existant.
[1696][2018.09.09 09:54:04]:[MainProcess][INFO ]:[root] Adding mimetype "video/3gp"/".3gp"
[1696][2018.09.09 09:54:04]:[MainProcess][WARNING ]:[root] Added mimetype "video/3gp"/".3gp" correctly.
[1696][2018.09.09 09:54:04]:[MainProcess][WARNING ]:[root] Checking token: nuflickr is None:[True] nuflickr.token_cache.token is None:[Not valid as nuflickr is None]
[1696][2018.09.09 09:54:04]:[MainProcess][INFO ]:[root] Authentication: Connecting...
[1696][2018.09.09 09:54:04]:[MainProcess][INFO ]:[root] Authentication: Connected. Getting new token...

Answer 23 · 2018-09-09T11:45:48.000Z

IMPORTANT UPDATE

I got it... run uploadr -a -vv without the output redirected to > uploadr.log. As it is showing you up the new authentication link to use but it is being redirected to the uplaodr.log and as such you can't see it.
Do follow also the rest of the remarks below

Initial remarks

Hmmm... ok... two things...

it is being launched from the /root/token folder and looking for the token on folde /root/token/token...
Can you maybe run it from the /root folder as I guess your previous token is on folder /root/token and not on folder /root/token/token

If it fails make sure you go the proper folder you want to run it from now on (should be ~ or /root) and let's reestablish a new token with uploadr.py -a on an interactive session.

Answer 24 · 2018-09-10T08:15:56.000Z

Hi @oPromessa

Well derp, I should've realised that too, thanks, currently converting files cause I enabled the exiftool for my CR2 files, but all seems well atm, will let you know if I achieve upload success.

Cheers :-)

Answer 25 · 2018-09-10T21:21:39.000Z

Excellent. I’m closing this issue has it’s now covering too many topics.

Do let me know the end result. I think you can still comment on closed issues

Do open a new issue report should the need arise for it. I’ll try to help to the best of my availability.