Suggestion: no.delete

[ze6killer]

Hello,

How are you doing these days?
I could see that the updates are still coming regularly and must admire your tenacity: congrats!

I just thought about a possible feature:
as I am a little paranoid about loosing my whole stock of photos, I keep getting worried that something goes wrong -> the photos get deleted from my NAS for some reason (not necessarily due to the script) and then the scrips comes along, syncs everything and my backup on Flickr disappears too! (just writing it I'm getting nervous)

-> suggestion: what about a "no delete" option in the .ini file? :>
Maybe something that instead of deleting the pictures simply tags them as "to be deleted" so that the user can check them before deleting them.

Hope this helps.

Take care,
Ze

[oPromessa]

Good idea. Will check it!

[oPromessa]

@ze6killer for this I have two options if one file is to be deleted. The tool would not delete the file from flickr:

1. and would also remove the file entry from the local database, meaning that on subsequent runs it will never try to delete this file again.
2. but would keep the file entry on the local database, meaning that on subsequent runs it will continue to try to delete this file again (unless of course the no.delete option is enabled).

On the second option you basically defer the "cleanup/delete" to a later run.
On the first option you basically loose any chance of future deletion.

Of course this could be an additional sub-option of the no.delete option but is complicating things too much.

So, my question, you'd go for option 1 (remove) or 2 (keep) the local listing of the file on the local DB.

Thanks, oPromessa

[ze6killer]

Hello,

The problem I see with option 2 is that, after some time, you will know that a lot of pictures will be deleted when you change the option in the ini file, but you won't remember which ones.
In doubt, you might never trust yourself to switch the parameter to 'deletion' again, for fear of deleting something which you had forgotten was listed there.

So I would go for option 1, but also put a tag on the pictures in Flickr (like "delete me" or "ze4ever") so that the user can easily find which ones were listed and delete them himself if he so wishes.

Does this make sense to you?

Regards,
Ze

[oPromessa]

Now available on Version 2.8.7-alpha36 onto testpy.py.

• use the argument --no-delete-from-flickr in the command line of uploadr.py
• Appreciate any help in testing.
• ❗️One remark I'd make is that if you move around the pics from folder to folder too much you'll end up with valid uploaded pics but with the nodelete tag; so you'll have to manage that yourself.
  • Still to be considered an improvement to delete the tag nodelete from the correctly uploaded pics. Would slow things down... but for now I'll leave it as this.

• To install this alpha-version 2.8.7-alpha.36 via PIP (Check readme for instructions) you should use a the following command-line:

pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple flickr-uploader==2.8.7a36 --prefix=~/apps/Python/ --no-cache-dir

[ze6killer]

Hello,

It's been nice to hear from you!
I've been intending to test for the last past days, but it seems the summer has not made me more computer-proficient.

Untill now I had upgraded manually. I tried to follow your recommendations and managed to kinda install pip but couldn't update the PATH correctly I guess.

As a result, today I managed to run this:

cd dev/
python ../apps/Python/lib/python2.7/site-packages/pip install flickr-uploader --prefix=~/apps/Python

... but the files were installed in the site-packages subfolder instead of ~/apps/Python (which did exist though)

I'll try to install in the right folder again. If I don't manage, I will update the .cron script to point to site-packages... :-/

Sorry for being so unskilled :']

[oPromessa]

Hi there.

• If you follow Step 3. from the Readme you should then have the pip utility installed.
• Remark: To run the pip utility you do not need to use the prefix python. Simply run pip.
• You can then follow the instructions on Step 5.1 from the Readme.

Thanks!

[oPromessa]

• BTW... regarding PIP, if a previous version of flickr-uploader was already installed on your system which you can check with the following command pip list ...

dev$ pip list

• IMPORTANT NOTICE To upgrade the version, one has to use the pip --upgrade option:

dev$ pip install flickr-uploader --prefix=~/apps/Python --upgrade

[ze6killer]

Ok, so long story short: I just needed to add :~/apps/Python/bin to my PATH.
This was not obvious for a noob like me.

Various remarks about the updating process

1. I don't understand those lines:

# Compatible with different installation modes of uploadr.py: PIP or Mannual
RUNCMD=`which uploadr.py`
RUNCMD=${RUNCMD:=$FOLDER/uploadr.py}

What should I put here?
Because PIP installed uploadr.py in the apps/Python/bin folder.

So I put: FOLDER=/volume2/homes/pTim/apps/Python/bin

2. By default you put FOLDER=~/apps/flickr-uploader-2.8.6 but for me this ends up being root. So this means I have to edit this line after each update.

3. If the folder flickr-uploader-2.8.6 is not created, the script fails: does this mean I have to create it manually after each update?
   (actually right now I use a folder without the version number it the name)

4. The .cron and .ini files are overwritten with the pip update. So if one forgets to back them up before the update, all parameters are reset.
   Could there be a way to automatically copy from the previous version the parameters that are not new?
   Or maybe easier at first: create a .previous_ini file?

Trying to test the nodelete parameter

1. I did not understand your remark about moving the files around:
   ❗️One remark I'd make is that if you move around the pics from folder to folder too much you'll end up with valid uploaded pics but with the nodelete tag; so you'll have to manage that yourself.

2. I couldn't get the script to work.
   .err message:

Traceback (most recent call last):
  File "/volume2/homes/pTim/apps/Python/bin/uploadr.py", line 68, in <module>
    import lib.FlickrUploadr as FlickrUploadr
ImportError: No module named lib.FlickrUploadr

3. I would have put to-be-deleted instead of nodelete but this is a detail :)

[oPromessa]

To your points...

0. PATH variable. My bad! What I do is I edit ~/.bashtc file and include the following lines. This way each time I login I get these variables set correctly.

export PYTHONPATH=~/apps/Python/lib/python2.7/site-packages
PATH=$PATH:~/apps/Python/bin

Various remarks about the updating process

1. No need to change those lines.

2. When installed via PIP, FOLDER is simply the folder where you'd like the log files to be written.

3. That's fine if you use a FOLDER without a version. Will update the default value as it does not make sense to have the version number. I have it as I test around multiple versions at the same. But you're right.

4. Yup. You're right. Will look into that!

Trying to test the nodelete parameter

1. In one previous message you mentioned you often reorganized your files within the folders movig pics from folder A to B. This resulted in flickr-uploader deleting pics from folder A and re-uploading them to folder B. Reason why you asked for the nodelete option.

   • Now with the --no-delete-from-flickr the pics in folder A will be marked nodelete and not actually deleted.
   • If you happen to move pics back from folder B to A...
   • flickr-uploader with the --no-delete-from-flickr will mark the pics in folder B with nodelete...
   • and the pics in fodler A (which were not deleted) are now valid but with the tag nodelete
     This is my warning.

2. Bummer! make sure PYTHONPATH variable is defined. And send me a bit more logs.

3. In fact the option --no-delete-from-flickr accepts a tag which you can name whatever you want. Try not to use spaces! --no-delete-from-flickr to-be-deleted

[ze6killer]

Edit:

Changed PATH from:

export PYTHONPATH=.:/var/services/homes/pTim/apps/Python/lib/python2.7/site-packages:/usr/local/lib/python:/usr/lib/python:/usr/lib/python2.7`

To:

export PYTHONPATH=.:/var/services/homes/pTim/apps/Python/lib/python2.7/site-packages:/usr/local/lib/python2.7:/usr/lib/python2.7

Now I get:

�[35m[1934][2018.10.03 11:55:09]:[MainProcess]�[0m[CRITICAL]:[root] Config folder [/volume2/homes/pTim/apps/Python/bin] and/or INI file: [/volume2/homes/pTim/apps/Python/bin/uploadr.ini] not found or incorrect format: [[Errno 2] No such file or directory]!
�[35m[1934][2018.10.03 11:55:09]:[MainProcess]�[0m[ERROR   ]:[root] +++ #662: Invalid sys.argv INI file [/volume2/homes/pTim/apps/Python/bin/uploadr.ini].
�[35m[1934][2018.10.03 11:55:09]:[MainProcess]�[0m[CRITICAL]:[root] FINAL ini_file:[/volume2/homes/pTim/apps/Python/etc/uploadr.ini]
�[35m[1934][2018.10.03 11:55:10]:[MainProcess]�[0m[CRITICAL]:[root] Exception: <type 'exceptions.IOError'>: [Errno 13] Permission denied: u'/root/.flickrlock'
�[35m[1934][2018.10.03 11:55:10]:[MainProcess]�[0m[CRITICAL]:[root]   File "/volume2/homes/pTim/apps/Python/bin/uploadr.py", line 684, in <module>
    FILELOCK(open(MY_CFG.LOCK_PATH, 'w'),

Which is curious because PIP put the .ini file in the etc folder, not the bin

For information in my .cron I have :

python2.7 $RUNCMD -v -b --dry-run --no-delete-from-flickr delete-me -p 10 >>$FOLDER/uploadr.out 2>>$FOLDER/uploadr.err

[oPromessa]

• Yes the .ini should now be on the /etc folder. Thats ok!
• Can you share with me the uploadr.ini file. Mask the api_key and secret entries.
• Issue is on the FOLDER definition. It is set to the current working directory by default.
• It seems your current working directory is /root and that's where the app tries to create the .flickrlock and flickrdb control database. For the .flickrlock it does not have permissions to write so it fails.
• So my suggestion edit FOLDER on the upload.ini to point out to the folder where you have the previous flickrdb database (so that it can use it now) and where you have write permissions.

[ze6killer]

Hello,

Well I guess it's the same thing as I mentioned before: my user name on the NAS is pTim but whenever I use ~/xxx in a script it points to root
Looks like the same happens with FOLDER = os.path.abspath(os.getcwd())

I edited the ini file
Then I launched once from the terminal to perform the authentication procedure again.
Then I launched again from .cron and got this error:

�[35m[1813][2018.10.03 15:03:08]:[MainProcess]�[0m[CRITICAL]:[root] Config folder [/volume2/homes/pTim/apps/Python/bin] and/or INI file: [/volume2/homes/pTim/apps/Python/bin/uploadr.ini] not found or incorrect format: [[Errno 2] No such file or directory]!
�[35m[1813][2018.10.03 15:03:08]:[MainProcess]�[0m[ERROR   ]:[root] +++ #662: Invalid sys.argv INI file [/volume2/homes/pTim/apps/Python/bin/uploadr.ini].
�[35m[1813][2018.10.03 15:03:08]:[MainProcess]�[0m[CRITICAL]:[root] FINAL ini_file:[/volume2/homes/pTim/apps/Python/etc/uploadr.ini]

Here is the
uploadr.out

[oPromessa]

• Ok. Those error messages are in fact more debugging than actual errors. Will change that.

• The .out file seems fine:

Photos count:
                Local:[  6189]
               Flickr:[ 12018]	[  5829] :Flickr-Local
Not in sets on Flickr:[   410]

• you’re using version 2.8.7-alpha36. Do install version 2.8.7 with pip upgrade and make sure to save the etc/ files (.ini and .cron).

[ze6killer]

So, just to let you know:

1. ❗️One remark I'd make is that if you move around the pics from folder to folder too much you'll end up with valid uploaded pics but with the nodelete tag; so you'll have to manage that yourself.

I understand now: if I put a pic, the script uploads it, if I remove it, the nodelete is added, if I put it again, the tag is NOT removed.
I think it's ok: as you said earlier, the goal is not to input/move/remove pictures from folders all the time.

2. When installed via PIP, FOLDER is simply the folder where you'd like the log files to be written

Ok, I guess I'll just use etc actually. This way I have the input (.ini and .cron) and output files in the same folder.

3. Is there anything else I can help you to test? :)

[ze6killer]

Latest run, without the dry-run option this time but with the nodelete option:

uploadr.ini
uploadr.err
uploadr.out
uploadr.cron

C ya :)

[oPromessa]

Thanks for the feedback!

1. Closed.
2. Ok.
3. Simply check if the "deleted" pics are marked correctly. Thanks!

• For some reason you are still using v 2.8.7-alpha36.
• Do install version 2.8.7 with pip upgrade and make sure to save the etc/ files (.ini and .cron):

pip install flickr-uploader --prefix=~/apps/Python --upgrade

[ze6killer]

Hi,

Here some more logs. Things seem to be working fine.
uploadr copie.err.txt

uploadr copie.out.txt

Note: although I have this in the .ini file, my .out file was 34MB big.
ROTATING_LOGGING_FILE_SIZE = 10*1024*1024 # 25 MBytes -> I changed to 10 for Github upload

Also, I have deleted an older video from a folder. As expected it had the delete-me tag on Flickr, but what surprised me was that Flickr listed it as having been uploaded today.

Ze

[oPromessa]

Finally @ze6killer ...

• the Rotating settings are for the file defined under ROTATING_LOGGING_PATH and do not apply to the .out file.