[FR] Allow expiration and deletion of unpublished shares

[Tolriq]

Currently using a public alias for users to upload private files (like logs) so I can access them securely.

This is a follow up to #1275 (comment)

I'd like to have a way for those shares to expire and be deleted too, without having to publish them as no one should access them.

From your proposal I think a setting (per alias or global) to also delete unpublished share would be perfect, automatic share publish does not work for privacy and security reason.

[eikek]

Also somewhat related: #573

[Tolriq]

Small bump to know if there's short term plans or not on this? I stupidly stopped manually purging so should do it again if not planned as the list grows fast :p

[eikek]

Hi @Tolriq I'm sorry, but there is currently no ETA. I'd like to address this for the next version. But not sure when this will be.

[Tolriq]

Ok no problem back to deleting, this list grows :)

BTW don't know if it's normal but the /app/uploads only list 100 entries with no way to navigate the rest (But show the proper size)

Your Shares #100/215.79M

While the limit is not that a big deal, maybe still showing the actual number of files too could be nice.

[Tolriq]

Hi there, any chance to have this feature implemented ?

Do you take donations or anything? Some users are uploading multiple times things and manual cleanup really have became a pain :(

[eikek]

hi @Tolriq I'm currently too busy to tackle all that :/. how do you do the manual cleanup now? Perhaps we can have not-so-great-but-automatic solution anyways?

Edit: I haven't had a closer look, I'll do that and see if there maybe is a quick way.

[Tolriq]

Currently doing 4 clicks per uploads, click the id, then details, then delete then yes.

Any kind of script that can run in docker would be nice yes.
I currently store the files in a file folder so I can easily script the file part deletion, and a script that delete the entries if the files are missing could work too.

I just don't really know how the app work to easily try to figure out something.

Thanks.

[eikek]

ok, so that means you only need to delete shares you own, right?

[Tolriq]

Yes, I have an alias where people can upload things to. They no more have access to those files after and I can see them via the your share page.

They do have a validity time, but it's not actually applied so the files are currently never removed.

All I need is a way to either automatically remove the files when the validity time is expired, or eventually a quick delete way from the interface. But 4 clicks + delays and different positions when there's dozens of uploads per day is no more working.

[eikek]

ok, so this is a bit easier, since we can use the api to write a simple shell script. It would be more difficult if you had to delete things from other users. So this is something I typed together very quick:

#!/usr/bin/env bash
# you must have curl, jq and coreutils
set -euo pipefail

username="${SHARRY_USER:-}"
password="${SHARRY_PASSWORD:-}"
sharry_url="${SHARRY_URL:-}"
created_before="${SHARRY_REMOVE_BEFORE:-}"

temp_dir=$(mktemp -d "sharry-delete.XXXXXX")
#temp_dir="sharry-delete.A2zWWn"

json_header='Content-Type: application/json'
auth_tpl='{
  "account":$username,
  "password":$password
}'

trap cleanup 1 2 3 6 ERR

debug() {
    printf "%s\n" "$*" >&2
}

cleanup() {
    if ! [ "$temp_dir" == "." ]; then
        debug "Cleanup temporary files: $temp_dir"
        rm -rf "$temp_dir"
    fi
    exit
}

if [ -z "sharry_url" ]; then
    debug "The url to sharry is required"
    debug "  > Set env var SHARRY_URL"
    exit 1
fi

if [ -z "$username" ] || [ -z "$password" ]; then
    debug "sharry login data is needed."
    debug "  > set env vars SHARRY_USER and SHARRY_PASSWORD"
    exit 1
fi

if [ -z "$created_before" ]; then
    debug "No SHARRY_REMOVE_BEFORE date set"
    debug "  > Set this to a date, where any share created before that you want to be deleted"
    debug "  > Use RFC-3339 format, like 2006-08-14 02:34:56-06:00"
    exit 1
else
    before_ms=$(date "+%s" -d "$created_before")
    before_ms=$(( $before_ms * 1000 ))
fi

# login
auth_file="${temp_dir}/auth.json"
if ! [ -e "$auth_file" ]; then
    jq --null-input --arg username "$username" --arg password "$password" "$auth_tpl" | \
        curl -sfSL -H $json_header "${sharry_url}/api/v2/open/auth/login" -d @- > "$auth_file"
fi


# get list of shares
token=$(jq -r '.token' "$auth_file")
shares_file="${temp_dir}/shares.json"
if ! [ -e "$shares_file" ]; then
    curl -sfSL -H $json_header -H "Sharry-Auth: $token" "${sharry_url}/api/v2/sec/share/search" > "$shares_file"
fi

# filter out those to delete
delete_file="${temp_dir}/delete.json"
if ! [ -e "$delete_file" ]; then
    cat "$shares_file" | jq ".items[] |. +{until:(.validity + .created)}  | select(.until < $before_ms)" > "$delete_file"
fi

# present and ask
count=$(cat "$delete_file" | jq -r .id | wc -l)
cat "$delete_file" | jq
read -p "Delete these $count shares? (y/n)" confirm

if [ "$confirm" == "y" ]; then
    while read id; do
        echo "Deleting share $id"
        curl -sfSL -X DELETE -H "Sharry-Auth: $token" "${sharry_url}/api/v2/sec/share/$id"
    done < <(cat "$delete_file" | jq -r .id)
fi

of course, before running, read and verify :-) The idea is that you could run this script periodically, the input is your username and password and a timestamp. Every share whose created + validity is below that time will be deleted. (you need to adopt a bit to be non-interacitve and probably replace the fixed timstamp with the current date-time…)

[Tolriq]

OMG, I'm so sorry, I absolutely did not thought about checking what the API provided ...

Thanks so much for the script, it makes me gain a lot of time.

[eikek]

You're welome and no worries! Perhaps it is useful later as well (might be even myself 😄)

[Tolriq]

Ok so it kinda works but the API have the same limitation as the GUI it only returns the last 100 shares.

And in my case what I need to delete is often after the first 100.

Edit: Yes seems you have a couple of take(100) in the code :(

[eikek]

Ah yes, you are right, I always forget about it. It is left from the very first days of the project 😞. So, I'd then suggest to make it a bit uglier and search directly on the database. There you have all the power…

#!/usr/bin/env bash
# you must have curl, jq and coreutils
# you must set PGPASSWORD to the postgres password
set -euo pipefail

sharry_username="${SHARRY_USER:-}"
sharry_password="${SHARRY_PASSWORD:-}"
sharry_url="${SHARRY_URL:-}"
created_before="${SHARRY_REMOVE_BEFORE:-}"
pg_user="${PGUSER:-}"
pg_host="${PGHOST:-}"
pg_db="${PGDB:-}"

temp_dir=$(mktemp -d "sharry-delete.XXXXXX")
#temp_dir="sharry-delete.VCiw1f"

json_header='Content-Type: application/json'
auth_tpl='{
  "account":$sharry_username,
  "password":$sharry_password
}'

trap cleanup 1 2 3 6 ERR

debug() {
    printf "%s\n" "$*" >&2
}

cleanup() {
    if ! [ "$temp_dir" == "." ]; then
        debug "Cleanup temporary files: $temp_dir"
        rm -rf "$temp_dir"
    fi
    exit
}

if [ -z "sharry_url" ]; then
    debug "The url to sharry is required"
    debug "  > Set env var SHARRY_URL"
    exit 1
fi

if [ -z "$sharry_username" ] || [ -z "$sharry_password" ]; then
    debug "sharry login data is needed."
    debug "  > set env vars SHARRY_USER and SHARRY_PASSWORD"
    exit 1
fi

if [ -z "$pg_user" ]; then
    debug "Set PGUSER env var to the postgres user"
    debug "And PGPASSWORD to the corresponding password"
    exit 1
fi
if [ -z "$pg_host" ]; then
    debug "Set PGHOST env var to the postgres host"
    exit 1
fi
if [ -z "$pg_db" ]; then
    debug "Set PGDB env var to the postgres database for sharry"
    exit 1
fi


if [ -z "$created_before" ]; then
    debug "No SHARRY_REMOVE_BEFORE date set"
    debug "  > Set this to a date, where any share created before that you want to be deleted"
    debug "  > Use RFC-3339 format, like 2006-08-14 02:34:56-06:00"
    exit 1
else
    before_ms=$(date "+%s" -d "$created_before")
    before_ms=$(( $before_ms * 1000 ))
fi

# get list of shares
shares_file="${temp_dir}/shares.json"
if ! [ -e "$shares_file" ]; then
    # curl -sfSL -H $json_header -H "Sharry-Auth: $token" "${sharry_url}/api/v2/sec/share/search" > "$shares_file"
    psql -t -h "$pg_host" -U "$pg_user" "$pg_db" -c "
select json_object('id': s.id, 'name': s.name_)
from share s
inner join account_ a on a.id = s.account_id
where a.login = '$sharry_username' AND
   (extract(epoch from s.created) * 1000 + s.validity) < $before_ms" > "$shares_file"
fi

# present and ask
count=$(cat "$shares_file" | jq -r .id | wc -l)

if [ $count -eq 0 ]; then
    echo "Nothing to delete"
    exit
else
    cat "$shares_file" | jq
    read -p "Delete these $count shares? (y/n)" confirm

    if [ "$confirm" == "y" ]; then
        # login and delete each
        auth_file="${temp_dir}/auth.json"
        if ! [ -e "$auth_file" ]; then
            jq --null-input --arg sharry_username "$sharry_username" \
               --arg sharry_password "$sharry_password" "$auth_tpl" | \
                curl -sfSL -H "$json_header" "${sharry_url}/api/v2/open/auth/login" -d @- > "$auth_file"
        fi
        token=$(jq -r '.token' "$auth_file")

        while read id; do
            echo "Deleting share $id"
            curl -sfSL -X DELETE -H "Sharry-Auth: $token" "${sharry_url}/api/v2/sec/share/$id"
        done < <(cat "$shares_file" | jq -r .id)
    fi
fi
cleanup

Again, needs to change as you see fit. It is probably nicer to remove the created_before argument and use postgres to refer to the current date.

[Tolriq]

Thanks, I must admit I prefer the full API version as I can run it anywhere in my normal scheduled tasks.
In docker with the DB not exposed this is a little more complicated.

Do you think you can increase the value or make it configurable?

[eikek]

Yes, I think as a preliminary solution we can make a config setting