ToDo list

Question

ToDo list

jarun opened this issue 6 years ago · 51 comments

jarun commented 6 years ago

Cool!

👍1

jarun commented 6 years ago

OK!

👍1

Answer 1 · 2018-04-29T23:51:09.000Z

model 1 is currently used template, either see #251 or master branch

model 2

model 3

i'm playing on bookmark view and see if there is anything that can be changed

i actually want to ask the user about the current view

does it need favicon?
do they want to create their own view or only configure each part of bookmark view?
how do they want to view url? full, shortened, netloc only or None?
do they want to see description?
do they want to see the tag on the same cell (model 1, and model 2) or on different cell (model 3)?

also netloc only for url is inspired by reddit.

this can be expanded further by adding netloc filter
work really well only on entry with good title
could eliminate url entry

also i found bug where bukuserver can't find chart.js but can't reproduce it.

Answer 2 · 2018-04-30T02:21:49.000Z

I think model 2 without favicon and short netloc is model 1, right?

if we add favicon do we have to fetch for every bookmark each time?
only configure each part of bookmark view should be fine. We should see if we really need to provide any configuration options. We should keep minimal.
keep netloc only, and show full link on mouse hover. Saves space.
I think we can have a config to show or not show description OR we can limit the desc to 1 line by default and expand on click (I am not sure of the effort).
same sell and in model 1 or 2. If there are 2 many tags the vertical height would be too much.

Answer 3 · 2018-04-30T03:17:05.000Z

if we add favicon do we have to fetch for every bookmark each time?

kinda, basically it just add filter to get netloc and add this line to html

<img src='http://www.google.com/s2/favicons?domain={netloc}' />

related https://stackoverflow.com/questions/5119041/how-can-i-get-a-web-sites-favicon

keep netloc only, and show full link on mouse hover. Saves space.

netloc only is easy. to show link there is actually 2 option.

don't do anything. each browser show link (on chromium it is bottom left) when mouse over the link.
use bootstrap3 tooltip. example here https://getbootstrap.com/docs/3.3/javascript/#tooltips

...we can limit the desc to 1 line by default and expand on click (I am not sure of the effort).

through css. related https://stackoverflow.com/a/1318119/1766261
limit character

for expand on click maybe bootstrap's collapse https://getbootstrap.com/docs/3.3/javascript/#collapse

only configure each part of bookmark view should be fine. We should see if we really need to provide any configuration options. We should keep minimal.

I think we can have a config to show or not show description OR we can limit the desc to 1 line by default and expand on click (I am not sure of the effort

2 option for setting bukuserver

use os env variable. this is similar to flask setting
use config file. load config file first time server started

Answer 4 · 2018-04-30T03:57:55.000Z

fetching favicon seems fine then
don't do anything. each browser show link (on chromium it is bottom left) when mouse over the link. (this one)
through css. related https://stackoverflow.com/a/1318119/1766261 (this one would take care of variable page size too)
for expand on click maybe bootstrap's collapse https://getbootstrap.com/docs/3.3/javascript/#collapse (good and we don't need a config anymore)
use os env variable. this is similar to flask setting (this)

Answer 5 · 2018-04-30T04:15:30.000Z

@rachmadaniHaryono once these are done we can plan for the next release. The server seems to be in good shape. What do you think?

Answer 6 · 2018-04-30T04:26:33.000Z

OK hopefully this will be the last thing before next release

Answer 7 · 2018-05-03T15:47:22.000Z

Hello, please consider the following issues/ideas.

Browser integration through add-on: I don't think Bukubrow's Cargo.lock structure will work with *BSD OS, but don't quote me on that. Rusqlite will have to be a separate port, plus there needs to be a list of Depends under "Building".

Wish list suggestions (by order of importance):

http front end so as to access BM's from tablet/phone (example: searx/webapp.py). The Rest API probably does this but documentation is missing?
Scrapbook function: Preserve page or site, index it as special BM with option to not refresh (immutable).
User based authentication if needed, for multiple users from webapp interface. Probably difficult, as DB structure will need some modification, location of buku.sqlite becomes issue, and "run-as user" must be addressed?

These probably don't belong here, but I'll mention them anyway:

PDF: Save single web page as pdf, then text search ability for saved/indexed pdfs.
Note taking: If working on a specific topic, you want to jot down notes and place urls relevant to topic inside the notes.

Answer 8 · 2018-05-03T15:59:42.000Z

http front end so as to access BM's from tablet/phone (example: searx/webapp.py). The Rest API probably does this but documentation is missing?

currently working on that bukuserver.

@tom-i mention he can run on on espresso server and can access that.

to run it locally and make it accesible to device on same network, just host it on 0.0.0.0

User based authentication if needed, for multiple users from webapp interface. Probably difficult, as DB structure will need some modification, location of buku.sqlite becomes issue, and "run-as user" must be addressed?

this feature may or may not included on bukuserver goal. the actual reason there is development for it is just an accessible frontend through hosted server.

currently bukuserver don't need additional database, and it may kept it that way for some time in the future.

adding more feature is possible as long as it is simple one (see statistic page for example)

Scrapbook function: Preserve page or site, index it as special BM with option to not refresh (immutable).

PDF: Save single web page as pdf, then text search ability for saved/indexed pdfs.

not quite sure i can talk much about this but there is similar project which more focused on archiving link see https://github.com/pirate/bookmark-archiver

it have pdf, webarchive.org and html saving.

there is planned feature for search ability but look like it will not be merged

ArchiveBox/ArchiveBox#24

e: related maybe possible to create buku integration with this?

https://www.gregmcleod.com/rip-xmarks-hello-syncmarx/

Answer 9 · 2018-05-03T17:17:34.000Z

I don't think Bukubrow's Cargo.lock structure will work with *BSD OS

Please check with the Bukubrow project on this one. And to be frank I don't see the need for another browser plugin. You can add any bookmarks you want directly to buku from the browser oR import later OR export from Buku to the browser later.

Scrapbook function: Preserve page or site, index it as special BM with option to not refresh (immutable).

@rachmadaniHaryono I believe he is requesting for this flag on the webpage. We already have immutable support at an API level.

User based authentication if needed, for multiple users from webapp interface. Probably difficult, as DB structure will need some modification, location of buku.sqlite becomes issue, and "run-as user" must be addressed?

Is there a web-plugin with it's own database and can handle this? Otherwise, we can add this to the ToDo list for bukuserver. It makes the server operations truly private.

PDF: Save single web page as pdf, then text search ability for saved/indexed pdfs.

Re-iterating - we are not writing a private google search engine alternative. Use the notes field for additional text.

Note taking: If working on a specific topic, you want to jot down notes and place urls relevant to topic inside the notes.

Please use the notes option and relevant tags.

Answer 10 · 2018-05-03T19:31:33.000Z

Bukuserver accessible from local network is awesome feature, but next level (from my side) would be, to open port (5001) on my router and access bukuserver page from Internet.
So I have questions:

Is it save to open that port to the Internet?
Is there any security issues for the front end? I mean if someone can hack the page and can easily access to my router.
If there is new update (via git or so) for bukuserver / buku, should I install newer version via pip3 again? Or which steps I should follow? E.g. uninstall older version via pip3 and then git pull repository and then install new bukuserver?

Answer 11 · 2018-05-03T20:24:21.000Z

Is there a web-plugin with it's own database and can handle this? Otherwise, we can add this to the ToDo list for bukuserver. It makes the server operations truly private.

If additional database is on the plan, I'm thinking about adding this https://pythonhosted.org/Flask-Security/quickstart.html

Is it save to open that port to the Internet?

currently no
- form is not using csrf
- everyone can edit / view your saved bookmarks
- secret keys on app is using random number
- other security think I didn't know

Is there any security issues for the front end? I mean if someone can hack the page and can easily access to my router.

Main issue when open the port to internet is user access as mentioned above.

If there is new update (via git or so) for bukuserver / buku, should I install newer version via pip3 again? Or which steps I should follow? E.g. uninstall older version via pip3 and then git pull repository and then install new bukuserver?

Cmiiw on this

Using git + virtualenv will always get the latest file, which will put on local main folder, so it is no problem

I'm still doubt with pip installation, but it should also update bukuserver on user installation folder if it is installed correctly (not tested for installation using sudo pip but that is also discouraged)

Answer 12 · 2018-05-03T21:23:08.000Z

Hi and thanks for the input.

Re my PDF/Notes comment: Was only hoping to get some input on how to solve this.

@rachmadaniHaryono suggestion looks like the exact thing I was hoping for - much thanks!

Please use the notes option and relevant tags.

That would be the reverse of what I'm trying to figure out.
As mentioned, it was a shot in the dark to see if any idea would surface.

You can add any bookmarks you want directly to buku from the browser

You're talking about key-binding + shell script here as I understand?
Direct-access to the sqlite db would be a personal preference at read/review stage. I'll take it up on Bukubrow issues.

HTTP server-side (Flask-Security) would be a better choice IMHO, probably has better credential check integration and one could define custom db path for each user from there so that Buku its self will not need modification as multi-user.

@tom-i: I would suggest structuring it differently and NOT exposing your servers directly to internet unless you have a specific reason for public instances.

Install some VPN software on device facing the internet (or forward the traffic from router to VPN server).
Create self-signed certificates and install client certs to your laptop/tablet.
Connect to your home LAN through VPN (also means traffic gets encrypted) and access bukuserver as if you were resident on your LAN.

Thanks again & regards

Answer 13 · 2018-05-03T21:29:22.000Z

As note there is this doc on how to deploy flask http://flask.pocoo.org/docs/1.0/deploying/ but I haven't try it yet (mostly due to my internet connection issue)

Answer 14 · 2018-05-09T02:50:51.000Z

example of current markdown on pypi https://test.pypi.org/project/buku/3.7.post4/

Answer 15 · 2018-05-14T01:27:38.000Z

https://github.com/rachmadaniHaryono/Buku/tree/feature/append-tag-on-import

current behaviour

import file
bookmark exist
nothing changed

in my use case, i move the folder and expect it also reflected on buku tag.

there are actually two options, replace or append, but i choose append to prevent any undoable step and probability of missing tag.

currently on branch only work on html import

Answer 16 · 2018-05-14T03:26:13.000Z

Are duplicate tags parsed out and final tags ordered?

If you call parse_tags() all the duplicates will be removed and they will be ordered.

Yes, I think preserving all tags is a great idea.

Answer 17 · 2018-05-14T03:59:26.000Z

If you call parse_tags() all the duplicates will be removed and they will be ordered.

tags = row[1] + tags_in[1:]
tags = parse_tags([tags])

does that part on append_tag_at_index method didn't remove duplicate tag already?

Answer 18 · 2018-05-14T06:23:47.000Z

Yes it does. Just wanted to ensure. Didn't look at the code. :)

Answer 19 · 2018-05-20T03:57:38.000Z

Hi Jarun, I am newbie to opensource. Please give me beginner tasks.

Answer 20 · 2018-05-20T04:52:27.000Z

@kirakrishnan I guess the core library is already quite matured and stable. I was thinking of pagination but I guess the server with GUI already does it very well.

@rachmadaniHaryono do you have anything in the server side... or need help with testing the features?

Answer 21 · 2018-05-20T05:51:12.000Z

can't think anything simple for server.

imo the next step should be designing the additional database, adding more javascript capability, pluginize some parts

Answer 22 · 2018-05-20T10:47:41.000Z

@kirakrishnan let us know if you are interested in the items @rachmadaniHaryono mentioned and more importantly, you think you can handle those. If not sure, feel free to ask.

Answer 23 · 2018-05-20T13:27:10.000Z

@jarun I am interested in the items. I am very comfortable with sql and I have some hands-on experience in javascript. Please tell me little more about the task.

Answer 24 · 2018-05-20T13:33:30.000Z

@rachmadaniHaryono can you check if it's possible to create small tasks for the items?

Answer 25 · 2018-05-21T05:50:08.000Z

@jarun @kirakrishnan not quite sure how small it is, but here is some items that can be done:

ajax call to edit/create bookmark
example urls for bookmark

Answer 26 · 2018-05-24T23:35:12.000Z

@rachmadaniHaryono seeing that you have responded 4 days back without any reply yet, I believe @kirakrishnan has demonstrated his enthusiasm. Please work on the ideas yourself.

People tend to love the idea they would write amazing code in amazing projects, only they never really find the time to start. ;)

Answer 27 · 2018-05-25T01:44:54.000Z

will be late for this and not sure how long. this may affect that image/screenshot task for bukuserver. hope some irl problem will be resolved fast so i can get back to it.

Answer 28 · 2018-05-25T01:51:51.000Z

will be late for this and not sure how long.

no hurry! we just made a release.

this may affect that image/screenshot task for bukuserver.

we need the screenshot as soon as possible. people are visiting the page and don't know what to expect. not everyone would install a sever by looking at the description, they need something more attractive than that.

Answer 29 · 2018-05-25T02:21:37.000Z

create simple pr for that

unrelated

Answer 30 · 2018-05-29T08:10:57.000Z

@jarun some reply here to not make it cluttered

a481e9c

name and description should be separated.

not quite sure why it have to be changed to md, but i will follow it.

also unrelated to this edit, but this part should be added

#278

aea87cb

i'm ok with moving it imgur.

88b0467

ok with that

also i will hopefully able to replace the included js file this week based on @szlin advice

e: unrelated to this but imgur is blocked here in indonesia, but i don't know how much indonesian buku user affected here.

Answer 31 · 2018-05-29T12:34:41.000Z

Updated the documentation at c1c43a0. Please review.

Moved it to markdown as the main page is also in markdown.

Is there a working alternative to imgur? I used another site earlier but it went down. imgur still lives. ;)

Answer 32 · 2018-05-29T12:50:33.000Z

The update is ok

Moved it to markdown as the main page is also in markdown

Ok that is fine

Is there a working alternative to imgur? I used another site earlier but it went down. imgur still lives. ;)

Nothing as simple as imgur. Maybe keep imgur for now and add alternative when user ask

Answer 33 · 2018-05-29T12:53:48.000Z

Can you please add server-side identified tasks to the task list? In case someone wants to contribute.

Answer 34 · 2018-05-29T13:06:27.000Z

designing the additional database,
adding more javascript capability,
- ajax call to edit/create bookmark
- example urls for bookmark
pluginize some parts
use non minified JavaScript
user buku library
login/permission

That is just copy paste from comments above.

Some idea

image screenshot
web status (404, 200)
GitHub Link details,( programming language, Stars,etc)

Some crazy idea that I write, so I don't forget it next time (maybe idea for plugin)

Can't specify too details because I'm on mobile right now

Answer 35 · 2018-05-29T13:37:45.000Z

I think you'll have to make the points more elaborate. For example:

designing the additional database - which database, where does it help etc.
adding more javascript capability - which capability

like that.

Answer 36 · 2018-05-30T21:15:54.000Z

Hello guys,
is it possible that I have 2 different versions of bukuserver?
I've updated bukuserver yesterday and today too, kill bukuserver from running processes and run bukuserver via:

source env/bin/activate
bukuserver run --host 0.0.0.0 --port 5000

I saw that there are some icons before URLs, so I realized, that it's some kind of new feature.
Then I've restarted my NAS server and my bukuserver started via cron after reboot:
@reboot cd /home/nashome/programs/Buku/bukuserver/env/bin && ./bukuserver run --host 0.0.0.0 --port 5000
showed me, that there is no icons of the webs before URLs in bookmarks page.
So can I have different versions there? Or how should I start bukuserver via cron correctly, please?
Thx guys

EDIT:
Ok, so I've made some small shell script with

source env/bin/activate
bukuserver run --host 0.0.0.0 --port 5000

inside and I run it each reboot via cron now.

But question is still unanswered, why I've 2 versions there :)

Answer 37 · 2018-05-31T03:56:55.000Z

this is just guess, but buku installation on this virtualenv /home/nashome/programs/Buku/bukuserver/env/bin is not updated. try running pip install -e . once again on virtualenv after updating buku to latest version to reinstall it.

based on your description, the upgrade should be

kill bukuserver running instance
activate virtualenv
git fetch && pip install -e . buku repo or just run pip install -U buku to get latest version
restart

e:

is it possible that I have 2 different versions of bukuserver?

have 2 different virtualenv on each virtualenv install different version/commit you want

i.e. using v3.7 and v.3.8

mkvirtualenv venv1
workon venv1
git checkout v3.7
pip install -e .
deactivate
mkvirtualenv venv2
workon venv2
git checkout v3.8
pip install -e .
deactivate

(based on your description) the cron command would be

@reboot cd /home/nashome/programs/Buku/bukuserver/venv1/bin && ./bukuserver run --host 0.0.0.0 --port 5000
@reboot cd /home/nashome/programs/Buku/bukuserver/venv2/bin && ./bukuserver run --host 0.0.0.0 --port 5001

not sure about the correctness on last part because i use different virtualenv, but basically just put the correct virtualenv path based on your configuration.

also use different port or host

Answer 38 · 2018-06-29T19:47:59.000Z

Random thought :

This part is not filled yet

https://github.com/jarun/Buku/community

Also Imo #287 can be closed now

Answer 39 · 2018-06-30T00:20:59.000Z

I saw that earlier but left untouched. Closed the issue.

Answer 40 · 2018-07-29T17:11:33.000Z

https://www.ghacks.net/2018/07/27/mozilla-plans-to-remove-bookmarks-descriptions

Answer 41 · 2018-07-29T17:17:37.000Z

I don't think they will remove the particular field from the specification. They will probably stop importing the description field.

Answer 42 · 2018-07-29T17:25:15.000Z

I have added a new item in the ToDo list, thanks to your link. ;)

Answer 43 · 2018-08-06T09:03:11.000Z

doc for new feature taken from #298

Noting down stuff we need to handle if we do this (to be done gradually, not in one PR):

- Trigger auto-fetch of tags and description only when the options are not provided in cmdline
- Provide options in editor to update description and tags from web
- Extend immutable to keep description and tags untouched
- ~Limit auto-fetched tags to certain word count?~
- Flag `immutable` will now apply to title, desc and tags.

doc:

Answer 44 · 2018-08-16T22:30:04.000Z

doc on slow test

based on this rachmadaniHaryono/Buku@f6cbbe8 here are slow tests

124.81s call     tests/test_bukuDb.py::test_delete_rec_range_and_delay_commit
61.51s call     tests/test_bukuDb.py::TestBukuDb::test_search_by_multiple_tags_search_all
61.49s call     tests/test_bukuDb.py::TestBukuDb::test_add_rec
61.41s call     tests/test_bukuDb.py::TestBukuDb::test_search_by_multiple_tags_search_any
61.38s call     tests/test_bukuDb.py::TestBukuDb::test_search_keywords_and_filter_by_tags
61.36s call     tests/test_bukuDb.py::TestBukuDb::test_search_by_tags_exclusion
61.32s call     tests/test_bukuDb.py::TestBukuDb::test_get_rec_id
61.25s call     tests/test_bukuDb.py::TestBukuDb::test_append_tag_at_index
61.24s call     tests/test_bukuDb.py::TestBukuDb::test_searchdb
61.24s call     tests/test_bukuDb.py::test_delete_rec_on_non_interger[a-a-1-True]
61.24s call     tests/test_bukuDb.py::TestBukuDb::test_suggest_tags
61.23s call     tests/test_bukuDb.py::TestBukuDb::test_delete_tag_at_index
61.23s call     tests/test_bukuDb.py::TestBukuDb::test_append_tag_at_all_indices
61.22s call     tests/test_bukuDb.py::TestBukuDb::test_replace_tag
61.22s call     tests/test_bukuDb.py::test_delete_rec_index_and_delay_commit
61.21s call     tests/test_bukuDb.py::test_compactdb
61.21s call     tests/test_bukuDb.py::TestBukuDb::test_get_rec_by_id
61.20s call     tests/test_bukuDb.py::TestBukuDb::test_search_by_tag
61.20s call     tests/test_bukuDb.py::TestBukuDb::test_search_and_open_in_broswer_by_range
61.19s call     tests/test_bukuDb.py::TestBukuDb::test_search_and_open_all_in_browser
60.33s call     tests/test_bukuDb.py::test_print_rec_hypothesis
14.83s call     tests/test_bukuDb.py::test_browse_by_index

related pr

#301

notes:

maybe use vcrpy to cache
check if any sites used take too long to load. replace them or cache them
maybe split slow test and fast test
last fast test is https://travis-ci.org/jarun/Buku/builds/411723456
first slow test is https://travis-ci.org/jarun/Buku/builds/412647388

Answer 45 · 2018-09-18T02:02:58.000Z

I have some free time this weekend. I'll try to work on -n and -p if nobody is still doing it. :D

Answer 46 · 2018-09-18T02:05:55.000Z

@jpdasma please go ahead. That would be a great boost to user experience.

Answer 47 · 2018-12-23T18:53:19.000Z

using urwid for frontend? argument interface is good but sometimes it's painful to edit bookmarks. i think it should be in todo list.

Answer 48 · 2018-12-24T02:39:13.000Z

Sorry, at this stage of the project I wouldn't like someone to spend valuable bandwidth on interface. There are many features to be implemented.

Answer 49 · 2018-12-29T10:16:30.000Z

Rolled at #343.

Notes

Identified tasks