ToDo List
jarun opened this issue · 56 comments
Continued from #233.
Notes
The list below is a growing one. While suggesting new features please consider contributing to Buku. The code is intentionally kept simple and easy to understand with comments. We'll be happy to assist any new contributor. We need your help!
Some of the fresh-baked features may not have been released yet. Grab the master branch for those.
Identified tasks
- Paging option (10 by default if value is omitted in option). The current behaviour stays the same as users already have scripts based on it.
- Android app (with the same schema) (probably a separate project; we'll be glad to help)
- Show results before deletion
- Support keyword filtering (records having keywords
aandbbut notcandd) (thanks @saltyCatfish) - Support filtering by tags with search options (see #250) (thanks @saltyCatfish)
- Port feature from
googler/ddgr- omniprompt keyO - Show results with most search keyword matches on top (thanks @mosegontar)
- Text-mode user agent for Buku
- Copy search result URL to cipboard
- Read default Firefox profile name from
profiles.ini(see #212, thanks @alex-bender) - Support
--formatin search results (ref, thanks @mosegontar) - API documentation (comments need to be in NumPy format) (thanks @mosegontar)
- Auto-import: optionally add parent folder name as tag, ask for unique tag [like
importdb()] - Support custom colours (thanks @shv-q3)
- Generate packages on Travis-CI using PackageCore (see #189) (thanks @shaggytwodope)
- Search multiple tags, exclusion in tag search (thanks @mosegontar)
- Auto-import Firefox and Google Chrome bookmarks (thanks @alex-bender)
- Suggest tags those go together
- Append/overwrite/remove tags from prompt
- Rest API for webapps (thanks @kishore-narendran)
- Add more tests (ongoing activity @rachmadaniHaryono)
- A browser plugin (thanks @samhh for bukubrow)
- Text editor support (thanks @ZwodahS)
- Need a PyPI maintainer (thanks @shaggytwodope)
- Make refreshdb faster using threads (record updates should be synchronized)
- Show usage count in tag list
- Proxy support (thanks @denisfalqueto)
- Continuous search at prompt
- Add prompt help
- Specify custom DB file to class BukuDb (library usage, no exposed option)
- Move to urllib3
- Handle redirects using referrer masking. Example URL. Fixed with urllib3.
- Support URL shortening. This helps to share URLs. (see #92 for limitations)
- Make a bookmark title immutable via refreshdb()
- Markdown import/export
- Regex search
- Ubuntu PPA (thanks @shaggytwodope)
- Export specific tags to HTML
- Exact word match using REGEX. Make substring match optional.
- Delete all records based on a search result
- Delete multiple items, support combination of indices and ranges
- Append tags
- Travis CI integration
- Ubuntu deb package generation on new tag
- Merge bookmark database files (for users who work on multiple systems)
- Export bookmarks in FF or Chrome html format.
- Option to add folder names as tags while importing HTML (thanks @Mohammadkhalifa)
- Check and show upstream version
- Anything else which would add value (please discuss in this thread)
related to idea on #255
example: https://github.com/rachmadaniHaryono/Buku/tree/feature/server
screenshots
currently only for view function on bookmark and tags. the style is taken from bootstrap starter template https://getbootstrap.com/docs/3.3/examples/starter-template/
there could be more advanced feature such whitelist/blacklist tag, so there will be + or - sign after the tag to do that
Looks pretty awesome, @rachmadaniHaryono are you going to integrate function for having buku-server on some other network place (server) than localhost?
Awesome!!! We must have it in the main project :). Probably we can add a directory named buku_server and have the implementation there. Slowly you can add the functionality we have in core. I am pathetic with GUI stuff but I believe we will get contributors as we keep adding ToDo items.
i haven't tried with other but you can put it on 0.0.0.0 so other device can reach it
https://stackoverflow.com/questions/31904761/what-does-app-runhost-0-0-0-0-mean-in-flask
so run the command like this
bukuserver run --host 0.0.0.0see also more run option with bukuserver run --help
e: command is changed from buku-api to bukuserver
Probably we can add a directory named buku_server and have the implementation there
imo we can rename the api folder to buku_api or buku_server and change the setup for the new path. the name api is too common and may conflict with other package
e:
I am pathetic with GUI stuff but I believe we will get contributors as we keep adding ToDo items.
i am also learning and most of this is done with less html and without javascript for now.
e2: changed both folder and command to bukuserver
buku_server would be fine.
Where do I run the command buku-api run --host 0.0.0.0 in your tree?
first just install it in your virtualenv
pip install -e .
pip install -e .[server]the run the command
buku_serverwould be fine.
I meant we can rename api to buku_server.
the run the command
cool!
should the command be changed as well (from buku-api to buku-server)?
How about bukuserver in both cases?
done, will edit recent comment to match new command
@rachmadaniHaryono
What am I doing wrong? :-(
(env) $ pip install -e .[server]
Obtaining file:///home/tomik/programs/buku/Buku
buku 3.6 does not provide the extra 'server'
Installing collected packages: buku
Found existing installation: buku 3.6
Uninstalling buku-3.6:
Successfully uninstalled buku-3.6
Running setup.py develop for buku
Successfully installed buku
(env) $ bukuserver run --host 0.0.0.0
bash: bukuserver: command not found
EDIT: Ok after some missing modules in virtualenv I finally can run:
./server.py run
Unfortunatelly there is 404 on page:
(env) [api] $ ./server.py run
* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
127.0.0.1 - - [23/Mar/2018 23:21:07] "GET / HTTP/1.1" 404 -
127.0.0.1 - - [23/Mar/2018 23:21:07] "GET /favicon.ico HTTP/1.1" 404 -
127.0.0.1 - - [23/Mar/2018 23:21:18] "GET /bookmarks HTTP/1.1" 404 -
127.0.0.1 - - [23/Mar/2018 23:21:19] "GET /favicon.ico HTTP/1.1" 404 -
-
first make you are on this commit 250758b (run
git show) -
check if
setup.pycontain entrypoint like below
entry_points={
'console_scripts': ['buku=buku:main', 'bukuserver=bukuserver.server:cli']
},- check also if
setup.pyhave extra server dependencies
extras_require={
'HTTP': ['urllib3'],
'CRYPTO': ['cryptography'],
'HTML': ['beautifulsoup4'],
'tests': tests_require,
'server': server_require,
},- check if templates file exist on
bukuserver/templates/bukuserverfolder. there should be 4 html files - i also make it possible to run it directly on
server.pylike sopython bukuserver/server.pybut i don't recommend it, usebukuservercommand instead.
also this is optional, but report if favicon is still showing error
Sorry, Friday night is family movie time and I was off the terminal! :) We watched Paddington 2.
I guess we need to document all the steps under a README.md under bukuserver directory and link to it from the main README.md. Not urgent, but eventually.
For documentation I can only think of 2 points
- additional installation package. See more below
- running the server. Show how to run on user defined host & port.
I just realize that on ubuntu installation it is recommended to install from ppa. I haven't check if required server package is on ppa so I recommend only to install from pip for now
Also maybe inform user that this feature is still on experimental stage
👍
Hi @rachmadaniHaryono,
I've looked to that git show and I had this one.. so not yours :-/
I don't know, how is it possible
commit 474c403521f89798952ac1c5e4b7c09546411487 (HEAD -> master, origin/master, origin/HEAD)
Author: rachmadani haryono <rachmadaniHaryono@users.noreply.github.com>
Date: Fri Mar 23 11:30:13 2018 +0800
but git pull shows me:
Already up to date
also entry_points have not that last part about 'bukuserver=bukuserver.server:cli'
and extras_require there is missing 'server': server_require,
So it looks, that I'm using wrong commit or so :-/
Because I have there api folder and not bukuserver as you have it in your git repository :(
Buku [ ls -l master * ] 10:55 AM
total 296
-rw-r--r-- 1 tomik tomik 14461 Mar 23 22:04 CHANGELOG
-rw-r--r-- 1 tomik tomik 35142 Mar 23 22:04 LICENSE
-rw-r--r-- 1 tomik tomik 125 Mar 23 22:04 MANIFEST.in
-rw-r--r-- 1 tomik tomik 508 Mar 23 22:04 Makefile
-rw-r--r-- 1 tomik tomik 22988 Mar 23 22:04 README.md
drwxr-xr-x 2 tomik tomik 4096 Mar 23 22:33 __pycache__
drwxr-xr-x 4 tomik tomik 4096 Mar 23 23:12 api
drwxr-xr-x 5 tomik tomik 4096 Mar 23 22:04 auto-completion
-rw-r--r-- 1 tomik tomik 24529 Mar 23 22:04 buku.1
drwxr-xr-x 2 tomik tomik 4096 Mar 23 22:22 buku.egg-info
-rwxr-xr-x 1 tomik tomik 145022 Mar 23 22:04 buku.py
drwxr-xr-x 3 tomik tomik 4096 Mar 23 22:04 docs
drwxr-xr-x 5 tomik tomik 4096 Mar 23 23:05 env
-rw-r--r-- 1 tomik tomik 1531 Mar 23 22:04 packagecore.yaml
-rw-r--r-- 1 tomik tomik 58 Mar 23 22:04 requirements.txt
-rwxr-xr-x 1 tomik tomik 1877 Mar 23 22:22 setup.py
drwxr-xr-x 2 tomik tomik 4096 Mar 23 22:04 tests
-rw-r--r-- 1 tomik tomik 1042 Mar 23 22:04 tox.ini
EDIT: Aha.. I'm using @jarun git repository :-/
But it's strange I thought that I've used @rachmadaniHaryono :-/
I'll try to repair it..
Try this to get server branch https://stackoverflow.com/questions/9153598/how-do-i-fetch-a-branch-on-someone-elses-fork-on-github
E: you can also add other fork to your local git repo. See git remote add
buku/Buku [ git remote add rachmadaniHaryono https://github.com/rachmadaniHaryono/Buku.git
buku/Buku [ git fetch rachmadaniHaryono master ] 11:05 AM
From https://github.com/rachmadaniHaryono/Buku
* [new branch] feature/codecov -> rachmadaniHaryono/feature/codecov
* [new branch] feature/exception-on-malformed-url -> rachmadaniHaryono/feature/exception-on-malformed-url
* [new branch] feature/fix-idx -> rachmadaniHaryono/feature/fix-idx
* [new branch] feature/fix-server -> rachmadaniHaryono/feature/fix-server
* [new branch] feature/release-branch -> rachmadaniHaryono/feature/release-branch
* [new branch] feature/server -> rachmadaniHaryono/feature/server
* [new branch] master -> rachmadaniHaryono/master
buku/Buku [ git checkout -b rachmadaniHaryono/Buku master ] 11:07 AM
Switched to a new branch 'rachmadaniHaryono/Buku'
buku/Buku [ git pull rachmadaniHaryono/Buku rachmadaniHaryono/Buku ] 11:08 AM
fatal: 'rachmadaniHaryono/Buku' does not appear to be a git repository
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
What should I put to pull?
You can checkout the branch git checkout rachmadaniHaryono/feature/server
@rachmadaniHaryono
I'm noob with this, anyway how to pull your branch?
buku/Buku [ git pull rachmadaniHaryono/feature/server ] 6:11 PM
There is no tracking information for the current branch.
Please specify which branch you want to merge with.
See git-pull(1) for details.
git pull <remote> <branch>
If you wish to set tracking information for this branch you can do so with:
git branch --set-upstream-to=<remote>/<branch> rachmadaniHaryono/feature/server
I'm using yours, but git pull or git pull rachmadaniHaryono/feature/server doesn't do anything :-/
Try
git remote add upstream ...
Otherwise, wait till we merge this on Buku master branch and then you can test it out much easily.
Hmm doesn't help. How long that merge will take, please? A week?
We don't know at this moment. Please have patience.
git remote add rachmadaniHaryono https://github.com/rachmadaniHaryono/Buku.git
git checkout rachmadaniHaryono
git checkout -b rh-server rachmadaniHaryono/feature/server
That will create rh-server branch on you local repo
Currently I don't have my laptop until few hours later, so please check the command again
@rachmadaniHaryono ha.. cool that helps ;)
Now I see that page with my bookmarks ;)
Tomorrow, I'll try to install it on my EspressoBin server, to check if it's working with --host 0.0.0.0 ;)
Thx man..
note on doc conversion for pypi upload
2 example from last time (#233) is already fixed directly by pypandoc
now it seems example 1 to 9 is not formatted correctly
see https://test.pypi.org/project/buku/3.7/
pypandoc seems merge some line on one of those example
other than that removing raw html part should make generated rst file ready to upload
Fine by me. Not a major issue.
@jarun FYI, not sure if anyone else is working on it yet, but I plan on having a PR for paging this weekend.
@saltyCatfish that would be awesome!
Any interests on orgmode markdown .org export?
So that bookmarks can be integrated with GTD.
So that bookmarks can be integrated with GTD.
which service is GTD?
OK, I have an idea now. How about importing a .org file?
Also, would you raise a PR for this?
@saltyCatfish can you please let us know how you are planning to implement paging? We can have a short review of the design and all of us can be on the same page as to what we expect. Please raise an issue for the discussion and add the details so we can review it.
- http://blog.jonparrott.com/github-flavored-markdown-on-pypi/
- https://dustingram.com/articles/2018/03/16/markdown-descriptions-on-pypi
This means release branch can be deleted and no conversion needed for next release
👍
@saltyCatfish any update on the paging task?
No problem! I know it's tricky. You can get some clues from ddgr interface and code.
Thanks for the update!
@saltyCatfish any update on paging?
if anyone else wants to tackle this request, feel free
I don't think it's happening... even I have one pending task and I can't find time due to other things... but please try to spare some time on it.
@jarun How should this work? Does the user type "buku -paging-option 10 20" to see rows 10 - 20, or do you want a something like unixs less/more?
The results should be paged:
buku -n 10 ...
should show 10 results per page and one should be able to navigate with the n and p keys at the prompt. Just run googler and ddgr for a feel of it.
https://github.com/rachmadaniHaryono/Buku/tree/feature/statistic
no link yet to statistic page and user have to go directly e.g. 127.0.0.1:5000/statistic
the chart is using chartjs from http://chartjs.org/docs/latest/
it can more expanded, e.g.
- tag statistic
- most common word on description,
- domain to tag relationship
- nlt for description
FUN idea: Badge:
- 0, 10, 100, 1k bookmark/tag
- sfw/nsfw/category bookmark achieved (have x website bookmark with max or min number)
- unique domain award (when unique domain is the most dominant)
- website bookmark (only bookmark certain website or it is 80% of the bookmark)
here is kinda concept that i imagine for additional feature on bukuserver. user/developer/thirdparty can add plugin which can be shown on each page.
this is kinda tricky because
- how plugin is managed
- enable/disable plugin
- plugin location
- link to plugin
- html/js/css?
- position on main page
We do not track usage data. If we want to have this feature (and it looks great) we should have it disabled by default and enable only if user wants it.
plugin
there are issues around the DB desing as well
shared DB: pugin may corrupt it
separate DB: plugin may get out of sync
We do not track usage data. If we want to have this feature (and it looks great) we should have it disabled by default and enable only if user wants it.
Not quite sure if this is categorized as tracking usage data, because all data calculated each time per request and the result is not saved anywhere. Also no other database request other the first time getting all records. So Imo opinion it is similar to display number of your bookmarks
Of course unless the flask bootstrap or the included chart.js is compromised, it will be different story
should have it disabled by default and enable only if user wants it.
There is several way I can think of to do this
- soft confirmation : just yes/no confirmation either the first time or every time user enter statistic page
- environment variables : user have to set
BUKUSERVER_STATISTICto enable the page
dB
I have no idea how to approach this. I imagine that the new side database never hold any buku index and buku dB is always prioritized (except some exceptional case). The disadvantage is more database request
The side database can hold url, tag, title and produce/process something about it.
E.g.Url icon plugin (or something similar). first server get requested url after that looking for paired icon in side database
But still this is just some idea because I think it is already out of scope of what buku should be.
If people really want it may have to create another repo for this
and the result is not saved anywhere
Then how do you get the count? I am missing something here for sure.
Use python collection.Counter (?)
E: this line https://github.com/rachmadaniHaryono/Buku/blob/feature/statistic/bukuserver/server.py#L432
If the server goes down, the data would be lost, right?
Yes, no result data recorded
I think if we add this feature, users may want the data to be persistent. Hence, at some point we will have to work on making it persistent and add that option to enable it.