jarun/buku

ToDo List

jarun opened this issue · 40 comments

jarun commented

Continued from #174.

Notes

The list below is a growing one. While suggesting new features please consider contributing to Buku. The code is intentionally kept simple and easy to understand with comments. We'll be happy to assist any new contributor. We need your help!

Some of the fresh-baked features may not have been released yet. Grab the master branch for those.

Identified tasks

  • Paging option (10 by default if value is omitted in option). The current behaviour stays the same as users already have scripts based on it.
  • Support filtering by tags with search options (see #250)
  • Support keyword filtering (records having keywords a and b but not c and d)
  • Android app (with the same schema) (probably a separate project; we'll be glad to help)
  • Port feature from googler/ddgr - omniprompt key O
  • Show results with most search keyword matches on top (thanks @mosegontar)
  • Text-mode user agent for Buku
  • Copy search result URL to cipboard
  • Read default Firefox profile name from profiles.ini (see #212, thanks @alex-bender)
  • Support --format in search results (ref, thanks @mosegontar)
  • API documentation (comments need to be in NumPy format) (thanks @mosegontar)
  • Auto-import: optionally add parent folder name as tag, ask for unique tag [like importdb()]
  • Support custom colours (thanks @shv-q3)
  • Generate packages on Travis-CI using PackageCore (see #189) (thanks @shaggytwodope)
  • Search multiple tags, exclusion in tag search (thanks @mosegontar)
  • Auto-import Firefox and Google Chrome bookmarks (thanks @alex-bender)
  • Suggest tags those go together
  • Append/overwrite/remove tags from prompt
  • Rest API for webapps (thanks @kishore-narendran)
  • Add more tests (ongoing activity @rachmadaniHaryono)
  • A browser plugin (thanks @samhh for bukubrow)
  • Text editor support (thanks @ZwodahS)
  • Need a PyPI maintainer (thanks @shaggytwodope)
  • Make refreshdb faster using threads (record updates should be synchronized)
  • Show usage count in tag list
  • Proxy support (thanks @denisfalqueto)
  • Continuous search at prompt
  • Add prompt help
  • Specify custom DB file to class BukuDb (library usage, no exposed option)
  • Move to urllib3
  • Handle redirects using referrer masking. Example URL. Fixed with urllib3.
  • Support URL shortening. This helps to share URLs. (see #92 for limitations)
  • Make a bookmark title immutable via refreshdb()
  • Markdown import/export
  • Regex search
  • Ubuntu PPA (thanks @shaggytwodope)
  • Export specific tags to HTML
  • Exact word match using REGEX. Make substring match optional.
  • Delete all records based on a search result
  • Delete multiple items, support combination of indices and ranges
  • Append tags
  • Travis CI integration
  • Ubuntu deb package generation on new tag
  • Merge bookmark database files (for users who work on multiple systems)
  • Export bookmarks in FF or Chrome html format.
  • Option to add folder names as tags while importing HTML (thanks @Mohammadkhalifa)
  • Check and show upstream version
  • Anything else which would add value (please discuss in this thread)

Since importing from different sources would inevitably introduce many duplications, a method of merging duplicated urls is crucial, I would suggest to pick the shortest title, and merge the tags.

jarun commented

Are you interested in implementing the feature?

Yes, I will try, I have only very little knowledge of python and know nothing about sql.

jarun commented

I'm afraid at this stage of the project we need a little more experienced hands. BTW, Buku doesn't store duplicate URLs. So the first URL and associated data would be stored, rest would be discarded. I've thought about merging earlier but then, as long as the URL is in (and the user thought at some point the entry is complete with the first set of data) I think we are reasonably safe.

I think storing bookmark save date would be nice.

@flashus commented on 25 січ. 2018, 06:37 GMT+2:

I think storing bookmark save date would be nice.

So I think it may be necessary for me too

@flashus , @putsan

Hi, please read the project policy:

no history, obsolete records, usage analytics or homing.

see also #103

Hello, I saw your thread here.
https://www.reddit.com/r/archlinux/comments/7tk50u/need_aur_maintainer/

I could volunteer for maintaining the AUR package, I have some experience in creating Fedora and Debian package. I might also look at the code sometime and contribute (currently using Python for work)

jarun commented

That's great! Can you please share your AUR username so I can add you as a maintainer?

Someone can make a package for Solus Os?

jarun commented

@putsan please contact the Solus OS team to add Buku to their repository. And probably some more of my utilities ;).

jarun commented

@jpdasma can you please share your AUR username?

jarun commented

@rachmadaniHaryono would it be possible for you to take care of the PyPi update? I am not expecting a next release of Buku soon.

Hello, although I have experience in creating Fedora and Debian packages due to work related reasons, I have minimal experience in creating an AUR package (thus still no AUR username). Is it okay if I could still maintain it? I'll try creating an AUR account later today.

Hello, I have created an AUR account with usernamen jpdasma. I'm willing to be the maintainer for this if you are okay with a maintainer with little experience with regards to AUR :-)

@jarun , i will try it on the next release. ping me when new release is finished

jarun commented

@jpdasma that should be fine. There's one change in the last release of Buku you should be aware of - you can remove python-requests from the list of dependencies.

I have added you as a co-maintainer on AUR. Welcome!

jarun commented

@rachmadaniHaryono we are not going to have a new release soon. The last release is not updated on PyPi yet.

@jarun I'll take note of that

I will also try publishing an update to the AUR later today.

jarun commented

@jpdasma that's awesome!

@jarun

  • it seems i can't upload to pypi.
Upload failed (403): The user 'rachmadaniHaryono' is not allowed to upload to project 'buku'. See https://pypi.org/help#project-name for more information.
error: Upload failed (403): The user 'rachmadaniHaryono' is not allowed to upload to project 'buku'. See https://pypi.org/help#project-name for more information.

based on error, i need to get permission to upload it

  • so i look at the pypi page and trying to fix the page.

    • first i use pypandoc to convert readme.md (https://stackoverflow.com/a/26737672/1766261)
    • write the converted text to readme.rst and check it with rstcheck. i found out that:
      • these example 336. DuckDuckGo > https://ddg.gg and 336. DDG (L) > https://ddg.gg # privacy,search engine Note that URL must precede tags. are recognized as enumerated list by rst.
      • between first and second example need newline.
    • those method above still didn't work, so as a test i remove the raw block.

    the result https://test.pypi.org/project/buku/3.6.post8/

  • here come the new problem, how do we maintain it?

    • remove the html part from readme.md. hopefully pypandoc can successfully convert it directly.
    • maintain separate branch. this branch contain readme.rst which should be generated by pypandoc and applied the above [pypi page fix] manually

for the second method i hope it can successfully upload to pypi. on these test i have to rename the version multiple time. i added 'postn' suffix so i can upload it to test.pypi (based on this https://stackoverflow.com/a/13919964/1766261)

jarun commented

I have added you for the permission issues. Try once more. For the other part, do as you see fit.

https://pypi.python.org/pypi/buku it is done

my browser load the page stop few seconds between these lines

        w [editor|id]          edit and add or update a bookmark
        ?                      show this help

maybe rst need few time to process ? show this help due to leading question mark

e: and i have no idea why the test.pypi format the readme back to raw rst

e2:https://github.com/rachmadaniHaryono/Buku/tree/feature/release-branch

e3: guide for uploading to pypi

  1. register to pypi and test.pypi
  2. create ~/.pypirc and fill it with your config (example below)
  3. remove readme.rst if exist
  4. generate readme.rst
  5. applied following fix:
    5.1 check any text that maybe recognized as enumerated list
    5.2 newline between example section
  6. upload first to test.pypi. command: python3 setup.py sdist upload -r pypitest
  7. if upload success but result is not as intended, change the version with postn-suffix format. fix the program and go to number 6.
  8. if upload sucsess and result is as intended:
    7.1 check the program version. maybe rolled it back to original if possible
    7.2 upload it to pypi. command: python3 setup.py sdist upload -r pypi

code to generate readme.rst, require pypandoc from pip and pandoc e.g. from ubuntu repo
(note that everytime setup.py executed it will generate readme.rst if not exist)

import pypandoc
long_description = pypandoc.convert_file('README.md', 'rst')
with open("README.rst", "w") as f:
    f.write(long_description)

example pypirc

[distutils]
index-servers =
  pypi
  pypitest

[pypi]
repository=https://pypi.python.org/pypi
username=RachmadaniHaryono
password=<password>

[pypitest]
repository=https://test.pypi.org/legacy/
username=rachmadaniHaryono
password=<password>
jarun commented

No issues with loading here! 👍
Where's the test.pypi file? Link please.

add the doc see the 3th edit of my comment above

Where's the test.pypi file? Link please

you mean this ?

https://test-files.pythonhosted.org/packages/c3/74/c4b3cedaf04e262d9a78bf1d291c08ddea0882d336c19baf5e2d78abe691/buku-3.6.post8.tar.gz

on this page https://test.pypi.org/project/buku/3.6.post8/#files

it is basically this commit 83165e2

jarun commented

Ignore ;). As long as users can download the package we are good!

jarun commented

@mosegontar would you have the time to try the first item? I added it today.

@jarun,hi! thanks for pinging me. I'll have to do some research how to do this in SQL and modify the search queries, but I can take a crack at it this weekend, if that's okay.

To clarify the task:
If bookmarkA has the title "title1" and the tag "tagA", and bookmarkB has the title "title2" and tag "tagA", and we run buku -s tagA title2, both bookmarks should be returned in the results (current behavior) and bookmarkB should appear before bookmarkA in the results (new behavior).

And this should apply to all searches with 'any' logic (but, I assume, not 'all' logic)?

jarun commented

Yes, your understanding is correct. 👍
Also, we are using sqlite in Buku so research based on sqlite.

@jarun, just an update. I finally figured out the query:

SELECT id, url, metadata, tags, desc,
    CASE
        WHEN (tags REGEXP 'python' OR URL REGEXP 'python' OR metadata REGEXP 'python' OR desc REGEXP 'python')
            THEN 1 ELSE 0 END
        + CASE WHEN (tags REGEXP 'ruby' OR URL REGEXP 'ruby' OR metadata REGEXP 'ruby' OR desc REGEXP 'ruby')
            THEN 1 ELSE 0 END
        + CASE WHEN (tags REGEXP 'programming' OR URL REGEXP 'programming' OR metadata REGEXP 'programming' OR desc REGEXP 'programming')
            THEN 1 ELSE 0
    END score
FROM bookmarks
WHERE score > 0
ORDER BY
score DESC;

Will hopefully be able to make a PR this week, but if you're in a hurry I am happy to hand it off too.

jarun commented

Please take your time.

@jarun I've created a snap package for this, for easy installation across my local Linux systems. It's very easy to maintain (when the snap is installed it pulls directly from your latest release, installs all the dependences etc).

Would you be interested in publishing the snap in the Ubuntu snap store? It'd make installing over most popular Linux distros as easy as 'snap install buku'. If you'd like me to assist in publishing the snap, or even maintain the published snap on your behalf, please let me know.

jarun commented

I do not maintain any downstream packages. If you can maintin I have no issues.

Please ensure the changes in the codebase are minimal.

Generally I discourage snaps but Buku has a few deps so we can have it.

@jarun Thank you for permitting your application to be snapped, much appreciated. The snap would build directly from your latest released version, so no changes to your codebase at all. I've got the ball rolling - it's now pushed to a buku repository in the Ubuntu snap store and should be published soon.

@jarun i create https://github.com/jarun/Buku/wiki/Upload-to-pypi, so anyone who want to upload the program to pypi can look at that instead searching in todo list thread

jarun commented

thanks @Aninstance and @rachmadaniHaryono!

jarun commented

@mosegontar would you have the time to work on the first item (paging support)?

@jarun hi! my apologies for the delay. I'm pretty tied up lately, so I don't think I'd have time to work on it, but thanks for checking with me.

jarun commented

@mosegontar Hey no problem!

jarun commented

Rolled at #251.