databrickslabs/dolly

[Question][Proposal] Libgreattao integration

nintyfan opened this issue · 4 comments

nintyfan commented

I am searching for AI/GPT way to add assistant options to Linux DE. I am select to integrate some AI/GPT with my own library, called libgreattao.

Libgreattao is GUI/CUI/Shell generating library. Main way to interact is load template (domain) and create paths (actions/data/others).

Code could looks like:
`
void *window = tao_create_window("/desktop/dialogs/question_dialog");
tao_add_handler(window, "/actions/ok", callback, 1);
tao_add_handler(window, "/actions/cancel", callback, 2);
tao_set_hint(window, "/message", HINT_NAME, "Error");
tao_set_hint(window, "/message", HINT_DESCRIPTION, "File exist. Overwrite?");

`
And that's is. We must only implement callback and we had nice question dialog with two buttons (if template was prefer to use buttons; modern design/tempalte set uses combobox for question_dialog domain, for example).

What I suppose to do is write new mode (so it will support also AI mode alongside GUI/CUI/Shell modes), called AI. We are giving NAME and DESCRIPTION. I wonder if AI could read these information to doing task (invoking actions), given by user. I thinking about giving new HINT type, called HINT_AI_DESCRIPTION_LANGUAGE, which will allow to pass query written in special language to give AI description, what happens or what some actions are doing.
Also, I must to add new file format, where I will describe what application can do (or extend Desktop file format to contains is_tao_AI_app field). For example:
[desktop] description[ai] = "burning CD's"
What do you think. Is this doable or are here somebody, who could join my trip or help me in other way?

srowen commented

I'm not quite sure what you are describing, like, I'm not sure what the output is supposed to be. If you want an LLM to understand a custom language, you are at least fine-tuning maybe pre-training a new small LLM, and that is expensive. Probably not a good fit. Just use natural language. But yeah not sure what this is doing.

nintyfan commented

I am searching for way to integrate assistant into libgreattao. So I need to use some AI, which understood natural language. Libgreattao is simple way to create application, which delivers a lot of way to interact with it - by GUI, CUI, Shell, network in past/future. I will integrate it with some assistant.
Way it could work:

  1. User give command , like burn ISO in /home/me/holiday_trip_movie.iso
  2. AI extension for libgreattao read DB contains program description and found TAO burner (in name), program to burn CD/ISO (in description)
  3. AI, using libgreattao extension, ran this program with --tao-na-AI-module switch
  4. Program was ran, create windows and put info main loop started onto stdout
  5. AI module read data from fd connected with app and see main loop ran, so asks for list each windows and each actions/etc. descriptions in language understood by AI
  6. AI sees there is /desktop/Objects/Attributes[1] window with /attribute/file_path path described like input iso path here and action /actions/burn
  7. So AI does think asked by user, send path to iso into /attribute/file_path and invoking /actions/burn

I wonder if it is possible to do with your tools and ask for help with implementing this if does. I do not have wide AI knowledge.

srowen commented

Maybe, I guess you could experiment with any LLM like this and langchain agents to piece together something like that. It's far beyond the scope of what an instruction following LLM does by itself.