Agently-Talk-to-Control

English | 中文

Introduction

Agently-Talk-to-Control is an open-source chat text to control actions agentic workflow showcase project powered by Agently AI application development framework.

This project presents how to create a complex agentic workflow to analyse user requirement in natural language and decide to:

directly answer user's question with current environment information or
make and execute an operation plan with one or many operation actions or
reject user's request with explanation or even suggestion

The applicable scenarios for this project include:

Acting as an input understanding module for smart home systems, interpreting and processing complex user input expressions to execute commands.
Acting as an auxiliary understanding module after speech-to-text conversion for devices used in manufacturing, monitoring, medical assistance, etc., aiding operators in providing auxiliary control during operations.
Acting as an input understanding and action planning module in enterprise office settings or other environments with numerous standardized action request interfaces, helping to comprehend and plan complex user input expressions and extract standardized information for requests.
...

How to Use

Step 1. Clone this repo: git clone git@github.com:AgentEra/Agently-Talk-to-Control.git
Step 2. Edit SETTINGS.yaml to fill in your model's API Key or change to other model [View Agently Supported Models]
Step 3. Run python script: python app.py [Install Python from Python Official Website]
Step 4. Open Gradio UI page in your explorer: http://127.0.0.1:7860/ by default

Run Result Screenshot with Default Settings

The image down below is a run result screenshot with the default settings of this project that only contain 2 cameras in component pool to be controlled.

Try Extend Controllers

Of course, as a showcase project, developers can extend more controllers and components to be controlled as followed.

Step 1. Add component initial status into SETTINGS.yaml

For example: add 2 lights initial status

INITIAL_STATUS:
    light_status:
        light_a:
            power: 1
            brightness: 50
        light_b:
            power: 0
            brightness: 50

Step 2. Add controller functions into controllers/controllers.py

For example: add 2 controller functions of light control

def control_light_power(light_name, target_power):
    power_status = ("Off", "On")
    print(f"⚙️ [Turn On/Off Light]: { light_name } -> Power:{ power_status[target_power] }")
    return { "light_name": light_name, "power": target_power }

def control_light_brightness(light_name, brightness):
    print(f"⚙️ [Adjust Light Brightness]: { light_name } -> Brightness:{ brightness }")
    return { "light_name": light_name, "brightness": brightness }

Step 3. Add controller settings into SETTINGS.yaml according status settings in step 1 and controller functions' definitions in step 2.

CONTROLLERS:
    control_light_power:
        desc: "turn on or off a target light"
        args:
            light_name:
                $type: "'light_a', 'light_b'"
                $desc: "[Required]"
            target_power:
                $type: "int"
                $desc: "[Required]0 - Off, 1 - On"
        func: "control_light_power"
        # Will get environment information from target key before calling operation
        get:
            - "light_status"
        # Will update environment information to target key with target value
        # Placeholder <$variable_name> will be replaced by key values in controller function's return dict
        set:
            "light_status.<$light_name>.power": "<$power>"
    control_light_brightness:
        desc: "adjust a target light's brightness"
        args:
            light_name:
                $type: "'light_a', 'light_b'"
                $desc: "[Required]"
            brightness:
                $type: "int"
                $desc: "[Required] Range: 0(darkest)-100(brightest)"

OK, now 2 light components have been added to component pool. Let's try to talk to control them.

Everything seems to be great! Enjoy it!

Mainly Dependencies

Agently AI Development Framework: https://github.com/Maplemx/Agently | https://pypi.org/project/Agently/ | https://Agently.tech
Gradio: https://github.com/gradio-app/gradio | https://gradio.app/

Please ⭐️ this repo and Agently main repo if you like it! Thank you very much!