genai-impact/ecologits

This package should work with most LLM clients

LucBERTON opened this issue · 8 comments

LucBERTON commented

Description

This package should work with most LLM clients :

  • Open AI
  • Mistral AI
  • Anyscale
  • Anthropic
  • transformers
  • Cloud APIs

Solution

The first versions of this package may focus on OpenAI client only but, future versions should allow all features to be used with any LLM clients.

Considerations

N/A

Additional context

N/A

LucBERTON commented

Based on Anyscale documentation, I think they do not have their own client but only using OpenAI client :
https://docs.anyscale.com/endpoints/model-serving/openai-migration-guide

LucBERTON commented

I started working for the Anthropic client on this branch :
/feat/anthropic-sdk

I only added Anthropic dependency in the project and created the anthropic wrapper file.

samuelrince commented

Excellent @LucBERTON I'll test today or tomorrow! Fyi, Anthropic gives you $5 credit when you create a new account to test.

LucBERTON commented

I think I managed to make a working wrapper for Anthropic library.

I tested it like so :

# from anthropic import Anthropic
from client.anthropic_wrapper import Anthropic

api_key = <your_api_key>

client = Anthropic(
    # defaults to os.environ.get("ANTHROPIC_API_KEY")
    api_key=api_key,
)
message = client.messages.create(
    model="claude-3-opus-20240229",
    max_tokens=1024,
    messages=[
        {"role": "user", "content": "Hello, Claude"}
    ]
)
print(message.content)
print(message.impacts)

Outputs :

[ContentBlock(text="Hello! It's nice to meet you. How can I assist you today?", type='text')]
energy=0.15561 energy_unit='Wh'
  • 🚀1
aqwvinh commented

Hello @LucBERTON and @samuelrince , I can try working on transformers from HuggingFace, I'll do it on the branch /feat/transformers-sdk if that's ok with you

I'll try to work on text generation, cf the documentation, please redirect me if that's the wrong one :)

samuelrince commented

Hey @aqwvinh, we definitely have to support HF for sure! But I am not clear on how we should integrate the transformers lib into the package.

The transformers lib is made to run models locally, so in that use case, I would definitely recommend using codecarbon instead of genai_impact to get real energy consumption metrics. Though I can see one use case where you would prefer using genai_impact over codecarbon, that's when the software sensors for energy consumption are not available. That is the case when either the "sensor API" is not supported by codecarbon OR the API is just disabled or non-existent (specific hardware, cloud instances, etc.)

Another thing we can consider for HuggingFace is to support their Inference Endpoints. I never personally used it, but I think that in Python you either query the API directly or use the huggingface_hub package.

  • 👍1
aqwvinh commented

Hey @samuelrince , good to me! I've never used Inference Endpoints. I took a quick look to the documentation and I found this. I can try working on the huggingface_hub then