CLI Tokenizer

CLI Tokenizer is a simple command-line tool that allows you to count tokens in text using the tiktoken library.

Installation

You can install CLI Tokenizer using the following command:

curl -L https://raw.githubusercontent.com/ericciarla/cli-tokenizer/main/install-cli-tokenizer.sh | bash

This script will:

Create a ~/bin directory if it doesn't exist.
Download the cli-tokenizer.sh script and save it as tokens in the ~/bin directory.
Make the tokens script executable.
Attempt to add ~/bin to your PATH if it's not already there.

After installation, you may need to reload your shell configuration:

For Bash:

source ~/.bash_profile

For Zsh:

source ~/.zshrc

Once installed, you can use the tokens command followed by your text:

tokens hello world

This will output the token count for the given text.

If you encounter issues after installation, follow these troubleshooting steps:

Verify the script exists:
```
ls -l ~/bin/tokens
```
This should show the tokens file in your ~/bin directory.
Check if ~/bin is in your PATH:
```
echo $PATH
```
Look for /Users/yourusername/bin or ~/bin in the output.
Add ~/bin to PATH if it's missing: For Bash:
```
echo 'export PATH="$HOME/bin:$PATH"' >> ~/.bash_profile
```
For Zsh:
```
echo 'export PATH="$HOME/bin:$PATH"' >> ~/.zshrc
```
Then reload your configuration with source ~/.bash_profile or source ~/.zshrc.
Run the script directly:
```
~/bin/tokens hello world
```
If this works but tokens hello world doesn't, it's likely a PATH issue.
Check the script content:
```
cat ~/bin/tokens
```
Ensure the script content is correct and complete.
Verify Python and tiktoken: The script requires Python 3 and the tiktoken library. Ensure they are installed:
```
python3 --version
pip3 list | grep tiktoken
```

If you're still experiencing issues after trying these steps, please open an issue in this repository with the following information:

Contributions are welcome! Please feel free to submit a Pull Request.

This project is licensed under the MIT License.