Binary Language Detection - ML Assessment, Exercise 1
This repository contains an implementation of a Binary Language Detection problem aimed at classifying sentences as either Italian or non-Italian. The proposed model is implemented in a Jupyter notebook that can be run locally or through Google Colaboratory. By downloading the dataset and running the cells, it is possible to train a new model and storing it in a pickle file, as well. The notebook includes data preprocessing, model selection and training, and performance evaluation. The trained model is selected with a GridSearchCV algorithm that enables the search for the best estimator for our task. The resulting model achieved high performance on the test set, with high accuracy and strong precision, recall, and F1 score.
You can directly try to detect the language of a sentence through a RESTful API, which allows to make a POST inference call. The final solution exposes the call to a service predict on port localhost:5000/predict via a web interface or curl command.
To further enhance the usability of this project, a dockerfile and a Docker image link on Docker Hub have been added to the repository. The Docker image allows users to easily deploy the language detection model without having to worry about setting up the necessary environment.
The final solution has been deployed as local endpoint to expose a REST API (POST) inference call to a service predict at http://localhost:5000/predict.
If you want to use the docker image, you can either build it from scratch given the provided dockerfile or you can directly use the one provided at this link. To build the image:
docker build -t binarylanguagedetection .docker run -dp 5000:5000 -ti --name LanguageDetectionContainer binarylanguagedetection
Now you can test the model going to Docker Desktop>Containers>Ports and clicking on port 5000.
In case you want to use pull the image from Docker Hub:
docker pull gioiamancini/binarylanguagedetection and run it with: docker run -dp 5000:5000 -ti gioiamancini/binarylanguagedetection
You can make the request by:
- Using a
curlvia the terminal:
curl -X "POST" "http://127.0.0.1:5000/predict" -H "accept: application/json" -H "Content-Type: application/json" -d "{\"text\": \"Is this an italian sentence?\", \"api_key\": \"apiKey1\"}"
- Using a
curlrequest through acurl_request.jsonfile:
curl -X "POST" "http://127.0.0.1:5000/predict" -H "accept: application/json" -H "Content-Type: application/json" -d @curl_request.json
If you want to build and test the REST API in your local environment, you can follow these instructions.
- Clone this repository to your local machine using:
git clone https://github.com/GioiaMancini/BinaryLanguageDetection.git - Navigate to the root directory of the cloned repository using the command line interface.
- Install the required dependencies by running the following command:
pip install -r requirements.txt - Start the server:
- you can run it through python command:
python main.py - or by using Uvicorn:
uvicorn main:app --port 5000Note that you can run it in a different port by:uvicorn main:app --port <int>, specifying the port number.
- The server should start running and you should see the following message:
* Running on http://localhost:5000/ (Press CTRL+C to quit) - Now, you can open a web browser and navigate to
http://localhost:5000to use the API via a web page, you should see a window like this:
- You can use a
curlto make requests via the terminal:
curl -X "POST" "http://127.0.0.1:5000/predict" -H "accept: application/json" -H "Content-Type: application/json" -d "{\"text\": \"Is this an italian sentence?\", \"api_key\": \"apiKey1\"}"
or
curl -X "POST" "http://127.0.0.1:5000/predict" -H "accept: application/json" -H "Content-Type: application/json" -d @curl_request.json
N.B.: If you want to test sentences with accented characters, please use a json file as the one provided in the main folder (curl_request.json).
N.B.: If you encounter any issue in running the server in your local machine (this solution has been developed and tested in Windows, so there could be conflicts in installed dependencies due to different OS environments), please refer to the previous instructions and use the docker image provided at [this link]](https://hub.docker.com/r/gioiamancini/binarylanguagedetection).
- You must have Python 3 installed on your system.
- You must have a Kaggle account.
-
Install the Kaggle API by running this command in your terminal:
pip install kaggle -
Log in to your Kaggle account by following these steps:
- Go to your Kaggle account and click on "Create New API Token". This will donwload a JSON file with your API credentials.
- Note that if you are using the Kaggle CLI tool, place this file in the location ~/.kaggle/kaggle.json on Linux, OSX, and other UNIX-based operating systems, and at C:\Users<Windows-username>.kaggle\kaggle.json on Windows. If you are using the Kaggle API directly, where you keep the token doesn’t matter, so long as you are able to provide your credentials at runtime.
-
Download the dataset using this command:
kaggle datasets download -d basilb2s/language-detection -
Unzip the .csv file.
- You can download the dataset to any directory you want by changing the path in step 3.
- If you don't have a Kaggle account, you can download the dataset from the Kaggle website directly by first creating an account, then skip 2. and 3 and unzip the downloaded file.
- You must have a Kaggle account.
- You must mount your drive in Colab.
- The instructions are also described in the colab notebook.
- Install Kaggle API
!pip install kaggle - Create and download your API token as shown in step 2. of previous section. Note that in this case the kaggle.json file must be in the root of your Google Drive.
- Mount your drive:
from google.colab import drive
drive.mount('/content/drive')
- Move your kaggle.json file in this directory:
!mkdir -p /root/.kaggle
!cp '/content/drive/My Drive/kaggle.json' /root/.kaggle/
-
Go in the directory in which you want to download the dataset, for example:
%cd /content/drive/MyDrive/Datasets -
Finally, you can download the dataset and unzip it:
!kaggle datasets download -d basilb2s/language-detection
!unzip language-detection.zip
Once you have downloaded the dataset and you have the right set-up, you can start exploring the notebook (colab link), involving the text preprocessing stage, the model selection and the final model evaluation.
As you can see from the notebook, at first you have to import the required dependencies, which involve preprocessing and machine learning models' libraries. Then, you have to load the csv file of the dataset into a pandas DataFrame. Some functions for data visualization are described, here are some examples of the DataFrame visualization:
| Text | Language | |
|---|---|---|
| 0 | Nature, in the broadest sense, is the natural, physical, material world or universe. | English |
| 1 | "Nature" can refer to the phenomena of the physical world, and also to life in general. | English |
| 2 | The study of nature is a large, if not the only, part of science. | English |
| 3 | Although humans are part of nature, human activity is often understood as a separate category from other natural phenomena. | English |
| 4 | [1] The word nature is borrowed from the Old French nature and is derived from the Latin word natura, or "essential qualities, innate disposition", and in ancient times, literally meant "birth". | English |
Here you can see the number of sentences per language in the dataset:
Here is the number of words per language in the dataset:
After visualizing information about the dataset, we have to perform data cleaning on the text data. In particular, we perform regex-based text cleaning operations, including removing special characters, symbols, numbers, URLs, html tags and extra large spaces. Then, the text is transformed to lowercase and finally a list of cleaned text is returned. After this process, the total number of words in the dataset goes from 202335 to 201905. Note that in presence of a more complex problem or more resources available, a more sophisticated data cleaning and embedding procedure should have been taken into account.
To perform our ML algorithms over our text data, we have to split the dataset into train and test sets. We define a random_state=23 to allow reproducibility.
The final data are split into 8269 sentences for train set and 2068 for test set.
In order to select the most suitable ML algorithm for our task, we perform a grid search for hyperparameters tuning among different algorithms using 5-Fold Cross-Validation with scoring F1. The resulting best model is the Multinomial Naive Bayes classifier. For technical details please refer to the ML_Exercise1.ipyng notebook.
This is the resulting confusion matrix and classification report obtained with the returned best model:
| precision | recall | f1-score | support | |
|---|---|---|---|---|
| 0 | 0.999 | 0.999 | 0.999 | 1925 |
| 1 | 0.986 | 0.993 | 0.990 | 143 |
| accuracy | 0.999 | 2068 | ||
| macro avg | 0.993 | 0.996 | 0.994 | 2068 |
| weighted avg | 0.999 | 0.999 | 0.999 | 2068 |
The obtained results confirms that MultinomialNB is the best classifier for our task, considering the performed hyperparameters grid search.
We have seen how the chosen model is a good strategy for solving this binary classification task. However, in presence of a more complex task or a larger amount of data, we could choose a different approach. For instance, if a larger dataset is available, together with more computational resources, the development of a more advanced model can be taken into account. In particular, a state-of-the-art Transformer-based architecture could be implemented. Moreover, a more sophisticated data cleaning and embedding procedures could be employed, as well as adding more features to help improving predictions or considering the chance of removing stop words. Also, since the data is imbalanced, SMOTE (Synthetic Minority Over-sampling Technique) or similar techniques could be tried to make the data more balanced. Considerations about memory, size and training/inference time efficiency could also make the result even more robust.
Finally, considering the REST API architecture, several additional improvements could be considered. For instance, a more robust authentication process should be used, together with a json response involving also information about status and response codes to make the user aware of specific issues. Moreover, the possibility of predicting among a list of sentences or an uploaded text file should be implemented.