This repository contains a runtime analysis of different similarity measures for documents larger than 50 words. The analysis includes the runtime comparison of Jaccard similarity and Cosine similarity measures using different representations and techniques. The runtime analysis is performed on a local machine with the following specifications: MacBook Pro, 2.4 GHz 8-Core Intel Core i9, 32 GB 2667 MHz DDR4.
The introduction section provides an overview of the similarity analysis techniques used, including Jaccard similarity and Cosine similarity. It explains their definitions, applications, and importance in various fields such as data mining, machine learning, and natural language processing.
The analysis is divided into several sections, each focusing on a specific aspect of similarity analysis for documents larger than 50 words. The sections covered in this repository are as follows:
- Analysis of the runtime of Jaccard similarity measure for documents represented as bags of words.
- Analysis of the runtime of Cosine similarity measure for documents represented as dense vectors without the use of np.dot().
- Analysis of the runtime of Cosine similarity measure for documents represented as dense vectors with the use of np.dot().
- Analysis of the runtime of Cosine similarity measure for documents represented as bags of words.
- Analysis of the runtime of all-pairs comparisons for more than 10 documents using either Jaccard similarity, Cosine similarity without np.dot(), or Cosine similarity with np.dot().
- Analysis of the runtime of all-pairs comparisons for more than 10 documents using a multiprocessing paradigm (MapReduce).
The results of each analysis section are presented in a clear and concise manner. Each section includes details about the experimental setup, the runtime measurements, and any insights or observations derived from the analysis. The runtime results are compared and discussed to understand the efficiency and effectiveness of different similarity measures.
To replicate the analysis or use the provided code, follow the instructions below:
- Clone the repository:
git clone https://github.com/your-username/repository-name.git - Install the required dependencies mentioned in the project's documentation.
- Run the specific analysis section of interest by executing the corresponding code files.
- Analyze the runtime results and compare the performance of different similarity measures.
- Modify or extend the code as needed for your specific use case or experiment.
The content of this repository is licensed under the MIT License.