In this project we we have developed a Deep Learning model designed to predict the pagination links on a web page.
The model classifies each <a>
and <button>
elements in a web page into the following categories:
- PREV - previous page link
- PAGE - a link to a specific page
- NEXT - next page link
- OTHER - for elements that are not a pagination link
The model is based on the research paper: "Large Scale Web Data API Creation via Automatic Pagination Recognition - A Case Study on Event Extraction"
We introduce features such as URL feature from URLNet1 and sentence embedding from Sentence Transformer2 into our model to enhance the performance beyond previous methodology.
We utilized the same dataset as used in "Large Scale Web Data API Creation via Automatic Pagination Recognition - A Case Study on Event Extraction."
This dataset is an extension of the original dataset used in the Autopager3 contains 319 pages extracted from 109 distinct websites.
Intall the required packages with anaconda
conda env create -f environment.yml
Activate the environment
conda activate pagination-prediction-pytorch
./train.sh
Checkpoint files will be available at ckpt
directory after training, which can than be used by pagination_prediction_api.py
for inference purpose.
./test_api.sh
Footnotes
-
URLNet: Source - arXiv:1802.03162 ↩
-
SentenceTransformers: Source - arXiv:1908.10084 ↩