The goal of the project is to apply neural architecture search (NAS) to deep neural networks (DNNs) specified for tabular data. We take a multi-layer perceptron [1] and apply it to California Housing [2], Jannis [3] and Covertype [4] datasets. To select an optimal architecture, we use DARTS [5]. The project's report is here.
We have three jupyter notebooks in the project's root: two for classification tasks (Covertype\Jannis) and one for regression (California housing) task. Before running the code from notebooks, you should install the needed dependencies from the root of the project:
pip install -r requirements.txtDataset and evaluators folders contain utilities to preprocess datasets and run NAS experiments. We use the same preprocessing techniques as in the original paper on tabular deep neural networks [1]. As a reference we've also used these tutorials:
- Revisiting deep learning models for tabular data
- DARTS tutorial from nni library
- NAS tutorial from nni library
Models folder contains sources for MLP model and selected search space for NAS.
[1] Gorishniy, Y., Rubachev, I., Khrulkov, V., & Babenko, A. (2021). Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34, 18932-18943.
[2] R. Kelley Pace and R. Barry. Sparse spatial autoregressions. Statistics & Probability Letters, 33(3): 291–297, 1997.
[3] I. Guyon, L. Sun-Hosoya, M. Boullé, H. J. Escalante, S. Escalera, Z. Liu, D. Jajetic, B. Ray, M. Saeed, M. Sebag, A. Statnikov, W. Tu, and E. Viegas. Analysis of the automl challenge series 2015-2018. In AutoML, Springer series on Challenges in Machine Learning, 2019.
[4] J. A. Blackard and D. J. Dean. Comparative accuracies of artificial neural networks and discriminant analysis in predicting forest cover types from cartographic variables. Computers and Electronics in Agriculture, 24(3):131–151, 2000.
[5] Liu, H., Simonyan, K., & Yang, Y. (2018). Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055.