This is an official implementation of the model described in:
"Structural Language Models of Code" [PDF]
To appear in ICML'2020.
An online demo is available at https://AnyCodeGen.org.
This repository currently contains the dataset and the data extractor that we used to create the Java dataset in the paper. The TensorFlow code will be released soon.
Feel free to open a new issue for any question. We always respond quickly.
python3 -c 'import tensorflow as tf; print(tf.__version__)'
This dataset contains ~1.3M examples (1.1GB).
mkdir data
cd data
wget https://codegen-slm.s3.us-east-2.amazonaws.com/data/java-small-preprocessed.tar.gz
tar -xvzf java-small-preprocessed.tar.gz
This will create a data/java-small/
sub-directory, containing the files that hold training, test and validation sets,
a dict file for various dataset properties and histograms, and a grammar file that is used during beam search to
distinguish between terminal and non-terminal nodes.
To create and preprocess a new dataset (for example, to compare SLM to a new model on another dataset):
- Edit the file preprocess.sh using the instructions there, pointing it to the correct training, validation and test directories.
- Run the preprocess.sh file:
bash preprocess.sh
To download the Java-small as raw *.java
files, use:
To download the preprocessed dataset, use:
To download the dataset in a tokenized format that can be used in seq2seq models (for example, with OpenNMT-py), use:
The following JSON files are the files that are created by the JavaExtractor. The preprocessed and the seq2seq files are created from these JSON files:
Every line is a JSON object
that contains the following fields: num_targets
, num_nodes
, targets
,
is_token
, target_child_id
, internal_paths
, relative_paths
, head_paths
,
head_root_path
, head_child_id
, linearized_tree
, filepath
, left_context
,
right_context
, target_seq
, line
.
The C# dataset that we used in the paper was created using the raw (*.cs
files) dataset of
Allamanis et al., 2018,
(https://aka.ms/iclr18-prog-graphs-dataset) and can be found here: https://aka.ms/iclr18-prog-graphs-dataset.
To extract examples from the C# files, we modified the data extraction code of Brockschmidt et al., 2019: https://github.com/microsoft/graph-based-code-modelling/.
Structural Language Models of Code
@article{alon2019structural,
title={Structural Language Models of Code},
author={Alon, Uri and Sadaka, Roy and Levy, Omer and Yahav, Eran},
journal={arXiv preprint arXiv:1910.00577},
year={2019}
}