URL

Change Log

When	What
April 27th, 2015	Donated by Miltiadis Allamanis

Reference

Studies who have been using the data (in any form) are required to include the following reference:

@inproceedings{Allamanis:2014:MIS:2635868.2635901,
    author = {Allamanis, Miltiadis and Sutton, Charles},
    title = {Mining Idioms from Source Code},
    booktitle = {Proceedings of the 22Nd ACM SIGSOFT International Symposium on Foundations of Software Engineering},
    series = {FSE 2014},
    year = {2014},
    isbn = {978-1-4503-3056-5},
    location = {Hong Kong, China},
    pages = {472--483},
    numpages = {12},
    url = {http://doi.acm.org/10.1145/2635868.2635901},
    doi = {10.1145/2635868.2635901},
    acmid = {2635901},
    publisher = {ACM},
    address = {New York, NY, USA},
    keywords = {code idioms, naturalness of source code, syntactic code patterns},
}

Additionally, the following reference can be used to cite the Java GitHub corpus:

@inproceedings{githubCorpus2013,
  author={Allamanis, Miltiadis and Sutton Charles},
  title=,
  booktitle={The 10th Working Conference on Mining Software Repositories},
  year={2013},
  pages={207--216}
  organization={IEEE}
}

About the Data

Overview of Data

This dataset cosists of XML containing stack overflow questions and answers as of August 2012 that can be used for text mining. The stack overflow dataset contains code snippets used by Haggis, a system for mining code idioms developed by the researchers.

The GitHub Java Corpus is a collection of Java code at a large scale, while it has been filtered to have an above-average quality (see next section). It can be used to study coding practice in Java at a large scale. It contains all project code, tracked by git with all the associated files. The containing open-source projects are from multiple domains. The code is not ours but is open-source. Please respect the license of each project.

How the corpus was collected

We processed GitHub’s event stream, provided by the GitHub Archive and filtered individual projects that were forked at least once. For each of these projects we collected the URL, the project’s name and its language. Filtering by forks aims to create a quality corpus: Since we filter away the majority of projects that have a below-average “reputation”, we obtain code of better quality. This criterion is not clear-cut and we could have chosen another method (e.g. GitHub’s star system), but we found this preprocessing sufficient.

We then downloaded (clone in Git’s terms) the full repositories for all remaining Java projects and removed any duplicates: We found about 1,000 projects that shared common commit SHAs indicating that they were most likely forks of the same project (but not declared on GitHub) and we manually picked those projects that seemed to be the original source. Project duplicates need to be removed to avoid a “leak” of data from the test to the train projects and also to create a more representative corpus of repositories.

We split the repository 75%-25% based on the lines of code assigning projects into either the training or test set. The split can be found in the RepositoryState.tar.gz

Top projects in the corpus

We have also ordered the projects in the corpus according to popularity, defined as the number of forks plus the number of watchers, where each is first seprately normalized into a z-score. The ordered list can be downloaded as top_projects.tar.gz.

Corpus Statistics

Statistic	Train	Test
Number of Projects	10,968	3,817
LOC	264,225,189	88,087,507
Tokens	1,116,195,158	385,419,678

Paper Abstract

We present the first method for automatically mining code idioms from a corpus of previously written, idiomatic software projects. We take the view that a code idiom is a syntactic fragment that recurs across projects and has a single semantic purpose. Idioms may have metavariables, such as the body of a for loop. Modern IDEs commonly provide facilities for manually defining idioms and inserting them on demand, but this does not help programmers to write idiomatic code in languages or using libraries with which they are unfamiliar. We present Haggis, a system for mining code idioms that builds on recent advanced techniques from statistical natural language processing, namely, nonparametric Bayesian probabilistic tree substitution grammars. We apply Haggis to several of the most popular open source projects from GitHub. We present a wide range of evidence that the resulting idioms are semantically meaningful, demonstrating that they do indeed recur across software projects and that they occur more frequently in illustrative code examples collected from a Q&A site. Manual examination of the most common idioms indicate that they describe important program concepts, including object creation, exception handling, and resource management

Data Categories

haggis

URL