Download Instructions for the WDC RDFa, Microdata, and Microformats Data Sets (December 2014)

This document contains instructions on how to download the December 2014 version of the Web Data Commons RDFa, Microdata, and Microformats data sets.

Download the Extracted Data

The extracted RDFa, Microdata and Microformat data is provided for download as N-Quads. Files are compressed using GZIP and each file is around 100 MB large. Overall 3,581 files with a total size of 373 GB are provided.

List of download URLs for RDF from the Winter 2014 corpus (Example Content)

The extracted RDF data can be downloaded using wget with the command wget -i http://webdatacommons.org/structureddata/2014-12/files/file.list. The files, containing quads for a specific formats can be found in the table below, together with more detailed statistics about the number of files and sizes.

Format Number Of Files Total File Size File List
html-rdfa 638 65 GB html-rdfa.list
html-microdata 2301 242 GB html-microdata.list
html-mf-geo 5 454 MB html-mf-geo.list
html-mf-hcalendar 24 2 GB html-mf-hcalendar.list
html-mf-hcard 504 51 GB html-mf-hcard.list
html-mf-hrecipe 7 688 MB html-mf-hrecipe.list
html-mf-hlisting 3 281 MB html-mf-hlisting.list
html-mf-hresume 1 6 MB html-mf-hresume.list
html-mf-hreview 26 2 GB html-mf-hreview.list
html-mf-species 1 9 MB html-mf-species.list
html-mf-xfn 23 2 GB html-mf-xfn.list

Get the Code

The source code can be checked out from our Github repository. For more information about the framework and a detailed description how to run a own extraction visit the framework page.
The code for the analysis of the quads can be checked out from the StructuredDataProfiler Github repository.

Get Support

Please send questions and feedback to the Web Data Commons mailing list or post them in our Web Data Commons Google Group.