RWebData: A High-Level Interface to the Programmable Web

Ulrich Matter

doi:10.5334/jors.201

RWebData: A High-Level Interface to the Programmable Web

Journal of Open Research Software

Volume 6 (2018): Issue 1

By: Ulrich Matter

Open Access

|Feb 2018

Abstract

The rise of the programmable web offers new opportunities for the empirically driven sciences. The access to, compilation and preparation of data from the programmable web for statistical analysis can, however, involve substantial up-front costs for the practical researcher. The R-package RWebData provides a high-level framework that allows data to be easily collected from the programmable web in a format that can be used directly for statistical analysis in R without bothering about the data’s initial format and nesting structure. It was developed specifically for users who have no experience with web technologies and merely use R as statistics software. This paper provides an overview of the high-level functions, explains the basic architecture of the package, illustrates the implemented data mapping algorithm, and discusses RWebData’s further development and reuse potential.

Funding statement: The author acknowledges financial support from the University of Basel Research Fund as well as support from the Swiss National Science Foundation (grant 168848).

References

Couture-Beil, A 2013 rjson: JSON for R. R package version 0.2.13. URL: http://CRAN.R-project.org/package=rjson.
Search in Google Scholar Back to article
Crosas, M, Honaker, J, King, G and Sweeney, L 2015 ANNALS of the American Academy of Political and Social Science, 659(1): 260–273. DOI: 10.1177/0002716215570847
Open DOI Search in Google Scholar Back to article
Csardi, G and Nepusz, T 2006 InterJournal Complex Systems, 1695. URL: http://igraph.org.
Search in Google Scholar Back to article
Dowle, M and Srinivasan, A 2017 data.table: Extension of ‘data.frame’. R package version 1.10.4. URL: https://CRAN.R-project.org/package=data.table.
Search in Google Scholar Back to article
Helbing, D and Pournaras, E 2015 Nature, 527(7576): 33–34. DOI: 10.1038/527033a
Open DOI Search in Google Scholar Back to article
Matter, U and Stutzer, A 2016 Does Public Attention Reduce the Influence of Special Interest Groups? Policy Positions on SOPA/PIPA Before and After The Internet Blackout Berkman Klein Center Research Publication 2016–22 Berkman Klein Center for Internet & Society at Harvard University. DOI: 10.2139/ssrn.2884316
Open DOI Search in Google Scholar Back to article
Men-hin, Y and Fu, A W c 2001 In: ‘Proceedings of the 81h International Workshop on Knowledge Representation Meets Databases (KRDB 2001), Rome, Italy’.
Search in Google Scholar Back to article
Müller, K and Wickham, H 2017 tibble: Simple Data Frames. R package version 1.3.4. URL: https://CRAN.R-project.org/package=tibble.
Search in Google Scholar Back to article
Moh, C H, Lim, E P and Ng, W K 2000 In: ‘Proceedings of the fth ACM Conference on Digital libraries’, 67–76. ACM.
Search in Google Scholar Back to article
Munzert, S, Rubba, C, Meissner, P and Nyhuis, D 2014 Automated Data Collection with R: A Practical Guide to Web Scraping and Text Mining John Wiley & Sons Chichester, UK. DOI: 10.1002/9781118834732
Open DOI Search in Google Scholar Back to article
Nolan, D and Temple Lang, D 2014 XML and Web Technologies for Data Sciences with R UseR! Springer: New York. DOI: 10.1007/978-1-4614-7900-0
Open DOI Search in Google Scholar Back to article
Ooms, J 2014 ArXiv e-prints. URL: https://arxiv.org/abs/1403.2805.
Search in Google Scholar Back to article
Ooms, J 2017 curl: A Modern and Flexible Web Client for R. R package version 3.0. URL: https://CRAN.R-project.org/package=curl.
Search in Google Scholar Back to article
Ooms, J, Temple Lang, D and Wallace, J 2014 jsonlite: A smarter JSON encoder/decoder for R. R package version 0.9.8. URL: http://CRAN.R-project.org/package=jsonlite.
Search in Google Scholar Back to article
ProgrammableWeb 2014 ‘Programmableweb research center: Growth in web apis from 2005 to 2013’. URL: www.programmableweb.com/api-research.
Search in Google Scholar Back to article
R Core Team 2013 R: A Language and Environment for Statistical Computing R Foundation for Statistical Computing Vienna, Austria. URL: http://www.R-project.org/.
Search in Google Scholar Back to article
R Core Team 2014 foreign: Read Data Stored by Minitab, S, SAS, SPSS, Stata, Systat, Weka, dBase, … R package version 0.8-59. URL: http://CRAN.R-project.org/package=foreign.
Search in Google Scholar Back to article
Reingold, E M and Tilford, J S 1981 IEEE Transactions on Software Engineering, 7(2): 223–228. DOI: 10.1109/TSE.1981.234519
Open DOI Search in Google Scholar Back to article
Richardson, L and Amundsen, M 2013 RESTful Web APIs O’Reilly Media Cambridge.
Search in Google Scholar Back to article
Ripley, B and Lapsley, M 2013 RODBC: ODBC Database Access. R package version 1.3-0. URL: http://CRAN.R-project.org/package=RODBC.
Search in Google Scholar Back to article
Sievert, C 2014 XML2R: EasieR XML data collection. R package version 0.0.6. URL: http://CRAN.R-project.org/package=XML2R.
Search in Google Scholar Back to article
Stephens, J 2014 yaml: Methods to Convert R Data to YAML and Back. R package version 2.1.11. URL: http://CRAN.R-project.org/package=yaml.
Search in Google Scholar Back to article
Swartz, A 2013 In: Hendler, J and Ding, Y (eds.), ‘Synthesis Lectures on The Semantic Web: Theory and Technology’. Morgan & Claypool Publishers.
Search in Google Scholar Back to article
Temple Lang, D 2013a RCurl: General network (HTTP/FTP/…) client interface for R. R package version 1.95-4.1. URL: http://CRAN.R-project.org/package=RCurl.
Search in Google Scholar Back to article
Temple Lang, D 2013b RJSONIO: Serialize R objects to JSON, JavaScript Object Notation. R package version 1.0-3. URL: http://CRAN.R-project.org/package=RJSONIO.
Search in Google Scholar Back to article
Temple Lang, D 2013c XML: Tools for Parsing and Generating XML Within R and S-Plus. R package version 3.95-0.2. URL: http://CRAN.R-project.org/package=XML.
Search in Google Scholar Back to article
Turner, V, Gantz, J F, Reinsel, D and Minton, S 2014 The digital universe of opportunities: Rich data and the increasing value of the internet of things White paper IDC Framingham, MA.
Search in Google Scholar Back to article
W3C 2013 ‘W3c semantic web activity: What is the semantic web?’ URL: www.w3.org/2001/sw/.
Search in Google Scholar Back to article
Warnes, G R, Bolker, B, Gorjanc, G, Grothendieck, G, Korosec, A, Lumley, T, MacQueen, D, Magnusson, A and Rogers, J and others 2014 gdata: Various R Programming Tools for Data Manipulation. R package version 2.13.3. URL: http://CRAN.R-project.org/package=gdata.
Search in Google Scholar Back to article
Wickham, H 2014a httr: Tools for Working with URLs and HTTP. R package version 0.4. URL: http://CRAN.R-project.org/package=httr.
Search in Google Scholar Back to article
Wickham, H 2014b Journal of Statistical Software, 59(10): 1–23. URL: https://www.jstatsoft.org/index.php/jss/article/view/v059i10/v59i10.pdf. DOI: 10.18637/jss.v059.i10
Open DOI Search in Google Scholar Back to article
Wickham, H 2015 rvest: Easily Harvest (Scrape)Web Pages. R package version 0.2.0. URL: http://CRAN.R-project.org/package=rvest.
Search in Google Scholar Back to article
Wickham, H and Francois, R 2015 readr: Read Tabular Data. R package version 0.1.0. URL: http://CRAN.R-project.org/package=readr.
Search in Google Scholar Back to article