Scaling R & Data Science Teams

Modern data science demands scaling computation beyond single machines and coordinating analysis workflows across multidisciplinary teams.

1. Scaling the Split-Apply-Combine Paradigm

The split-apply-combine philosophy for data tables scales up to much larger problems. Sometimes the tables themselves grow too big for one computer (or CPU); other times the tables may be reasonably sized, but the tasks are numerous. This split-apply-combine at scale has been called map-reduce, hadoop and high throughput computing. The idea is to take a big task; split it into many (1000s or millions) small tasks; send each of those to a separate CPU; and combine the results back together (reduce) once all are completed. These projects typically take 1000s of compute hours, and in some cases millions.

This works best when tasks are “parallel” – can be done independently of each other, such as separate simulations from the same model system. Unfortunately, some loosely coupled tasks depend on each other and require more complicated arrangements, perhaps even needing high performance computing such as graphical CPUs, or GPUs.

High Throughput & High Performance Computing

2. Statistical Computing Beyond R

R is one of many languages and other electronic tools of value to staticians.

General References

3. Team Science Workflows & Curriculum Organization

The project is for individuals in teams who are using, or planning to use, R to address problems in the data sciences. This assumes a first course in Data Carpentry or some similar introduction to working with data using computer tools. However, for those who have not had that, links to appropriate material will be provided along the way.

This is not fully laid out as a course, although it could be used as such. Much of the material has been tried out in various courses and training venues. There are many references out to course and research material of others so that teams can build on their experience. That, in fact, is how any science progresses. A web version is at http://www.stat.wisc.edu/~yandell/R_for_data_sciences. This material is being used in Fall 2017 for the COMBEE R Study Group at UW-Madison.

The syllabus provides the broad organization. Material is organized into verb folders roughly based on this syllabus.

  • curate data structures
  • visualize data relationships
  • organize data methods and documentation
  • analyze data with linear and other models
  • profile code for efficiency and error checking
  • connect with other data science tools beyond R

Other material in this directory

item contents
overview Overview of R Material
reference list of references
repositories list of data repositories
learnR R Language Tips & Tricks
history background material on R and Douglas Bates

Many people contributed to this material. Thanks to Douglas Bates for his coteaching and help in Fall 2014 and 2015. The COMBEE R Study Group kindly shared their study material, and are allowing me to use the R Study Group in Fall 2017 to test out this.

4. Professional Skills for Data Science

These materials are for statisticians at all levels who want to learn more about modern network and computing tools for statistics. The course for Fall 2015 is: Stat 627 Professional Skills for Data Science (1-2 cr, Thursdays 1-2:15pm, 1010 MSC). [This is NOT the Stat 327 “Data Analysis with R” course.] This is recommended for all new graduate students. It will emphasize training in best practices with R, focusing on how to organize your work (using Rstudio), how to build excellent graphics (using ggplot2) and how to document your work (using R markdown), among other things. This is being led by Brian Yandell and Doug Bates in Fall 2015; see moodle site ⚠️, Stat 627 Syllabus and Jenny Bryan’s Stat 545 at UBC ⚠️. The course covers

  • installation of the RStudio application on personal computers and use of RStudio on departmental computers.
  • Representation of data in R, including factors and data frames.
  • Manipulation and display of data in R.
  • Visualization of data with the ggplot2 package for R.
  • The “R Markdown” language for literate programming and reproducible research.
  • The R formula language and its use for fitting linear models and generalized linear models.
  • Contrast specifications for factors and their impact on interpretation of coefficient estimates.

The raw material for Fall 2014 was at https://github.com/dmbates/stat692 with more material in Yandell’s Stat 692 Notes related to Bates’s talks.

Much of the material below was added for Fall 2013. We used the following network tools to deliver information: drupal (open access web pages–this page); box (drop box for collaboration); and moodle ⚠️ (course collaboration environment).

We are in the age of big data, when it is not enough to think of what you can do with the computer on your desk or lap. Further, statistics as a field is being transformed by analytics, the process of discovery and communication of meaningful patterns in data. Today, statisticians of all flavors–from the most pragmatic to the most theoretical–need a variety of computational tools, and we need access to vast resources across computer networks. While many of us work in closed shops, behind proprietary walls, the world of open source is core to sharing methods and information. Thus, it is important to understand the power and role of the linux operating system and of the R language. However, these only the beginning. While we focus on computational skills, communication is key to the field of statistics, and to science in general. And visualization is at the core of communicating complex ideas, providing a window of insight into the world of data. Today, much communication is online, and we must learn how to leverage online data and network tools to advance in the field, and to do our work effectively.

This site uses Just the Docs, a documentation theme for Jekyll.