CS598: Human-in-the-loop Data Management

The course explores two complementary roles for humans as applied to interactive data analytics: one, where humans are the analysts performing or supervising the analysis; here, the emphasis is on building usable tools for these analysts, and second, where humans are the crowdsourced workers assisting with the computation and analysis; here, the emphasis is on having humans process as little data as possible while gaining maximum benefit.

Students will read a number of papers: both important landmark papers as well as cutting-edge papers, act as a discussant for a paper at least once, and complete a semester-long implementation project. Familiarity with basic databases, machine learning, and algorithms expected.


  • August 25: The final list of papers and schedule is up. Following the instructions given below, by midnight September 1: you need to send us the list of 5 papers that you'd like to present.
  • August 25: There will be no classes on 8/31 and 9/02 as Aditya will be attending the VLDB conference

Course Significance

Crowd-Powered Analytics: An IBM study estimated that 80% of the data recorded every day is unstructured: i.e., it consists of images, videos and text. Fully automated processing of unstructured data is not yet a solved problem. Humans, on the other hand, are very good at understanding, interpreting, and processing unstructured data. How do we use humans to effectively process large volumes of unstructured data?

Interactive Analytics: A McKinsey Big Data Study estimated that 10s of Millions of new data analysts will be needed by 2015. With so many novice data analysts interacting with data, how do we enable them to quickly get valuable insights? Quickly could mean generating the same results faster, but approximately; it could mean showing them visualizations instead of raw data; it could mean helping the users to ``guess'' the query or insight in mind.

Grading Policy

  • Class Reviews: 20%
    • Due day before class at midnight. Starts on 8th Sep
    • Lightly graded; allowed to miss 3 (three) in total. Late submissions count towards the 3
    • After 3 misses, you lose 1% of grade per missed review
  • Class Participation: 15%
  • Paper Presentation: 15%
    • Send us top 5 papers you’d like to present on 1st Sep
    • The presentation slides should be mailed to Tarique 48 hrs in advance of your presentation
  • Research Project: 50%
    • Proposal (23rd Sep) + report + presentation

Instructions for Submitting Class Reviews

You must use the following link to submit class reviews: Link.

Remember to cover the 5 key questions: what is the problem, why is it important, what sets it apart from previous work, what are the key technical ideas, what are the key flaws and open issues, all within 500 words.

The class reviews must be submitted by midnight the day before class.

Instructions for Submitting List of Papers to Present

You must use the following link to submit your list of top-5 papers: Link.

The papers you provide can be from the list given below. You are also free to list a paper of your choice, but it must match the themes of the class. This list must be submitted by midnight September 1. .


Date Paper Presenter Notes
8/24/2015 Introduction to Course Content Aditya
8/26/2015 CrowdScreen: Algorithms for Filtering Data Using Humans Aditya
8/31/2015 VLDB Conference--No Lecture
9/2/2015 VLDB Conference--No Lecture Send ranked list of 5 papers by midnight September 1
9/7/2015 Holiday- Labor Day
9/9/2015 So Who Won: Dynamic Max Discovery with the Crowd Vipul First time for Class Review- Send it by midnight the day before; Student Presentations Start
9/14/2015 Crowdsourced Enumeration Queries Ruihan
9/16/2015 Human-Powered Sorts and Joins Aditya
9/21/2015 Turkit: Human Computation Algorithms on Mechanical Turk Alex Morales
9/23/2015 CrowdDB: Answering Queries Using Crowdsourcing Aditya
9/28/2015 Deco: Declarative Crowdsourcing Andrew Submit project proposal by midnight the day before. Proposals won't be graded, but are compulsory
9/30/2015 Potter’s Wheel: An Interactive Data Cleaning System Donald
10/5/2015 Wrangler: interactive visual specification of data transformation scripts Tifany
10/7/2015 Profiler: Integrated Statistical Analysis and Visualization for Data Quality Assessment ; Mid-term review for projects Yulun
10/12/2015 Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing; Map-Reduce basics Suhansanu
10/14/2015 Dremel: Interactive Analysis Of Web-Scale Datasets; Column Stores basics Alexander Zahdeh
10/19/2015 Spark SQL: Relational Data Processing in Spark; Shark basics Doris
10/21/2015 BlinkDB: Queries with Bounded Errors and Bounded Response Times on Very Large Data Liqi
10/26/2015 Trust Me, I’m Partially Right: Incremental Visualization Lets Analysts Explore Large Datasets Faster Shengliang
10/28/2015 Polaris: A System for Query, Analysis, and Visualization of Multidimensional Relational Databases Aditya
11/2/2015 SeeDB: Automatically Recommending Query Visualizations Aditya
11/4/2015 GraphLab: A New Framework For Parallel Machine Learning Stephen
11/9/2015 MLbase: A Distributed Machine-learning System Henry
11/11/2015 MAD Skills: New Analysis Practices for Big Data Ritwika
11/16/2015 dbTouch: Analytics at your Fingertips Yishuo
11/18/2015 Gestural Query Specification Himel
12/30/2015 Making Database Systems Usable Aditya
12/2/2015 Google fusion tables: web-centered data management and collaboration Donald Cha Final Project Report Guidelines
12/7/2015 Project Presentations in class
Submit project report the same night by 11.59 PM

Tentative List of Papers

Theme 1: Dealing with Unstructured and Noisy Data

Crowd-Powered Systems

Crowd-Powered Algorithms

Data Cleaning Systems

Theme 2: Dealing with Huge Data

Scalable Data Processing Systems

Approximate Analytics

Scalable Visualizations

Theme 3: Dealing with Novice Analysts

Visual Analytics Systems

New Interfaces and Usability

Autocompletion and Query Refinement

Theme 4: Dealing with New Data Analytics Scenarios

Machine Learning and Graph Processing

Collaborative Query Processing

Implementation Project

As part of this course, you need to complete a semester-long project. See the instructor for ideas. Alternatively, you are free to look for ideas in your domain of expertise: for instance, if you work in computational journalism, building a new way to browse and manage large collections of textual archives could be a perfectly reasonable project. Either way, you must speak to the instructor to verify that the project is indeed "challenging" enough.