Last week, I was invited by Heather Chaplin of the Design + Journalism program at The New School to give a talk about our work at The New York Times R&D Lab. I gave an overview of a bunch of our recent projects and then took a deeper dive into some of the design values and research that form the conceptual foundations for those projects.
Over the course of the year, we at The New York Times R&D Lab continually share and discuss links to research, analysis, inventions and other items of interest. In 2014, we were reading about everything from sentient articles to bees as 3D printers. Here is a roundup of the year’s links, organized by topic.
*Inclusion in this list does not imply endorsement by the lab or The New York Times; this is simply a collection of signals and ideas we have been tracking.
FUTURES & FORECASTING (GENERAL)
CONNECTED OBJECTS & ENVIRONMENTS
Sub-themes: responsive cities, physical-world search, object/system transparency, security vulnerability in connected objects, smart homes, quantified populations, sentinel devices, contextual public spaces, planned obsolescence
Sub-themes: wearable authentication, activity/health tracking, emotion tracking, social wearables, personal drone companions
Sub-themes: 3D printing, printable circuits, nanopower, zero-resistance electricity, supercharging batteries
Sub-themes: Making the invisible visible, infrastructure design and socioeconomic power dynamics, infrastructure to support autonomous systems, localized solutions vs. generalized standards, block chains, future network strategies
NEWS & MEDIA
Sub-themes: beyond the CMS, transparency in data journalism, structured data, drone journalism, cross-platform news experiences, creating context, sentient articles
UAS / DRONES
SOCIETAL ADAPTATION TO NEW TECHNOLOGIES
Sub-themes: Robot tourism, data liability / litigation, exaptation
PRIVACY & SURVEILLANCE
Sub-themes: privacy as privilege, sub-internets with different privacy policies, crypto-phones, reputation and identity management / hacking, decaying permissions
OFFLINE NETWORKS / MESH NETWORKS / PERSONAL AREA NETWORKS
Sub-themes: Hertzian personal space, presence-based consensus, presence-based authentication, body as network node, controlling personal visibility to networks
DATA, ANALYTICS, AI
Sub-themes: algorithm bias / ethics, deep learning, impulse response modeling, post-mortem data markets, personal data transactions, engineered filter failure, real-time decision modeling
DESIGN & USER EXPERIENCE
Sub-themes: haptics, gestural UI, AR, trust as a design value, evolving reading / consumption interactions, designing for underrepresented audiences, haptic Hertzian visibility
BIOTECH & HUMAN AUGMENTATION
Sub-themes: body as network node, cyborgs, biological computation / networking, networked biology, body-powered gadgets, assistive tech → augmented tech, emotional augmentation, synthetic biology, organic-synthetic sensing
In our last blog post we introduced you to Madison, a crowdsourcing project where readers provide data on the historical ads within our archives. Madison required a system that could keep track of our tasks, users and candidate ads; combine the three to create assignments, based on which ads and users are eligible for that task; and validate the crowdsourced data we receive according to custom criteria.
Madison’s planning stages involved a review of the current landscape of crowdsourcing solutions. We looked at existing hosted and open source platforms and libraries, paying special attention to ProPublica’s Transcribable Rails plugin, Zooniverse’s Scribe and PyBossa. While these projects do an excellent job handling certain use cases, none of them could support what we envisioned for Madison. Further, we realized that the kind of workflow we were designing could be abstracted to support a wider variety of crowdsourcing requirements than just the assets within our archives. We were intrigued by the idea of designing a service that defined the pieces of a crowdsourcing process in a highly flexible, customizable and modular way, and how such a platform could expand the set of things available to crowdsource.
The system we built is Hive, an open-source platform that lets developers produce crowdsourcing applications for a variety of contexts. Informed by our work on Streamtools, Hive’s technical architecture takes advantage of Go’s efficiency in parsing and transmitting JSON along with its straightforward interface to Elasticsearch. Combining the speed of a compiled language with the flexibility of a search engine means Hive is able to handle a wide variety of user-submitted contributions on diverse sets of tasks.
The workflow looks like this:
As you can see, there is a loop forming the core of Hive: a particular task is repeatedly done on an asset until accurate data is received, at which point the asset (now with new information on it) can reenter the pipeline with a different, newly-available task. (For example, in Madison, an asset can only enter the Tag phase once it has been verified during the Find phase as having “exactly one ad.”) This frees the developer from having to define every possible combination of assets and tasks. It also means that more tasks can be added as the developers get a clearer view of the type of data they’re dealing with (e.g. if an asset is marked as “multiple ads,” we could, in the future, implement a Cut task that is solely available for those assets). This loop lets developers gather increasingly granular information about a particular asset and, therefore, ask better questions for their audience to answer.
Of course, no crowdsourcing platform could be successful without the crowd. Hive has an intentionally flexible definition of a user: you may require a signup and login process, or simply allow anonymous contributions to lower the barrier of entry. Hive keeps track of each user’s number of contributions by task, both as a total, and further broken down by how many were skipped, completed or verified. The platform also supports asset favoriting, a feature we use on Madison’s profile pages.
Hive’s technology was chosen with an eye towards supporting an iterative design process without sacrificing performance. Elasticsearch, “a swiss army knife” of a data layer, seemed an obvious choice: adjustable enough to store structured and schemaless documents, indexed almost instantly and retrievable through a RESTful API with all the full-text search power of Apache’s Lucene. Hive uses the elastigo library to manage all the data associated with crowdsourcing applications, from project configuration to user submissions and statistics, in Elasticsearch.
Hive speaks to the Lab’s overall goal of providing new methods for receiving, storing, and understanding the sources of data around us. The past two months of dedicated and active contributors on Madison show that Hive can not only support a wide variety of crowdsourcing tasks, but it can do so even under high traffic.
Hive is free and open source, and we look forward to seeing the diverse crowdsourcing projects it powers in the future.
Our recent work at the Labs has focused on semantic listening: systems that obtain meaning from the streams of data surrounding them. Chronicle and Curriculum are recent examples of tools designed to extract semantic information (from our corpus of news coverage and our group web browsing history, respectively). However, not every data source is suitable for algorithmic analysis–and, in fact, many times it is easier for humans to extract meaning from a stream. Our new projects, Madison and Hive, are explorations of how to best design crowdsourcing projects for gathering data on cultural artifacts, as well as provocations for the design of broader, more modular kinds of crowdsourcing tools.
Madison is a crowdsourcing project designed to engage the public with an under-viewed but rich portion of The New York Times’s archives: the historical ads neighboring the articles. News events and reporting give us one perspective on our past, but the advertisements running alongside these articles provide a different view, giving us a sense of the culture surrounding these events. Alternately fascinating, funny and poignant, they act as commentary on the technology, economics, gender relations and more of that time period. However, the digitization of our archives has primarily focused on news, leaving the ads with no metadata–making them very hard to find and impossible to search for them. Complicating the process further is that these ads often have complex layouts and elaborate typefaces, making them difficult to differentiate algorithmically from photographic content, and much more difficult to scan for text. This combination of fascinating cultural information with little structured data seemed like the perfect opportunity to explore how crowdsourcing could form a source of semantic signals.
There were endless data points we were interested in collecting, but the challenge was how to do so while keeping our audience engaged. A very complex task might garner us a wealth of data in theory–but not if no one, in reality, wanted to do it. Further, we could try to punch up an extremely dry task with external incentives for the most active contributors–but we risked having our system gamed by those who simply wanted the rewards, potentially setting ourselves up for a bank of incorrect data. In order to avoid these problems, we took an approach that centered around reducing friction as much as possible, and that limited gamifying elements in favor of highlighting the interesting parts of the task at hand.
We settled on a set of design principles to reduce friction as much as possible, inspired by previous cultural and science-oriented crowdsourcing projects (such as the Zooniverse projects, the NYPL’s Building Inspector and The Guardian’s MP’s Expenses):
- Add more tasks rather than making one task very complex. Keeping tasks clear and streamlined meant that the user didn’t have to constantly switch contexts, and could knock out a few assignments in a row without having to think too much about it.
- Make the tasks self-explanatory. Building off the first concept, a task whose question is obvious is much easier to answer than one that requires looking at and interpreting instructions.
- Design for a variety of use cases. If the tasks are simple and modular, chances are they can be done on a variety of devices in a variety of situations. Our Find task works nicely on mobile, and can be done in a few seconds while waiting for the bus; our Transcribe task is much more oriented to someone at home on a desktop computer, looking for a way to spend 10 minutes.
- Permit anonymous contributions. Asking users to sign up first thing would give them an excuse to leave the site. By letting users contribute without logging in, we allow them to try it out and get into it before they commit to creating an account.
We also purposefully chose an approach that downplayed gamification in order to place the the fun part of a potentially dry process at the forefront: discovering and sharing interesting cultural items and artifacts with your friends. In their paper on their Menus project, the NYPL points out that, for cultural institutions, “the incentives reside in the materials themselves and in the proposition of working in partnership with a public trust.” Rather than trying to tempt a user into participation with external, material rewards, we aimed to design a system whose biggest rewards came from engaging with it: namely, discovering and sharing a piece of culture that probably only a handful of people have seen since its original publication. This has a double benefit: because the rewards of Madison are largely in the delight of finding new things (not earning points, climbing a leaderboard, completing missions or getting badges), there is little incentive to game the system with bad information, in turn bolstering our confidence in opening up the project to anonymous users. Madison also has built-in validation criteria that require agreement from a number of contributors to ensure that the ads are annotated with correct metadata.
These choices formed the basis of Madison, and also shaped the platform underneath it: Hive. Hive is a modular, open-source framework for building crowdsourcing projects like Madison with any set of assets. We will be sharing more information about Hive in an upcoming blog post!
News publishing is an inherently ephemeral act. A big story will consume public attention for a day, or a month or a year only to fade from memory as quickly as it erupted. But news coverage, aggregated over time, can provide a fascinating “first draft of history” — a narrative of events as they occurred. At The New York Times, we have an incredibly rich resource in our 162-year archive of Times reporting, and one of the areas we occasionally explore in the lab is how to harness our archive to create new kinds of experiences or tools.
Two years ago, I created Chronicle, a tool for graphing the usage of words and phrases in New York Times reporting. Inspired by my own love of language and history, it’s a fascinating way to see historical events, political shifts, cultural trends or stylistic tropes. Chronicle can reveal things like the rise of feminism, evolution of cultural bêtes noires or when we shifted from talking about the “greenhouse effect” to talking about “climate change”. The Times’ corpus is particularly interesting as a reflection of culture because our style guide carefully informs how our reporters use language to describe the world, which allows us to see those changes more clearly than if we were looking at a heterogenous archive of text. More broadly, Chronicle acts as another example of “semantic listening” approaches we have been researching in the lab — methods for extracting useful semantic signals from streams as diverse as conversations, web browsing history, or in this case, a historic corpus of news coverage.
Since its creation, Chronicle has been in use internally as a research tool, and has occasionally made its way into our report, most notably in Margaret Sullivan’s election day look at hot-button issues in past presidential elections. While we have made a multitude of discoveries through using Chronicle within The Times, we want to see what our readers can unearth about our history as well. As of today, Chronicle is now open and available to the public. Go explore and tell us about your best finds by tweeting them to @nytlabs!
In the course of our work, we make a lot of small experiments, often in code. Sometimes we hit upon something that may not be a signal from the future, but is quite useful in the present. Vellum is one such project.
One of my primary uses for Twitter is to find interesting reading material: breaking news, long reads, research relevant to my work, or funny things about maps. However, Twitter’s interface treats commentary as primary and content as secondary, which can make it difficult to discover things to read if I’m mostly interested in that secondary content.
To address this use case, we created Vellum. Vellum acts as a reading list for your Twitter feed, finding all the links that are being shared by those you follow on Twitter and displaying them each with their full titles and descriptions. This flips the Twitter model, treating the links as primary and the commentary as secondary (you can still see all the tweets about each link, but they are less prominent). Vellum puts a spotlight on content, making it easy to find what you should read next.
We also wanted to include signals about what might be most important to read right now, so links are ranked by how often they have been shared by those you follow on Twitter, allowing you to stay informed about the news your friends and colleagues are discussing most.
Vellum was built as a quick experiment, but as we and other groups within The New York Times have been using it over the past few months, it has proven to be an invaluable tool for using Twitter as a content discovery interface. So today we are opening up Vellum to the public. We hope you find it as useful as we have. Happy reading!
Note: This was also published as a guest post for the Superflux blog.
Earlier this year, I saw a video from the Consumer Electronics Show in which Whirlpool gave a demonstration of their new line of connected appliances: appliances which would purportedly engage in tightly choreographed routines in order to respond easily and seamlessly to the consumer’s every need. As I watched, it struck me how similar the notions were to the “kitchen of the future” touted by Walter Cronkite in this 1967 video. I began to wonder: was that future vision from nearly fifty years ago particularly prescient? Or, perhaps, are we continuing to model technological innovation on a set of values that hasn’t changed in decades?
When we look closely at the implicit values embedded in the vast majority of new consumer technologies, they speak to a particular kind of relationship we are expected to have with computational systems, a relationship that harkens back to mid-20th century visions of robot servants. These relationships are defined by efficiency, optimization, and apparent magic. Products and systems are designed to relieve users of a variety of everyday “burdens” — problems that are often prioritized according to what technology can solve rather than their significance or impact. And those systems are then assumed to “just work”, in the famous words of Apple. They are black boxes in which the consumer should never feel the need to look under the hood, to see or examine a system’s process, because it should be smart enough to always anticipate your needs.
So what’s wrong with this vision? Why wouldn’t I want things doing work for me? Why would I care to understand more about a system’s process when it just makes the right decisions for me? [Read more...]
We see a moment coming when the collection of endless streams of data is commonplace. As this transition accelerates it is becoming increasingly apparent that our existing toolset for dealing with streams of data is lacking. Over the last 20 years we have invested heavily in tools that deal with tabulated data, from Excel, MySQL and MATLAB to Hadoop, R and Python+Numpy. These tools, when faced with a stream of never ending data, fall short and diminish our creative potential.
In response to this shortfall we have created streamtools – a new, open source project by The New York Times R&D Lab which provides a general purpose, graphical tool for dealing with streams of data. It provides a vocabulary of operations that can be connected together to create live data processing systems without the need for programming or complicated infrastructure. These systems are assembled using a visual interface that affords both immediate understanding and live manipulation of the system.
Noah, Matt and I just returned from Geneva, Switzerland, where we were attending Lift 2014. Lift is a conference on technology, design and innovation that we have followed from afar for a while now, as it is co-founded by Nicolas Nova, of the Near Future Laboratory (whose work we all greatly admire) and always has a great lineup of speakers. This year, we not only had the opportunity to attend but we all led a workshop and I presented a talk.
My talk was entitled “In the Loop: Designing Conversations with Algorithms”. In it, I shared signals we’re seeing that indicate a shifting relationship between people and algorithmic systems, discussed how those changes are at odds with some of the implicit ideas we’ve been building into innovation for decades, and advocated for a set of design principles that can help us create better interactions for the future: ones where we are empowered to engage in negotiations with the complex and increasingly pervasive systems around us. Full video of the talk (20 minutes) is below:
Our workshop was framed around the idea of “impulse response”, which refers to a means of sounding out the properties of an unknown system by sending a known signal into it. As increasing aspects of our lives are mediated by algorithmic systems, we adapt our behavior according to our understanding of how these systems sense, track, and analyze what we do. Some of these systems show us what they know and how they work; other systems may behave as black boxes, recording their observations and making inferences we don’t fully understand. As we learn more about how these systems work, what behaviors are emerging / will emerge to optimize or obscure our participation with them? We had participants design strategies, products, or other interventions that apply this idea to the systems around them. An overview video is below and you can also check out Lift’s Storify of the workshop.
Understanding these emerging trends around how people engage with algorithmic systems deeply informs the work we do in the R&D Lab. As we design and build new kinds of interactions with information, we strive to make those interactions embody the values of transparency, agency and virtuosity in order to create compelling, satisfying and empowered experiences for our users now and in the future.