The life and times of Yahoo Pipes by Glenn Fleishman

Metadata

• Author: retool.com
• Full Title: The life and times of Yahoo Pipes by Glenn Fleishman
• Category:#articles
• URL: https://retool.com/pipes

Highlights

• Yahoo Pipes—or, officially, Pipes by Yahoo!, a rare switcheroo of the company’s name—was a service that offered all that and more in a single platform. Individual “Pipes,” as it were, were both personal and public—the service seemed like a portent of the future that techno-utopians were then predicting. But like many great products and services of its time, it never quite came to pass. (View Highlight)
• Sadri built Pipes as a cloud-hosted service alongside just four other core team members. Users dragged and dropped elements in a web browser to roll their own information feeds, taking published items from websites in many formats—news, database outputs, and blog posts among them. Pipes’ users could then merge the results, filter and process them, and produce a regularly updated stream that could be subscribed to in a desktop app—or texted to a phone. Geotagged results could even pass through a Yahoo Maps filter to restrict results based on location and proximity to features like parks or city neighborhoods. Every Pipes’ mapping was public; every Pipe could be copied and modified. “That was design principle one for us, and I think that had a lot to do with making it accessible to non-developers,” Sadri said. This openness that encouraged—really, required—knowledge-sharing embodied some of the internet’s best qualities that were on the verge of being lost at that time. (View Highlight)
• Sadri’s return to Yahoo after a year at Google might seem baffling in 2023. But times were different, and there was intrigue. While Google was rapidly churning out successful projects, Yahoo held a different sort of possibility, always seeming to be on the precipice of what could be some next big thing. (View Highlight)
• Horowitz gave him a clear mandate: “You want to work on some interesting projects? Take some time to come up with something to work on.” Though he had several ideas, it was the mélange of RSS news feeds, cloud-hosted software agents, visual programming, and in-browser interaction that would become Pipes that got Horowitz’s attention. (“Pipes,” as it were, stemmed from the pipeline concept in Unix, in which one kind of program could straightforwardly feed, or pipe with a |, its output to the next, and so on.) (View Highlight)
• YQL allowed for Yahoo-wide mashups, and ultimately let developers outside the company even store the results of queries in SQL tables at Yahoo. Effectively, it tried to turn Yahoo into a platform for other websites, while leveraging the advantage for new internal offerings and combinations. You could see this as a flicker (not Flickr) of what Amazon Web Services would ultimately offer: a way to offload certain kinds of programming structures and resources to a specialty provider instead of building out the tools and services on one’s own data center servers. (View Highlight)
• Pipes output could be embedded into a page or formatted as HTML, JSON, KML, RSS, or XML. There was even a way for a non-Yahoo website to include JavaScript that queried a Pipe workflow and received the results back without any front-end browser or back-end server involvement. Pipes, in effect, could power other sites. (View Highlight)
• Raffel recalls these gaming sessions as critical to Pipes product development. “It was so much easier to get feedback and to have a discussion while people were playing games and having fun,” he said. After a game ended, “We would usually go into a conversation mode and whiteboard out these ideas.” (View Highlight)
• Yahoo Pipes encompassed an array of control structures and programming constructs to allow it to work more like a general-purpose computing language, despite its limitations. You could use if…then conditionals, create loops to iterate through items, use splits and joins, and, yes, pipe the output to other web services. Pipes even used link syntax coloring, with control structure lines marked in gray and data flows in blue. While Automator facilitated program execution, Pipes went a step further by crafting mini-platforms. (View Highlight)
• Pipes offered reactive data pipelines and dynamic workflows designed to provide that same real-time visibility: a Pipe started running as soon as any element was connected, using visual indicators to show action. “We wanted the data that is flowing through the ‘pipes’ to be very visible,” Sadri said. An optional text debugger at the bottom of the screen provided additional feedback by displaying the current results as individual items. “It was really hard to write the wrong program in Pipes,” Sadri noted. “As soon as you dragged something on screen, it would immediately start working and produce some output.” (View Highlight)
• “Every time it came up, everybody jumped on it and it crashed again, and everybody cycled back into a waiting state,” said Trevor, the lead engineer. “It felt bad at the time, but in retrospect, it was awesome.” Raffel said Yahoo co-founder Jerry Yang told him to use his name to get whatever he needed to keep Pipes running then: “I’d never done that before. I certainly haven’t done that since.” The beta launch’s popularity caught the team so off guard that they hadn’t even prepared a crash page. (Though that soon changed: Trevor said that the first thing they created after the beta launch was a page that read something like “our pipes are clogged.”) (View Highlight)
• More broadly, Pipes was an anomaly in its heyday in that it offered something akin to programming for the non-coding masses—while creating entry points to programming for others and enthusiasm amongst developers eager to experiment with neoteric means of building (often for fun). On its end-of-life HackerNews thread, poster juliendorra shared:

  “I used Pipes as the basis of data mashup classes, with non-developers students that were mostly afraid/bored of anything related to programming. Using Pipes, in just a few hours they would make small, custom data apps.[…] I had a student using the location entity extraction to geocode his favorite street art blog that cited cities in the text but didn’t use rssgeo, and then output the augmented feed to a map view. […] It was a really effective tool for non-developers to learn about data markup standards, to think in term[s] of data flows, to get introduced to the idea of a data or web API.” (View Highlight)