A tool for connecting data across the web that never came to be

  • started off as a “skunkworks” effort within Yahoo as part of cutting edge ideas
    • finished version was intended for end users to help program websites
  • Final product was a cloud-hosted service where users could drag and drop elements to “roll their own information feeds”
    • these feeds could take data directly from websites, be filtered, merged, and processed and produce a final stream that could be subscribed to
  • Pasha Sadri lead this effort
  • The problem emerged when they launched they got 100,000s of users rather than thousands and each pipe in aggregate pulled a ton of data and required a lot of processing
  • Supported RSS, Atom, CSS, RDF, etc. and eventually YQL
  • and very powerful output formatting:
    • Pipes output could be embedded into a page or formatted as HTML, JSON, KML, RSS, or XML. There was even a way for a non-Yahoo website to include JavaScript that queried a Pipe workflow and received the results back without any front-end browser or back-end server involvement. Pipes, in effect, could power other sites.

  • But other websites also started blocking pipes
    • ex: craigslist, supposedly because of resource hogging after someone created a map of craigslist listings (source)
      • lol they literally hardcoded “” and blocked it
      • but they brought it back 2 weeks later
  • HN discussion on the article
    • effectively allowed you to bypass cors to build clients that used data from other sites
  • A youtube video with title that changes with the view count: