Edgescope - Sharing the View from a Distributed Internet Telescope

News

  • 4/5/2011: Ars Technica wrote about our analysis of BitTorrent usage during the recent Japanese earthquake and Internet shutdowns in Egypt and Libya.
  • 2/1/2010: Added description for Ono dataset, links to AS links/relationships dataset.
  • 9/18/2009: Site created. Data will be made available on a rolling basis. Please check back periodically for updated listings.

Motivation and Approach

Sound Internet research requires comprehensive network measurements. While our measurement techniques are continually improving, however, the visible portion of the Internet, and thus the representativeness of our measurement, is conversely shrinking. Although some distributed research platforms offer many vantage points in the network, much of the Internet's growth occurs in domains beyond their reach, such as behind NAT boxes and firewalls or in regions of the Internet not exposed by public BGP feeds. Providing access to universal network information would seem to require a widespread deployment of measurement agents in the edge of the network, introducing regular test traffic at sufficient scale to drive general experimentation. While a straightforward implementation of this vision at a global scale appears economically and socially infeasible, we posit that much of this traffic is already being generated and can be opportunistically measured.

The goal of EdgeScope is to expose the network view of end systems located at the edge of the network. To expose this view, we exploit the fact that peers in large-scale peer-to-peer systems (P2P) have a unique and valuable perspective on network conditions, one to which today's researchers, network operators and users have limited or no access. Through extensions to a popular BitTorrent client, we have collected diagnostic information that not only validates our design goals but also provides a rich source of data regarding edge system network views. Combined, these datasets cover hundreds of thousands of users located in hundreds of counties and thousands of networks worldwide.

Any dataset gathered from end users is subject to important challenges regarding data archival, distribution, processing and privacy. We will address this issues in the course of this project.

People

Faculty

Grad students and postdocs

Publications

Resources

  • Datasets:
    • We are currently building our archival and anonymization infrastructure and will publish new datasets as they become available. In the meantime, datasets from the Ono project are available upon request.
    • Read before requesting access: Data from the Ono/NEWS projects will be made available only to researchers. To get access to the data, you must send e-mail to This email address is being protected from spambots. You need JavaScript enabled to view it. in which you explicitly agree to the terms of this CAIDA-style release document.
    • AS links discovered through P2P traceroutes
    • AS links discovered through P2P traceroutes (updated May 2010)
  • Other public data