3R: Sustainable Scalability for Globally-Distributed Systems

Motivation and approach

Over the past few years, we have witnessed an impressive growth in the number of globally-distributed networked systems from data sharing and content distribution to web caching, virtual clusters and general communication infrastructures. Interest in such systems is not restricted to academic environments, as shown by the popularity of peer-to-peer (P2P) applications like content sharing and Internet telephony.

Most of these systems are built following an overlay approach, where routing is controlled at the application layer and each edge in the overlay corresponds to a unicast path between two end systems in the underlying Internet. All of these systems have been designed under the basic assumption that no information is available on the underlying hosts' loads and network conditions. Consequently, each of them regularly and independently probes its environment as it attempts to identify better paths, route around problematic links, pair up nodes that are near each other or find good hosts to replicate content for availability. We believe that this approach is fundamentally unsustainable, negatively impacts the accuracy of our measurements and significantly restricts the variety, number and span of large-scale distributed services.

We believe that globally-distributed systems can be built to ensure long-term scalability by strategically reusing the view of the network gathered by long-running, ubiquitous services, such as Content Distribution Networks and popular P2P systems. We have started to explore a new approach to the design and implementation of distributed systems -- one that focuses on minimizing aggregated control and administrative overhead by strategically reusing environment's views and/or recycling previously gathered measurements. We call this approach 3R for Reduce, Reuse, Recycle, highlighting the analogy between our natural habitat and the artificial, Internet environment shared by these systems. Thus, the goal of sustainable scalability is to ensure that distributed systems are developed in a way that meets their current needs without compromising their own and next-generation systems' scalability.

Our work focus on identifying potential sources of information on network and hosts, designing efficient techniques to gather and access this information, architecting ways to match it with the needs of distributed systems, and deriving approaches for the design of globally-distributed systems that build on this information.

People

Publications

Resources

  • Ono plugin for the Azureus BitTorrent client. The main goal of this plugin is simple -- to improve download speeds for your BitTorrent client by proactively finding peers that are close to you. We identify nearby peers by reusing network measurements from content distribution networks (CDNs), i.e. without performing extensive path measurement or probing.