advertise
« The I.H.S.D.F. Theorem: A Proposed Theorem for the Trade-offs in Horizontally Scalable Systems | Main | Gigaspaces curbs latency outliers with Java Real Time »
Saturday
Dec202008

Second Life Architecture - The Grid

Update:Presentation: Second Life’s Architecture. Ian Wilkes, VP of Systems Engineering, describes the architecture used by the popular game named Second Life. Ian presents how the architecture was at its debut and how it evolved over years as users and features have been added.

Second Life is a 3-D virtual world created by its Residents. Virtual Worlds are expected to be more and more popular on the internet so their architecture might be of interest. Especially important is the appearance of open virtual worlds or metaverses.
What happens when video games meet Web 2.0? What happens is the metaverse.

Information Sources

 

 

Platform

 

  • MySQL

  • Apache

  • Squid

  • Python

  • C++

  • Mono

  • Debian

 

What's Inside?

 

 

The Stats

 

  • ~1M active users

  • ~95M user hours per quarter

  • ~70K peak concurrent users (40% annual growth)

  • ~12Gbit/sec aggregate bandwidth (in 2007)

 

 

Staff (in 2006)

 

  • 70 FTE + 20 part time


"about 22 are programmers working on SL itself. At any one time probably 1/3 of the team is on infrastructure, 1/3 is on new features and 1/3 is on various maintenance tasks (bug fixes, general stability and speed improvements) or improvements to existing features. But it varies a lot."

Software


Client/Viewer

  • Open Source client

  • Render the Virtual World

  • Handles user interaction

  • Handles locations of objects

  • Gets velocities and does simple physics to keep track of what is moving where

  • No collision detection


Simulator (Sim)
Each geographic area (256x256 meter region) in Second Life runs on a single instantiation of server software, called a simulator or "sim." And each sim runs on a separate core of a server.
The Simulator is the primary SL C++ server process which runs on most servers. As the viewer moves through the world it is handled off from one simulator to another.

  • Runs Havok 4 physics engine

  • Runs at 45 frames/sec. If it can't keep up, it will attempt time dialation without reducing frame rate.


  • Handles storing object state, land parcel state, and terrain height-map state

  • Keeps track of where everything is and does collision detection

  • Sends locations of stuff to viewer

  • Transmits image data in a prioritized queue

  • Sends updates to viewers only when needed (only when collision occurs or other changes in direction, velocity etc.)

  • Runs Linden Scripting Language (LSL) scripts

  • Scripting has been recently upgraded to the much faster Mono scripting engine

  • Handles chat and instant messages
    •  
      • One big clustered filesystem ~100TB

      • Stores asset data such as textures.
      • Eventlet is a networking library written in Python. It achieves high scalability by using non-blocking io while at the same time retaining high programmer usability by using coroutines to make the non-blocking io operations appear blocking at the source code level.

      • Mulib is a REST web service framework built on top of eventlet
      • 2000+ Servers in 2007

      • ~6000 Servers in early 2008

      • Plans to upgrade to ~10000 (?)

      • 4 sims per machine, for both class 4 and class 5

      • Used all-AMD for years, but are moving from the Opteron 270 to the Intel Xeon 5148

      • The upgrade to "class 5" servers doubled the RAM per machine from 2GB to 4GB and moved to a faster SATA disk

      • Class 1 - 4 are on 100Mb with 1Gb uplinks to the core. Class 5 is on pure 1Gb


    • Asset Server


      MySQL database
      Second Life has started with One Database, and have subsequently been forced into clustering.

      They use a ton of MySQL databases running on Debian machines to handle lots of centralized services. Rather than attempt to build the one, impossibly large database – all hail the Central Database – or one, impossibly large central cluster – all hail the Cluster – Linden Lab instead adopted a divide and conquer strategy based around data partitioning. The good thing is that UUIDs– 128-bit unique identifiers – are associated with most things in Second Life, so partitioning is generally doable.

      Backbone
      Linden Lab has converted much of their backend architecture away from custom C++/messaging into web services. Certain services has been moved off of MySQL – or cached (Squid) between the queries and MySQL. Presence, in particular Agent Presence, ie are you online and where are you on the grid, is a particularly tricky kind of query to partition, so there is now a Python service running on the SL grid called Backbone. It proved to be easier to scale, develop and maintain than many of their older technologies, and as a result, it plays an increasingly important role in the Second Life platform as Linden Lab migrates their legacy code to web services.
      Two main components of the backbone are open source:


      Hardware





      Do you have more details?



Reader Comments (7)

Thanks geekr, that was very interesting.

November 29, 1990 | Unregistered CommenterTodd Hoff

You've assembled a nice collection of facts. Interesting to compare this with the setup about 18 months ago (see http://blogs.computerworld.com/node/5122).

Also, while the 40% annual growth rate in concurrent visitors is encouraging, peak currency and other metrics have actually levelled off somewhat this year (see http://nwn.blogs.com/nwn/demographics/index.html)

Ian Lamont
The Industry Standard

November 29, 1990 | Unregistered CommenterIan Lamont

Todd, I am happy that you find this very interesting. It was challenging to collect all these facts :)

I think that the scalability of virtual worlds will be as important as it is for web sites.

November 29, 1990 | Unregistered Commentergeekr

Awesome article! Nice to read about something a little different than your standard website structure, especially an MMO. Not a lot about how those are set up floating around out there.

November 29, 1990 | Unregistered CommenterUltimateBrent

Each geographic area (256x256 meter region) in Second Life runs on a single instantiation of server software, called a simulator or "sim." And each sim runs on a separate core of a server.

So that explains why popular regions bog down to a crawl. Doh!

November 29, 1990 | Unregistered CommenterAnonymous

Isn't SL a bit... stagnating?

November 29, 1990 | Unregistered CommenterMichal Frackowiak

FYI

With an innovative, business-model approach, the book http://www.amazon.com/gp/product/1418052671?ie=UTF8&tag=innoblog-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=1418052671">Game Development Essentials: Online Game Developmenthttp://www.assoc-amazon.com/e/ir?t=innoblog-20&l=as2&o=1&a=1418052671" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" /> provides the essentials needed to achieve long-term success with Massively Multiplayer Online Games (MMOGs) such as Second Life and WoW. This book takes a significant and valuable departure from traditional game development books by addressing MMOG development as a complex, multi-faceted, service-oriented business, instead of focusing solely on technical, artistic, or design techniques. The resulting multi-dimensional focus allows readers to design their game and organize their development process with the entire business in mind. Coverage includes the key differences between single player games and MMOGs, as well as how the various components of the development process, such as the business model, marketing plan, gaming community and technical constraints, influence one another and determine the success of the MMOG.

November 29, 1990 | Unregistered Commentergeekr

PostPost a New Comment

Enter your information below to add a new comment.
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>