advertise
« Server load balancing architectures, Part 1: Transport-level load balancing | Main | Alternatives to Google App Engine »
Wednesday
Oct222008

EVE Online Architecture

Sorry, the content for this post apparently did not make the transition from the old HighScalability website, it's all messed up, but there's still a some useful content...

EVE Online is "The World's Largest Game Universe", a massively multiplayer online game (MMO) made by CCP. EVE Online's Architecture is unusual for a MMOG because it doesn't divide the player load among different servers or shards. Instead, the same cluster handles the entire EVE universe. It is an interesting to compare this with the Architecture of the Second Life Grid. How do they manage to scale?

Information Sources

Platform

  • Stackless Python used for both server and client game logic. It allows programmers to reap the benefits of thread-based programming without the performance and complexity problems associated with conventional threads.
  • SQL Server
  • Blade servers with SSDs for high IOPS
  • Plans to use Infiniband interconnects for low latency networking
  • Founded in 1997
  • ~300K active users
  • Up to 40K concurrent users
  • Battles involving hundreds of ships
  • 250M transactions per day
  • Proxy Blades - These are the public facing segment of the EVE Cluster - they are responsible for taking player connections and establishing player communication within the rest of the cluster.
  • SOL Blades - These are the workhorses of Tranquility. The cluster is divided across 90 - 100 SOL blades which run 2 nodes each. A node is the primarily CPU intensive EVE server process running on one core. There are some SOL blades dedicated to one busy solar systems such as Jita, Motsu and Saila.
  • Database Cluster - This is the persistence layer of EVE Online. The running nodes interact heavily with the Database, and of course pretty much everything to do with the game lives here. Thanks to Solid-state drives, the database is able to keep up with the enormous I/O load that Tranquility generates.
  • With innovative ideas MMO games can scale up to the hundreds of players in the same battle.
  • SSDs will in fact bridge the gap huge performance gap between the memory and disks to some extent.
  • Low latency Infiniband network interconnect will enable larger clusters.

Lessons Learned

  • There are many interesting facts about the architecture of the EVE Online MMOG such as the use of Stackless Python and SSDs.
  • Check out the information sources for detailed insights to the development and operation of the EVE Online game.

Reader Comments (9)

  • Battles involving 1000+ ships

No, no, a thousand times no. If you want an experience that is remotely playable for the majority of people involved, we're looking at battles of 400 or 500 as an absolute maximum. Everything above that is unplayable and potentially node/game-crashing. I do not contest any of the other numbers though, all in all they have a monster of a cluster running the Eve-O universe.

November 29, 1990 | Unregistered Commentersiliciferous

I'm calling BS - we'd routinely crash the node with 60 - 100 ship battles.

November 29, 1990 | Unregistered CommenterAnonymous

Thank you for the feedback. I have updated the entry.

November 29, 1990 | Unregistered Commentergeekr

They've updated recently and they've managed to get a system to hold at a thousand ships, of course this doesn't mean that battles will be possible on this scale, it merely means that the system in question, we'll say Jita, will not crash. But 400-500 ship battles may not be so far fetched as we may once have believed. With the massive updates they've been doing at CCP it would not surprise me to see such battles taking place on a regular basis. Of course I won't be participating in them because I am a coward and I rather enjoy having my HIC in one piece.

Fly Safe.

November 29, 1990 | Unregistered CommenterShaydeStar

yes and no..

I am the CEO of a Goonswarm corporation and take part in battles with over 1000 players in local most weeks. Since CCP launched "Stackless IO" as well as upgrading to 64-bit architecture on the serverside we have noticed serious improvements in server-generated lag. If we warn CCP 24 hrs in advance of a node being put under stress, they will reinforce that node for us and 99% of the time that fight will be more than playable.

You will also notice that you can undock in jita without waiting 10 mins to load grid, which wasn't the case a few months ago.

Even the highest end graphics card/CPU combinations will struggle with displaying and keeping track of thousands of individual objects on screen (ships, missiles, wrecks, capsules, fighters & drones), so large battles these days generally rely on the power of your PC as opposed to server responsiveness.

November 29, 1990 | Unregistered CommenterAnonymous

I can see how the server architecture could be applied to a first person game but the design challenges make it incredibly difficult to get right. It's not just a matter of a company buying the designs or engine from CCP, it would need to be completely gutted and redesigned for a first person perspective game. The games industry's major investors seem unwilling to risk their money on big changes like that, unfortunately, but it would make one hell of a game if someone managed it.

http://www.mustuniversity.com/Schools-Majors/Applied-Arts/Architecture.html">online architecture degree

November 29, 1990 | Unregistered CommenterAnonymous

Even the highest end graphics card/CPU combinations will struggle with displaying and keeping track of thousands of individual objects on screen (ships, missiles, wrecks, capsules, fighters & drones), so large battles these days generally rely on the power of your PC as opposed to server responsiveness. http://www.carrentaladvice.net/">car rental

November 29, 1990 | Unregistered CommenterGregorr

I think anything over 500 ships is overkill anyway. I'm quite comfortable with the current limit as I tend to steer clear of the large scale smirmishes in any event.

Dave | http://www.pilkingtonselfcleaningglass.co.uk/where-to-use/glass-doors/index.html ">Glass Doors

November 29, 1990 | Unregistered CommenterDave

If we warn CCP 24 hrs in advance of a node being put under stress, they will reinforce that node for us and 99% of the time that fight will be more than playable.http://www.livescores.com.sg/">Live Scores

November 29, 1990 | Unregistered CommenterJeffSmith

PostPost a New Comment

Enter your information below to add a new comment.
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>