Yandex Architecture

Mon, 02/25/2008 - 05:53 — Todd Hoff

Yandex Architecture (1063)

Update: Anatomy of a crash in a new part of Yandex written in Django. Writing to a magic session variable caused an unexpected write into an InnoDB database on every request. Writes took 6-7 seconds because of index rebuilding. Lots of useful details on the sizing of their system, what went wrong, and how they fixed it.

Yandex is a Russian search engine with 3.5 billion pages in their search index. We only know a few fun facts about how they do things, nothing at a detailed architecture level. Hopefully we'll learn more later, but I thought it would still be interesting. From Allen Stern's interview with Yandex's CTO Ilya Segalovich, we learn:

3.5 billion pages in the search index.

Over several thousand servers.

35 million searches a day.

Several data centers around Russia.

Two-layer architecture.

The database is split in pieces and when a search is requested, it pulls the bits from the different database servers and brings it together for the user.

Languages used: c++, perl, some java.

FreeBSD is used as their server OS.

$72 million in revenue in 2006.

Login or register to post comments
Visit Yandex Architecture
4794 reads

Comments

Wed, 01/23/2008 - 17:52 — Anonymous (not verified)

Re: Yandex Architecture

Yandex is more than a search engine. It's a portal. I have a friend with a yandex.ru email address.

Thu, 01/24/2008 - 06:55 — Anonymous (not verified)

Re: Yandex Architecture

Wow.

I never heard of Yandex but they obviously do some serious business over there. it's always cool to see successful things that make it out of russia / eastern europe: The top devs over there are actually really excellent: very bright and ultra-hardcore.

Fri, 02/01/2008 - 16:41 — Insight IT (not verified)

Re: Yandex Architecture

Yes, Yandex isn't just a search engine, it also provides fairly long list of services, e-mail included.
But from my point of view, their services are not really provided on the competitive level of quality, especially comparing with Google services: sometimes i think that Yandex search engine chooses results not by relevancy, but according to random numbers generator, mailboxes are constantly flooded by spam and so on.
As for me, I prefer using international service providers even considering a fact that I live in Russia, but still Yandex remains the largest russian provider of internet-services and the really huge amount of russian people uses it just because of several strange reasons, such as for example they just "used to perform search in Yandex", or maybe that's too time-consuming to type google.com instead of ya.ru...

Sat, 02/09/2008 - 19:35 — Anonymous (not verified)

Re: Yandex Architecture

Google does not search well in russian as russian has many forms of words. So too make good search in russian you should know russian very well. Right now google searches in russian as if it was english.

+ May be Russia supports its search engines to control russian part of internet.

There are russian search engines yandex and ramber, russian money transfer webmoney and yander money, there is russian social networks and of course rutube.

There is also russian domains that you can't type in your browser as you do not have russian keyboard.

Tue, 02/19/2008 - 14:44 — Anonymous (not verified)

Re: Yandex Architecture

(Russian ):
http://www.seotools.ru/biblioteka-optimizatora/yandeks/arhitektura.html - Yandex architecture (2000)

http://www.searchrank.ru/arxitektura-yandeks-poiska/ (2007)

Slides from HighLoad 2007 conference (russian)
http://www.google.ru/url?sa=t&ct=res&cd=7&url=http%3A%2F%2Fwww.jug.ru%2F...

Thu, 02/21/2008 - 18:08 — Anonymous (not verified)

Re: Yandex Architecture

Google does make stemming for cyrillic languages like Russian or Bulgarian (my native tongue). Perhaps Yandex does it better.

Thu, 02/21/2008 - 18:10 — Anonymous (not verified)

Re: Yandex Architecture

Damn, this captcha is case sensitive! Not a good choice, really!

Mon, 02/25/2008 - 19:00 — Anonymous (not verified)

Re: Yandex Architecture

Perhaps I'm picking a nit, but there are a couple of things off in the summary of the 'Anatomy of a crash' article.

They weren't writing to any magic variable. They were modifying the session data on every request (unnecessarily), which caused it to be saved to the DB. Nothing magic about that.

Lengthy index rebuilds were caused by the use of non-sequential primary keys (MD5 hashes) -- with InnoDB, that meant it would rebuild the whole thing on every request... thus poor performance.

Mon, 02/25/2008 - 19:32 — Todd Hoff

Re: Yandex Architecture

To me it's magic because the consequences are hard to deduce from the code. And the choice of key values having such a tremendous negative impact is another bit of wild magic. Would one expect such an effect by looking at that line? Not me, which is why magic popped to mind.

Fri, 06/20/2008 - 23:41 — Anonymous (not verified)

Re: Yandex Architecture

Wed, 06/25/2008 - 00:09 — youtube (not verified)

Re: Yandex Architecture

Google does make stemming for cyrillic languages like Russian or Bulgarian (my native tongue). Perhaps Yandex does it better.

High Scalability Building bigger, faster, more reliable websites.

Navigation

Who's online

User login

Syndicate