Strategy: Efficiently Geo-referencing IPs

A lot of apps need to map IP addresses to locations. Jeremy Cole in On efficiently geo-referencing IPs with MaxMind GeoIP and MySQL GIS succinctly explains the many uses for such a feature: Geo-referencing IPs is, in a nutshell, converting an IP address, perhaps from an incoming web visitor, a log file, a data file, or some other place, into the name of some entity owning that IP address. There are a lot of reasons you may want to geo-reference IP addresses to country, city, etc., such as in simple ad targeting systems, geographic load balancing, web analytics, and many more applications. This is difficult to do efficiently, at least it gives me a bit of brain freeze. In the same post Jeremy nicely explains where to get the geo-rereferncing data, how to load data, and the performance of different approaches for IP address searching. It's a great practical introduction to the subject.

Click to read more ...


Starting a website from scratch - what technologies should I use?

Hi, if you were to design your own highly scalable website from scratch, what technologies would you use? Based on Web 2.0 popularity, LAMP seems to be high in the running. But would you tack on CakePHP? Drupal? or build your framework/CMS from scratch? What version of Linux runs best for a scalable website? Would you consider Windows and .NET? Java? Or do you want to throw a brick at me for even suggesting such heresies? Would you prefer Postgres, Tomcat, Perl, Python, or any of that other *NIX fancy stuff...why or why not? Please forget for the moment, "use what you know" argument. I am pretty versatile, and can look for an expert in whatever platform I choose. So all skills being equal, I'm looking for the best community support, the fastest development time and most importantly, the best scaling approach. Let's say, for fun, that I'm planning for the website to have as many messages going back & forth as an eBay. Definitely building this on a budget.. Jason

Click to read more ...


Solving the Client Side API Scalability Problem with a Little Game Theory

Now that the internet has become defined as a mashup over a collection of service APIs, we have a little problem: for clients using APIs is a lot like drinking beer through a straw. You never get as much beer as you want and you get a headache after. But what if I've been a good boy and deserve a bigger straw? Maybe we can use game theory to model trust relationships over a life time of interactions over many different services and then give more capabilities/beer to those who have earned them? Let's say Twitter limits me to downloading only 20 tweets at a time through their API. But I want more. I may even want to do something so radical as download all my tweets. Of course Twitter can't let everyone do that. They would be swamped serving all this traffic and service would be denied. So Twitter does that rational thing and limits API access as a means of self protection. As does Google, Yahoo, Skynet, and everyone else. But when I hit the API limit I think, but hey it's Todd here, we've been friends a long time now and I've never hurt you. Not once. Can't you just trust me a little? I promise not to abuse you. I never have and won't in the future. At least on purpose, accidents do happen. Sometimes there's a signal problem and we'll misunderstand each other, but we can work that out. After all, if soldiers during WW1 can learn how to stop the killing through forgiveness, so can we. The problem is Twitter doesn't know me so we haven't built up trust. We could replace trust with money, as in a paid service where I pay for each batch of downloads, but we're better friends than that. Money shouldn't come between us. And if Twitter knew what a good guy I am I feel sure they would let me download more data. But Twitter doesn't know me and that's the problem. How could they know me? We could set up authority based systems like the ones that let certain people march ahead through airport security lines, but that won't scale and I have feeling we all know how that strategy will work out in the end. Another approach to trust is a game theoretic perspective for assessing a user's trust level. Take the iterated prisoner's dilemma problem where variations on the tit for tat strategy are surprisingly simple ways cooperation could evolve in API world. We start out cooperating and if you screw me I'll screw you right back. In a situation where communication is spotty (like through an API) there can be bad signals sent so if people have trusted before then they'll wait for another iteration to see if the other side defects again, in which case they retaliate. Perhaps if services modeled API limits like a game and assessed my capabilities by how we've played the game together, then capabilities could be set based on earned and demonstrated trust rather than simplistic rules. A service like Mashery could takes us even further by moving us out of the direct reciprocity model, where we judge each other on our one on one interactions, and into a more sophisticated indirect reciprocity model, where agents can make decisions to help those who have helped others. Mashery can take a look at how API users act in the wider playing of multiple services and multiple agents. If you are well behaved using many different services, shouldn't you earn more trust and thus more capabilities? In the real world if someone vouches for you to a friend then you will likely get more slack because you have some of the trust from your friend backing you. This doesn't work in one on one situation because there's no way to establish your reputation. Mashery on the other hand knows you and knows which APIs you are using and how you are using them. Mashery could vouch for you if they detected you were playing fair so you get more capabilities initially and transit the capability scale faster if you continued to behave. You can obviously go on and on imaging how such a system might work. Of course, there's a dark side. Situations are possible like on Ebay where people spend eons setting up a great reputation only to later cash in their reputations in some fabulous scam. That's what happens in a society though. We all get more capabilities at the price of some extra risk.

Click to read more ...


Scale to China

Hello all, does anyone have experience in scaling a european website to china? The main problem in china is the internet connectivity to websites outside china, that means latency and packetloss (and perhaps filtering) make things difficult. The options I see are: 1. Host you application in china, but where? I haven't got a answer from any chinese ISP I contacted. On the other hand I don't really want to host in china. 2. Build your own CDN. Wikipedia shows how it goes. Get a bunch of machines (but where? see point 1) put squid on them, implement intelligent cache invalidation and you're set. But where can I get machines in china? Where do I need them in china? There are soe big isps with limited peering capability, so I'd need servers in every network. 3. Get professional CDN services. Akamai, ChinaCache, CDNetworks, etc etc.. They all provide services in china. The problem is: they are all very expensive. 4. Amazon EC2/S3 ? Is it worth thinking about this way? I am not sure, because they only have US and Ireland based datacenters. So we are stuck to the connectivity problem.. My favourite way: Rent a bunch of linux servers in 4-5 big cities in china in different networks and build my own CDN. What do you think? Regards Bjoern

Click to read more ...


Why not Cache from Intersystems?

I have some experience with a very large OLTP system that is 7+ TB in size and performs very well for 30K+ concurrent users. It is built using Intersystems Cache based on the very old but very scalable MUMPS platform. Why don't I see more discussions about archiectures such as these in this forum? I am curious why this platform scales so much better then the typical RDBMS.

Click to read more ...


n-phase commit for FS writes, reads stay local

I am trying to find a Linux FS that will allow me to replicate all writes synchronously to n nodes in a web server cluster, while keeping all reads local. It should not require specialized hardware.

Click to read more ...


what is j2ee stack

I see everyone talk about lamp stack is less than j2ee stack .i m newbie can anyone plz explain what is j2ee stack

Click to read more ...


Product: SmartFrog a Distributed Configuration and Deployment Framework

From Wikipedia: SmartFrog is an open-source software framework, written in Java, that manages the configuration, deployment and coordination of a software system broken into components. These components may be distributed across several network hosts. The configuration of components is described using a domain-specific language, whose syntax resembles that of Java. It is a prototype-based object-oriented language, and may thus be compared to Self. The framework is used internally in a variety of HP products. Also, it is being used by HP Labs partners like CERN.

Related Articles

  • Distributed Testing with SmartFrog
  • Puppet the Automated Administration System

    Click to read more ...

  • Monday

    Tailrank Architecture - Learn How to Track Memes Across the Entire Blogosphere

    Ever feel like the blogosphere is 500 million channels with nothing on? Tailrank finds the internet's hottest channels by indexing over 24M weblogs and feeds per hour. That's 52TB of raw blog content (no, not sewage) a month and requires continuously processing 160Mbits of IO. How do they do that? This is an email interview with Kevin Burton, founder and CEO of Kevin was kind enough to take the time to explain how they scale to index the entire blogosphere.


  • Tailrank - We track the hottest news in the blogosphere!
  • Spinn3r - A blog spider you can specialize with your own behavior instead of creating your own.
  • Kevin Burton's Blog - his blog is an indexing mix of politics and technical talk. Both are always interesting.


  • MySQL
  • Java
  • Linux (Debian)
  • Apache
  • Squid
  • PowerDNS
  • DAS storage.
  • Federated database.
  • ServerBeach hosting.
  • Job scheduling system for work distribution.


  • What is your system is for? Tailrank originally a memetracker to track the hottest news being discussed within the blogosphere. We started having a lot of requests to license our crawler and we shipped that in the form of Spinn3r about 8 months ago. Spinn3r is self contained crawler for companies that want to index the full blogosphere and consumer generated media. Tailrank is still a very important product alongside Spinn3r and we're working on Tailrank 3.0 which should be available in the future. No ETA at the moment but it's actively being worked on.
  • What particular design/architecture/implementation challenges does your system have? The biggest challenge we have is the sheer amount of data we have to process and keeping that data consistent within a distributed system. For example, we process 52TB of content per month. this has to be indexed in a highly available storage architecture so the normal distributed database problems arise.
  • What did you do to meet these challenges? We've spent a lot of time in building out a distributed system that can scale and handle failure. For example, we've built a tool called Task/Queue that is analogous to Google's MapReduce. It has a centralized queue server which hands out units of work to robots which make requests. It works VERY well for crawlers in that slower machines just fetch work at a slower rate while more modern machines (or better tuned machines) request work at a higher rate. This ends up easily solving one of the main distributed computing fallacies that the network is homogeneous. Task/Queue is generic enough that we could actually use it to implement MapReduce on top of the system. We'll probably open source it at some point. Right now it has too many tentacles wrapped into other parts of our system.
  • How big is your system? We index 24M weblogs and feeds per hour and process content at about 160-200Mbps. At the raw level we're writing to our disks at about 10-15MBps continuously.
  • How many documents, do you serve? How many images? How much data? Right now the database is about 500G. We're expecting it to grow well beyond this in 2008 as we expand our product offering.
  • What is your rate of growth? It's mostly a function of customer feature requests. If our customers want more data we sell it to them. In 2008 we're planning on expanding our cluster to index larger portions of the web and consumer generated media.
  • What is the architecture of your system? We use Java, MySQL and Linux for our cluster. Java is a great language for writing crawlers. The library support is pretty solid (though it seems like Java 7 is going to be killer when they add closures). We use MySQL with InnoDB. We're mostly happy with it though it seems I end up spending about 20% of my time fixing MySQL bugs and limitations. Of course nothing is perfect. MySQL for example was really designed to be used on single core systems. The MySQL 5.1 release goes a bit farther to fix multi-core scalability locks. I recently blogged about how these the new multi-core machines should really be considered N machines instead of one logical unit: Distributed Computing Fallacy #9.
  • How is your system architected to scale? We use a federated database system so that we can split the write load as we see more IO. We've released a lot of our code as Open Source a lot of our infrastructure and this will probably be released as Open Source as well. We've already opened up a lot of our infrastructure code:
  • - load balancing JDBC driver for use with DB connection pools.
  • - Java RSS/Atom parser designed to elegantly support all versions of RSS
  • - Java (and UNIX) equivalent of Windows' perfmon
  • - Client bindings to access the Spinn3r web service
  • - Clone a MySQL installation and setup replication.
  • - Logger facade that supports printf style message format for both performance and ease of use.
  • How many servers do you have? About 15 machines so far. We've spent a lot of time tuning our infrastructure so it's pretty efficient. That said, building a scalable crawler is not an easy task so it does take a lot of hardware. We're going to be expanding FAR past this in 2008 and will probably hit about 2-3 racks of machines (~120 boxes).
  • What operating systems do you use? Linux via Debian Etch on 64 bit Opterons. I'm a big Debian fan. I don't know why more hardware vendors don't support Debian. Debian is the big secret in the valley that no one talks about. Most of the big web 2.0 shops like Technorati, Digg, etc use Debian.
  • Which web server do you use? Apache 2.0. Lighttpd is looking interesting as well.
  • Which reverse proxy do you use? About 95% of the pages of Tailrank are served from Squid.
  • How is your system deployed in data centers? We use ServerBeach for hosting. It's a great model for small to medium sized startups. They rack the boxes, maintain inventory, handle network, etc. We just buy new machines and pay a flat markup. I wish Dell, SUN, HP would sell directly to clients in this manner. One right now. We're looking to expand into two for redundancy.
  • What is your storage strategy? Directly attached storage. We buy two SATA drives per box and set them up in RAID 0. We use the redundant array of inexpensive databases solution so if an individual machine fails there's another copy of the data on another box. Cheap SATA disks rule for what we do. They're cheap, commodity, and fast.
  • Do you have a standard API to your website? Tailrank has RSS feeds for every page. The Spinn3r service is itself an API and we have extensive documentation on the protocol. It's also free to use for researchers so if any of your readers are pursuing a Ph.D and generally doing research work and needs access to blog data we'd love to help them out. We already have the Ph.D students at the University of Washington and University of Maryland (my Alma Matter) using Spinn3r.
  • Which DNS service do you use? PowerDNS. It's a great product. We only use the recursor daemon but it's FAST. It uses async IO though so it doesn't really scale across processors on multicore boxes. Apparenty there's a hack to get it to run across cores but it isn't very reliable. AAA caching might be broken though. I still need to look into this.
  • Who do you admire? Donald Knuth is the man!
  • How are you thinking of changing your architecture in the future? We're still working on finishing up a fully sharded database. MySQL fault tolerance and autopromotion is also an issue.

    Click to read more ...

  • Sunday

    Reverse Proxy

    Hi, I saw an year ago that Netapp sold netcache to blu-coat, my site is a heavy NetCache user and we cached 83% of our site. We tested with Blue-coat and F5 WA and we are not getting same performce as NetCache. Any of you guys have the same issue? or somebody knows another product can handle much traffic? Thanks Rodrigo

    Click to read more ...