Does anyone know of any articles or papers that discuss the nuts and bolts of how web analytics is implemented at organizations with large volumes of web traffic and a critcal business need to analyze that data - e.g. places like Amazon.com, eBay, and Google? Just as a fun project I'm planning to build my own web log analysis app that can effectively index and query large volumes of web log data (i.e. TB range). But first I'd like to learn more about how it's done in the organizations whose lifeblood depends on this stuff. Even just a high level architectural overview of their approaches would be nice to have.
Hi, Some time ago , martin fowler bloged about how HTTP cache headers can be very effectively used in web site design. http://www.martinfowler.com/bliki/SegmentationByFreshness.html How actively HTTP cache headers are considered in web site design? I think it is a great tool to reduce lot of load on server and should be considered before designing any complex caching strategy. Thoughts? Thanks, Unmesh
I found this resources: High Scalable Architecture: - YouTube Architecture - Facebook Chat Architecture - Amazon Architecture Blogs: - Scalability Guidelines for building scalable software system (part 1) - Scalability Guidelines for building scalable software system (part 2) - Scalability Guidelines for building scalable software system (part 3) - Scalability Worst Practices - how to minimize load time for fast user experiences - Scalability principles - Challanges for Developing Enterprise Application on the Cloud - high-performance web page real-world examples netflix case study - Intro to Caching,Caching algorithms and caching frameworks part 1 - Amdahl’s low - How I Learned to Stop Worrying and Love Using a Lot of Disk Space to Scale - Top 25 Most Dangerous Programming Mistakes
There were many talks recently about twitter scalability and their specific choice of language such as Scala to address their existing Ruby based scalability. In this post i tried to provide a more methodical approach for handling twitter scalability challenges that is centered around the right choice of architecture patterns rather then the language itself. The architecture pattern are given in a generic fashion that is not specific to twitter itself and can serve anyone who is looking to build a scalable real time web application in the near future.
This post I provided a summary of recent discussions outlining the main challenges that developers face today when deploying their existing JEE application to the cloud such as complexity, database integration, security, standard JEE support etc. In this post i also provided summary of how we managed to handle those challenges with our new Cloud Computing Framework by pointing to an existing production reference of a leading Telco provider.
For those interested in building scalable systems, today I will speak about the Facebook Char architecture. Starting keynote:
''When your feature’s userbase will go from 0 to 70 million practically overnight, scalability has to be baked in from the start.''Eugene Lutuchy, lead engineer on Facebook Chat
Facebook's engg. director aditya talks about facebook architecture. How they use mysql, php and memcache. How they have modified the above to suit their requirements.
I'm seeking for a design pattern or advice or directions. I need to count views/downloads of a set of resources, let them to be identified by their respective URLs. This is not a big problem. I also need to keep a list of viewed/downloaded resources in the last X days. This list needs to be updated every now and then to reflect real last X days of usage. So resources that were requested prior to X days get evicted from it. So it's sort of a black box, you feed messages (download request) in and it gives you that list of URLs with counters on the other end. How would you go about designing it?