Building Scalable Databases: Denormalization, the NoSQL Movement and Digg

Database normalization is a technique for designing relational database schemas that ensures that the data is optimal for ad-hoc querying and that modifications such as deletion or insertion of data does not lead to data inconsistency. Database denormalization is the process of optimizing your database for reads by creating redundant data. A consequence of denormalization is that insertions or deletions could cause data inconsistency if not uniformly applied to all redundant copies of the data within the database.

Read more on Carnage4life blog...


How to handle so many socket connection

In my application, we receive request from many clients through socket.Client can connect to this socket and send data. this connection has to be maintained for indefinite hours. Client are continously sending data . There are so many clients simultaneously to server. I am using java to make application which listen on port and do processing on data. How can i scale this socket overhead. Is there any product which helps in maintening socket .


GridwiseTech revolutionizes data management

GridwiseTech has developed AdHoc, an advanced framework for sharing geographically distributed data and compute resources. It simplifies the resource management and makes cooperation secure and effective.
The premise of AdHoc is to enable each member of the associated institution to control access to his or her resources without an IT administrator’s help, and with high security level of any exposed data or applications assured.
It takes 3 easy steps to establish cooperation within AdHoc: create a virtual organization, add resources and share them. The application can be implemented within any organization to exchange data and resources or between institutions to join forces for more efficient results.
AdHoc was initially created for a consortium of hospitals and institutions to share medical data sets. As a technical partner in that project, GridwiseTech implemented the Security Framework to provide access to that data and designed a graphical tool to facilitate the administration of the entire system.

Every participant agreed to grant access to its resources to other partners in the project. Analysis of more patients’ records meant bigger samples and, potentially, better research. As most of these data are subject to a strict privacy policy, they could only be accessible for specific research purposes within defined time periods. In each case, patients’ identity remained anonymous and they provided consent to use their data for experiments. AdHoc enabled easy dynamic access rights management and, at the same time, prevented unauthorized access to sensitive information.
“Advanced international scientific consortia need to set up ad-hoc collaborations. For this reason, we used the concept of Virtual Organizations, introduced by international Grid projects. However, to create such a VO and grant people access to different resources, a lot of administrative effort is needed, including admins’ time and paperwork. GridwiseTech's AdHoc software is the first application I know of truly dynamic Virtual Organizations, where users themselves are responsible for their resources and can share them easy in real time without involving an administrator” said Andrea De Luca, Clinician and Researcher at the Institute of Clinical Infectious Diseases, Catholic University of Rome, Italy.
In this critical domain, the GridwiseTech software system proved to be versatile. Its combination of security and simplicity makes it a unique tool for rapid collaborations and modern e-Science.

-AdHoc bases on open–source components such as Shibboleth from Internet2.
-AdHoc was used within the ViroLab,project, an EU-funded research initiative in the scope of the 6th Framework Programme. ViroLab’s main objective is to develop a “Virtual Laboratory” for medical experts enabling clinical studies, medical knowledge discovery, and decision support for HIV drug resistance.


Product: Infinispan - Open Source Data Grid

Infinispan is a highly scalable, open source licensed data grid platform in the style of GigaSpaces and Oracle Coherence.

From their website:

The purpose of Infinispan is to expose a data structure that is highly concurrent, designed ground-up to make the most of modern multi-processor/multi-core architectures while at the same time providing distributed cache capabilities. At its core Infinispan exposes a JSR-107 (JCACHE) compatible Cache interface (which in turn extends java.util.Map). It is also optionally is backed by a peer-to-peer network architecture to distribute state efficiently around a data grid.

Offering high availability via making replicas of state across a network as well as optionally persisting state to configurable cache stores, Infinispan offers enterprise features such as efficient eviction algorithms to control memory usage as well as JTA compatibility.

In addition to the peer-to-peer architecture of Infinispan, on the roadmap is the ability to run farms of Infinispan instances as servers and connecting to them using a plethora of clients - both written in Java as well as other popular platforms.

A few observations:

  • Open source is an important consideration, depending on your business model. As you scale out your costs don't go up. The downside is you'll likely put in more programming effort to implement capabilities the commercial products have already solved.
  • It's from the makers of Jboss Cache so it's likely to have a solid implmentation, even so early in it's development cycle. The API looks very well thought out.
  • Java only. Plan is to add more bindings in the future.
  • Distributed hash table only. Commercial products have very advanced features like distributed query processing which can make all the difference during implementation. We'll see how the product expands from its caching roots into a full fledged data manipulation platform.
  • MVCC and a STM-like approach provide lock- and synchronization-free data structures. This means dust off all those non-blocking algorithms you've never used before. It will be very interesting to see how this approach performs under real-life loads programmed by real-life programmers not used to such techniques.
  • Data is made safe using a configurable degree of redundancy. State is distributed across a cluster. And it's peer-to-peer, there's no central server.
  • API based (put and get operations). XML, bytecode manipulation and JVM hooks aren't used.
  • Future plans call for adding a compute-grid for map-reduce style operations.
  • Distributed transactions across multiple objects are supported. It also offers eviction strategies to ensure individual nodes do not run out of memory and passivation/overflow to disk. Warm-starts using preloads are also supported.

    It's exciting to have an open source grid alternative. It will be interesting to see how Infinispan develops in quality and its feature set. Making a mission critical system of this type is no simple task.

    I don't necessarily see Infinispan as just a competitor for obvious players like GigaSpaces and Coherence, it may play even more strongly in the NoSQL space. For people looking for a reliable, highly performant, scalable, transaction aware hash storage system, Ininispan may look even more attractive than a lot of the disk based systems.

    Related Articles

  • Video Interview with Manik Surtani, Founder & Project Lead at JBoss Cache, Infinispan Data Grid
  • Infinispan Interview by Mark Little on InfoQ.
  • Are Cloud Based Memory Architectures the Next Big Thing?
  • Infinispan - data grids meets open source on
  • Technical FAQs
  • Anti-RDBMS: A list of distributed key-value stores
  • Infinispan Wiki
  • Distribution instead of Buddy Replication
  • Sunday

    Some Hot Links


    Hot Links for 2009-9-4 

  • A tour through hybrid column/row-oriented DBMS schemes by DANIEL ABADI. Approaches: PAX, Fractured Mirrors, and Fine-grained hybrids.
  • The Future of Database Clustering by ROBERT HODGES. Simple management and monitoring, Fast, flexible replication, Top-to-bottom data protection, Partition management, Cloud and virtualized operation, Transparent application access, Open source.
  • Some perspective to this DIY storage server mentioned at Storagemojo by Joerg Moellenkamp. Quality costs. Period.
  • Turn up the volume: API Scalability with Caching by Scott.
  • Disk I/O Bottlenecks by Ryan Thiessen. My first approach to diagnosing a performance problem is to start by trying to find the system’s bottleneck.
  • Patterns for Cloud Computing by Simon Guest. Using the Cloud for Scale, Using the Cloud for Multi-Tenancy, Using the Cloud for Compute, Using the Cloud for Storage, Using the Cloud for Communications
  • Server Processor Roadmaps Show Change in Direction By Michael J. Miller. What fascinates me is the big change in direction we're seeing on server chips...The focus seemed to be on putting more cores on a chip, something we're still seeing with these new 8-, 12-, and 16-core chips. But now a lot of focus seems to be going into increasing memory bandwidth and new cache architectures, as designers are addressing the memory issues that are often the bottleneck in a multicore system, as well as core-to-core communications.
  • Confronting the Data Center Crisis: A Cost - Benefit Analysis of the IBM
    Computing on Demand (CoD) Cloud Offering

  • Azul's Experiences With Hardware / Software Co-Design by Dr. Cliff Click. Owning whole stack allows progress, Some really hard HW problems “solved” in SW, GC is “solved” w/HW Read Barrier, Simple HTM can do Lock Elision, Huge count of simple cores really useful in production.
  • Java Memory Problems - Memory problems in Java applications are manifold und easily lead to performance and scalability problems. Especially in J EE applications with a high number of parallel users memory management must be a central part of the application architecture.
  • Noob question: how do you [Reddit] join on so much data?
  • Transactional Memory versus Locks -
    A Comparative Case Study
    by Victor Pankratius. TM alone is no silver bullet.
  • Looking at Redis by Peter Zaitsev. With Redis I got about 3 times more updates/sec – close to 100.000 updates/sec with about 1.5 core being used.

    The fantasy sponsor for this post are those little food kiosks outside Home Depot stores. I love their Fire Dogs. Hot and yummy. I bet most home improvement projects in America are inspired by cravings for one of these little beauties.
  • Thursday

    Storage Systems for High Scalable Systems presentation

    The High Scalable Systems (i.e. Websites) such as: Google, Facebook, Amazon, etc. need high scalable storage system that can deal with huge amount of data with high availability and reliability. Building large systems on top of a traditional RDBMS data storage layer is no longer good enough. This presentation explores the landscape of new technologies available today to augment your data layer to improve performance and reliability.

    Remember: All of my presentations contents is open source, please feel free to use it, copy it, and re-distribute it as you want.

    Download the presentation


    Cheap storage: how backblaze takes matters in hand

    Blackblaze blogs about how they built their own storage infrastructure on the cheap to run their cloud backup service. This episode: the hardware.

    Sorry, just a link this time.


    Scaling MySQL on Amazon Web Services

    I've recently started working with a large company who is looking to take one of their heavily utilized applications and move it to Amazon Web Services. I'm not looking to start a debate on the merits of EC2, the decision to move to aws is already made (and is a much better decision than paying a vendor millions to host it).

    I've done my reasearch and I'm comfortable with creating this environment with one exception, scaling MySQL. I havent done much work with MySQL, i'm more of an Oracle guy up to now. I'm struggling to determine a way to scale MySQL on the fly in a way so that replication works, the server takes its proper place in line for master candidacy, and the apache servers become aware of it.

    So this is really three questions:

    1. What are some proven methods of load balancing the read traffic going from apache to MySQL.
    2. How do I let the load balancing mechanism know when I scale up / down a new Mysql Server?
    3. How to alert the master of the new server and initiate replication in an automated environment?

    Personally, I dont like the idea of scaling the databases, but the traffic increases exponentially for three hours a day, and then plummets to almost nothing. So this would provide a significant cost savings.

    The only way I've read to manage this sort of scaling I read here on slides 18-25:
    Has anyone tried this method and either had success or have scripts available to do this? I try not to remake the wheel when I dont have to. Thanks in advance.


    Squarespace Architecture - A Grid Handles Hundreds of Millions of Requests a Month 

    I first heard an enthusiastic endorsement of Squarespace streaming from the ubiquitous Leo Laporte on one of his many Twit Live shows. Squarespace as a fully hosted, completely managed environment for creating and maintaining a website, blog or portfolio was of interest to me because they promise scalability and this site doesn't have enough of that. But sadly, since they don't offer a link preserving Drupal import our relationship was not meant to be.

    When a fine reader of High Scalability, Brian Egge, (and all my readers are thrifty, brave, and strong) asked me how Squarespace scaled I said I didn't know, but I would try and find out. I emailed Squarespace a few questions and founder Anthony Casalena and Director of Technical Operations Rolando Berrios were kind enough to reply in some detail. The questions were both from Brian and myself. Answers can be found below.

    Two things struck me most about Squarespace's approach:

  • They based their system on a memory grid, in this case Oracle Coherence. I'm not aware of too many customer facing systems that have moved to a grid as the backbone of their scalability strategy. It's good to see a successful system visible out in the wild.
  • They use a sort of Private Cloud internally. Everything is highly automated and easy to expand. They scale by adding additional resources like CPUs and disks and the system just adapts without a lot of human fussing involved. Now that's scaling with gas.

    Learn more about how Squarespace has learned how to scale to tens of thousands of customers, hundreds of thousands of signups, and serve hundreds of millions of hits per month.


    The Stats

  • Tens of thousands of customers.
  • Hundreds of thousands of signups.
  • Serves hundreds of millions of hits per month.


  • Java - well supported and an advanced language to work in, and the components out there (Apache Foundation, etc.) are second to none.
  • Tomcat - the stability of the server is extremely impressive.
  • Grid - Oracle Coherence for the re-balancing and caching layers.
  • Storage - Isilon Cluster. This allows them to treat their storage like another "grid" as the storage pool is easily scaled by adding more diskspace.
  • Monetiziation Strategy - charge money. No free customers. Pricing starts at $8/month.
  • Uptime - 99.98%
  • Hosting - Peer1, they do not yet operate in multiple datacenters.
  • Competitors - TypePad and WordPress
  • Hardware - they don't use "commodity nodes" or low cost hardware units. These end up costing more in the long run as datacenter power is extremely expensive.
  • Cacti - a cacti instance is used to graph statistical data which helps see trends over time, predict when a hardware upgrade is necessary, and troubleshoot any problems that do show up.

    Lessons Learned

  • Cache as much as you can and load balance requests intelligently across a cluster.
  • Use an infrastructure that scales automatically merely by adding more resources (CPU, disk).
  • Build a scalable design up front. Make scaling easy by designing the application and infrastructure with scaling in mind.
  • Build a hands-off capable maintenance system. Automate processes. Make them as simple as possible. Monitor programatically so people don't have to.
  • Release code early and often. Running on the latest code means problems can be detected quickly when the problem are small.
  • Keep things simple. Apply simplicity to every part of your infrastructure, including both your software and those of your outside vendors. Examples of this are: Grid for the application infrastructure, Isilon cluster for storage, automation, creating their own tools.
  • Use as few technologies as possible by selecting or building simple, powerful and robust tools.
  • Don't be afraid to implement your own code to ensure simplicity. Build or buy is a huge balancing act.
  • Don't be afraid to spend money on technology that helps you get where you need to go. It can save you months and months of headaches that would have prevented you from working on core functionality.

    Interview Questions and Responses

    They say they run on a grid. I'd be interested to know if they built their own grid?

    Partially. We rely on Oracle's Coherence product for the re-balancing
    and caching layers of our system -- which we consider a real workhorse
    for the "grid" aspects of the system. Each node in our infrastructure
    can handle a hit for any single site on the system. This means that in order to increase capacity, we just increase node count. No site is handled by a single node.

    2. How much traffic they can really handle?

    We've had several customer sites on the front page of Digg on multiple
    occasions, and didn't notice any performance degradation for any of our
    sites. In fact, we didn't even realize the surge happened until we reviewed our traffic reports a few hours later. For 99% of sites out there, Squarespace is going to be sufficient. Even larger sites with millions of inbound hits per day are servable, as the bulk of the traffic serving on those sites is in the media being served.

    3. How do they scale up, and allow for certain sites to become quite busy?

    We've tried to make scaling easy, and the application and infrastructure
    have been designed with scaling in mind. Because of this, we're luckily not
    in a situation where we need to keep getting bigger and beefier hardware to handle more and more traffic -- we try to scale out by supplementing the
    grid. Since we try to cache as much as we can and every server
    participates in handling requests for every site, it's generally just a
    matter of adding another node to the environment.

    We try to apply this simplicity to every part of our infrastructure, both
    with our own software and when deciding on purchases from outside vendors. For instance, we just increased the amount of available storage another few terabytes by adding another node to our Isilon cluster.

    4. Are there any stats you can share about how many customers, how many users, how many requests served, how many servers, how much disk, how fast, how reliable?

    We, unfortunately, can't share these numbers as we're a private company
    -- but we can say we have tens of thousands of customers, hundreds
    of thousands of signups, and serve hundreds of millions of hits per
    month. The server types and disk configurations (RAID, etc) are a bit
    irrelevant, as the clustering we implement provides redundancy -- not
    anything implemented into a particular single machine. Nothing in
    hardware is too particular to our setup. I will say we don't purchase
    "commodity nodes" or other low cost hardware units, as we find these
    end up costing more in the long run as datacenter power is extremely

    5. What technology stack are you using and why did you make the choices you made?

    We currently use Java along with Tomcat as our web server. After
    trying a few other solutions, we really appreciated the ability to use
    as few technologies as possible, and have those always remain things
    that are understandable for us. Java is an incredibly well supported
    and advanced language to work in, and the components out there (Apache
    Foundation, etc.) are second to none. As for Tomcat, the stability of
    the server is extremely impressive. We've implemented our own
    controller mechanisms on top of Tomcat (instead of going with some
    other library) in order to ensure extreme simplicity.

    6. How are you handling...


    As mentioned above, every web node handles traffic for all sites, so a
    customer doesn't have to worry about an underpowered server unable to handle their traffic, or a node going down.


    Backups are obviously important to us, and we have several copies of user
    and server data stored in multiple locations. We gather backups with a
    combination of various home-grown scripts customized for our environment.

    Failover? Monitoring?

    Since this company originally was solely maintained by Anthony when he
    first started it, things needed to be as simple and automated as possible.
    This includes failover and monitoring. Our monitoring systems check every
    aspect of our environment we can think of several times a minute, and can
    restart obviously dead services, or alert us if it's something an
    actual person needs to handle.

    Additionally, we've set up a cacti instance to graph as much statistical
    data as we can pull out of our servers, so we can see trends over time.
    This allows us to easily predict when a hardware upgrade is necessary. It also helps us troubleshoot any problems that do show up.

    Operations? Releases? Upgrades? Add new hardware?

    With our customer base constantly growing, it's getting tough to manage our systems and still keep our workload under control. There are some projects on the road map to move to a much more hands-off maintenance of our environment, including automatic code deployments and system software upgrades. Most operations can be done without taking the grid offline.

    Multiple data centers?

    We do not have multiple data centers, but have some plans in the works to
    roll one out within the next year.


    This is a really broad question, so it's a bit hard to succinctly
    answer. One thing (amongst many) that has consistently served us very
    well is trying to ensure our development environment is always
    releasable into production. By ensuring we're always out there with
    our latest code, we can usually detect problems very rapidly, and
    as a result, those problems are generally extremely small. Everyone on our development team tends to be responsible for wide, sweeping aspects of the system -- which gives them a lot of flexibility to determine how
    their components should work as a whole. It's incredibly important
    that everything fits seamlessly together in the end, so we spend a lot
    of time iterating on things that other groups might consider finished.


    Support is something we take extremely seriously. As we've grown from
    the ground up without an external investor, most of our team members
    are versed in support, and understand how critical this component is.
    Our support staff is completely hired from our community, and is
    incredibly passionate about their jobs. We try and get every single
    customer support inquiry answered within 15 minutes or less, and have all sorts of metrics related to our goals here.

    7. What have you done that's really cool that you think other people could learn from?

    We spend a lot of time internally writing scripts and other
    applications that simply run our business. For instance, our
    persistence layer configuration files are generated by applications
    we've written that read our database model directly from the database.
    We develop a lot of these programs, and a lot of "standard naming"--this, again, means that we can move very rapidly as we have less monotonous tasks and searching to think about.

    While this sort of thing is appropriate for small tasks, for the big
    ones, we also aren't afraid to spend money on well developed
    technology. Some of our choices for load balancing and storage are
    very costly, but end up saving us months and months of time in the
    long haul, as we've avoided having to "put out fires" generated by
    untested home grown solutions. It's a huge balancing act.

    The End

    Often the best way to judge a product is to peruse the developer forums. It's these people who know what's really happening. And when I look I see an almost complete absence of threads about performance, scalability, or reliability problems. Take a look at other CMSs and you'll see a completely different tenor of questions. That says something good about the strength of their scalability strategy.

    I'd really like to thank Squarespace for taking the time and making the effort to share they've learned with the larger community. It's an effort we all benefit from. If you would also like to share your knowledge and wisdom with the world please get in touch and let's get started!

    Related Articles

  • Implementation Focus: Squarespace
  • Are Cloud Based Memory Architectures the Next Big Thing?
  • Up and running on Squarespace by Peter Efland
  • Kevin Rose Comes to Squarespace by D. Atkinson
  • Squarespace Vs Wordpress a thread in their developer forum.