Update 2: customer support is handled in real-time using Campfire.
Update: highly useful information on creating a customer billing system.
In the giving spirit of Christmas the folks at 37signals have shared a bit about how their system works. 37signals is most famous for loosing Ruby on Rails into the world and they've use RoR to make their very popular Basecamp, Highrise, Backpack, and Campfire products. RoR takes a lot of heat for being a performance dog, but 37signals seems to handle a lot of traffic with relatively normal sounding resources. This is just an initial data dump, they promise to add more details later. As they add more I'll update it here.
Site: http://www.37signals.com
Information Sources
Ask 37signals: Numbers?
Ask 37signals: How do you process credit cards?
Behind the scenes at 37signals: Support
Ask 37signals: Why did you restart Highrise?
Platform
Ruby on Rails
Memcached
Xen
MySQL
S3 for image storage
The Stats
30 servers ranging from single processor file servers to 8 CPU application servers for about 100 CPUs and 200GB of RAM.
Plan to diagonally scale by reducing the number of servers to 16 for about 92 CPU cores (each significantly faster than what are used today) and 230 GB of combined RAM.
Xen virtualization will be used to improve system management.
Basecamp (web based project management)
* 2,000,000 people with accounts
* 1,340,000 projects
* 13,200,000 to-do items
* 9,200,000 messages
* 12,200,000 comments
* 5,500,000 time tracking entries
* 4,000,000 milestones
Backpack (personal and small business information management)
* Just under 1,000,000 pages
* 6,800,000 to-do items
* 1,500,000 notes
* 829,000 photos
* 370,000 files
Overall storage stats (Nov 2007)
* 5.9 terabytes of customer-uploaded files
* 888 GB files uploaded (900,000 requests)
* 2 TB files downloaded (8,500,000 requests)
The Architecture
Memcached caching is used and they are looking to add more. Yields impressive performance results.
URL helper methods are used rather than building the URLs by hand.
Standard ActiveRecord built queries are used, but for performance reasons they will also "dig in and use" find_by_sql when necessary.
They fix Rails when they run into performance problems. It pays to be king :-)
Amazon’s S3 is used for storage of files upload by users. Extremely happy with results.
Credit Card Processing Process
Bill monthly. It makes credit card companies more comfortable because they won't be on the hook for a large chunk of change if your company goes out of business. Customers also like it better because it costs less up front and you don't need a contract. Just pay as long as you want the service.
Get a Merchant Account. One is needed to process credit cards. They use Chase Bank. Use someone you trust and later negotiate rates when you get enough volume that it matters.
Authorize.net is the gateway they use to process the credit card charge.
A custom built system handles the monthly billing. It runs each night and bills the appropriate people and records the result.
On success an invoice is sent via email.
On failure an explanation is sent to the customer.
If the card is declined three times the account is frozen until a valid card number is provided.
Error handling is critical because problems with charges are common. Freeze to fast is bad, freezing too slow is also bad.
All products are being converted to using a centralized billing service.
You need to be PCI DSS (Payment Card Industry Data Security Standard) compliant.
Use a gateway service that makes it so you don't have to store credit card numbers on your site. That makes your life easier because of the greater security. Some gateway services do have reoccurring billing so you don't have to do it yourself.
Customer Support
Campfire is used for customer service. Campfire is a web-based group chat tool, password-protectable, with chatting, file sharing, image previewing, and decision making.
Issues discussed are used to drive code changes and the subversion commit is shown in the conversation. Seems to skip a bug tracking system, which would make it hard to manage bugs and features in any traditional sense, ie, you can't track subversion changes back to a bug and you can't report what features and bugs are in a release.
Support can solve problems by customers uploading images, sharing screens, sharing files, and chatting in real-time.
Developers are always on within Campfire addressing problems in real-time with the customers.
Lessons Learned
Take a lesson from Amazon and build internal functions as services from the start. This make it easier to share them across all product lines and transparently upgrade features.
Don't store credit card numbers on your site. This greatly reduces your security risk.
Developers and customers should interact in real-time on a public forum. Customers get better service as developers handle issues as they come up in the normal flow of their development cycle. Several layers of the usual BS are removed. Developers learn what customers like and dislike which makes product development more agile. Customers can see the responsiveness of the company to customers by reading the interactions. This goes a long ways to give potential customers the confidence and the motivation to sign up.
Evolve your software by actual features needed by users instead of making up features someone might need someday. Otherwise you end up building something that nobody wants and won't work anyway.
Comments
Re: 37signals Architecture
very nice! other companies could release some of their internal IT info/structure as well ,)
Re: 37signals Architecture
You don't mention what kind of load balancer you use.
Also, does Mongrel handle requests directly or do you have a proxy in front of it? (Perhaps doing load balancing?)
Re: 37signals Architecture
The architecture looks ok. But the numbers aren't impressive...
I would expect that the number of their servers is due to low speed of ruby in generating content.
And by the looks of it, they use a lot of S3 that means that they have unloaded a lot of file storage to Amazon infra.
Good luck to them with expanding without expansion :)
Re: 37signals Architecture
What kind of db servers are they using?
Is it one db or one db per account.
Re: 37signals Architecture
Great article, keep up the good work.
----------------------------------------------------
برامج نت|free software|افضل المواقع العربية|منتدى
برامج نت|العاب فلاش - العاب بنات|برامج|دليل
المواقع|عيادة طب |
الأرشيف|برامج مشروحة|برامج ترجمة|برامج الفاكس|برامج طباعة|برامج تحرير|برامج التقاط الصور والشاشات|برامج سطح المكتب|برامج البريد الالكتروني|برامج خدمات البريد الاكتروني|برامج القوائم البريدية|برامج ادوات البريد الاكتروني|برامج مكافحة الرسائل المزعجة|برامج الإنترنت|برامج مشاهده القنوات الفضائيه|برامج تسربع الانترنت|برامج تحميل الملفات والصور|برامج المحادثة|برامج ماسنجر|اتصال دولي - الرسائل القصيره|إدوات خدمية وتعاريف قطع جهاز|برامج نسخ الأحتياطي|ادارة الملفات|تقارير الاداء|ضغط وفك ضغط الملفات|الصيانة والمعالجة|ادارة النظام|برامج تحرير الذاكره|الحفظ الاحتياطي|برامج الاداره والتحكم|برامج شبكات|برامج الحماية|برامج مكافحة الفايروسات|مكافحة ملفات التجسس|برامج صد الهاكرز والمخترقين|برامج تشفير ملفات|اخر تحديثات|برامج الرسوم والتصاميم|برامج الفلاش|برامج تحرير الصور|برامج استعراض الصور|برامج ادوات الصور الرقمية|برامج تحويل صور|برامج التقاط الصور|اضافات الصور|برامج جوالات سوني أريكسون
Re: 37signals Architecture
Interesting...
Do you know what each of their 30 servers do in details and how do they backup the huge amount of data in their databases?
Re: 37signals Architecture
Is it one db or one db per account.