advertise
Saturday
Jul282007

Product: FastStats Log Analyzer 

FastStats Log Analyzer enables you to: * Determine whether your CPC advertising is profitable: Are you spending $0.75 per click on Google or Overture, but only receiving $0.56 per click in revenue? * Tune site traffic patterns: FastStats's Hyperlink Tree View feature lets you visually see how traffic flows through your web site. * High-performance solution for even the busiest web sites: Our software has been clocked at over 1000 MB/min. Other popular log file analysis tools (we won't name names), run at 1/40th the speed. We've been in the business for over 6 years, delivering value, quality, and good customer service to our clients. Our products are used for data mining at some of the world's busiest web sites -- why not give FastStats a try at your web site? FastStats log file analysis supports a wide variety of web server log files, including Apache logs and Microsoft IIS logs.

Click to read more ...

Saturday
Jul282007

Product: Web Log Expert

WebLog Expert is a fast and powerful access log analyzer. It will give you information about your site's visitors: activity statistics, accessed files, paths through the site, information about referring pages, search engines, browsers, operating systems, and more. The program produces easy-to-read HTML reports that include both text information (tables) and charts. View the WebLog Expert sample report to get the general idea of the variety of information about your site's usage it can provide. WebLog Expert can analyze logs of Apache and IIS web servers. It can even read GZ and ZIP compressed logs so you won't need to unpack them manually. The log analyzer features intuitive interface. Built-in wizards will help you quickly and easily create a profile for your site and analyze it.

Click to read more ...

Friday
Jul272007

Product: Munin Monitoriting Tool

Munin the monitoring tool surveys all your computers and remembers what it saw. It presents all the information in graphs through a web interface. Its emphasis is on plug and play capabilities. After completing a installation a high number of monitoring plugins will be playing with no more effort. Using Munin you can easily monitor the performance of your computers, networks, SANs, applications, weather measurements and whatever comes to mind. It makes it easy to determine "what's different today" when a performance problem crops up. It makes it easy to see how you're doing capacity-wise on any resources.

Click to read more ...

Thursday
Jul262007

Product: eAccelerator a PHP Accelerator

eAccelerator is a free open-source PHP accelerator, optimizer, and dynamic content cache. It increases the performance of PHP scripts by caching them in their compiled state, so that the overhead of compiling is almost completely eliminated. It also optimizes scripts to speed up their execution. eAccelerator typically reduces server load and increases the speed of your PHP code by 1-10 times.

Click to read more ...

Thursday
Jul262007

Product: Symfony a Web Framework

symfony is an open-source PHP web framework based on the best practices of web development, thoroughly tried on several active websites, symfony aims to speed up the creation and maintenance of web applications, and to replace the repetitive coding tasks by power, control and pleasure. Symfony provides a lot of features seamlessly integrated together, such as: * simple templating and helpers * cache management * smart URLs * scaffolding * multilingualism and I18N support * object model and MVC separation * Ajax support * enterprise ready

Click to read more ...

Thursday
Jul262007

ThemBid Architecture

ThemBid provides a market where people needing work done broadcast their request and accept bids from people competing for the job. Unlike many of the sites profiled at HighScalability, ThemBid is not in the popular press as often as Paris Hilton. It's not a media darling or a giant of the industry. But what I like is they have a strategy, a point-of-view for building websites and were gracious enough to share very detailed instructions on how to go about building a website. They even delve into actual installation details of the various software packages they use. Anyone can benefit by taking a look at their work. Site: http://www.thembid.com/

Information Sources

  • Build Scalable Web 2.0 Sites with Ubuntu, Symfony, and Lighttpd

    Platform

  • Linux (Ubuntu)
  • Symfony
  • Lighttpd
  • PHP
  • eAccelerator
  • Eclipse
  • Munin
  • AWStats

    What's Inside?

    The Stats

  • Started work in December of 2006 and had a full demo by March 2007.
  • One developer/sys admin worked with a part-time graphics designer.
  • Targeted a few thousand users after launch.

    The Architecture

  • Hardware. Dual core server with 2GB RAM
  • Storage. 2 x 36SCSI 10K RPM on RAID1.
  • Data Center. They went with with Layeredtech for the managed server because of past positive experiences.
  • Development Environment. Ubuntu and Eclipse.
  • OS. They chose the server distribution of Ubuntu because that's what they use on the client side and Ubuntu supports "simpler installation and easier maintenance than typical IT deployments."
  • Web Server. Lighttpd is used to handle static content and forward the dynamic PHP page requests to FastCGI.
  • Database. MySQL. When growth is necessary the idea is to move to a master-slave arrangement and them maybe MySQL cluster.
  • Web Framework. Went with PHP because they knew it and other successful sites like Digg and Yahoo successfully deploy PHP. They chose Symfony as there framework because of its nice documentation and active development community. And Yahoo also uses Symfony. It's a decision that has worked well for them.
  • PHP Cache. eAccelerator is used to compile and cache PHP scripts.
  • Object and Content Cache. The plan is to cache a lot of content. For a bid site like theirs this makes sense. Many of the pieces are used over and over again so putting them in memory will speed up the entire system and take pressure off the database and the IO system. Initially the used a SQLite cache on top of of a memory based file system. This choice was because it was supported by Symfony. When a memcached plugin is available they'll try that.
  • Client Side Cache. Lighttp's mod_expire module is used to prevent Javascript, style sheets, and images that rarely change from being uncessarily redownloaded by the browser.
  • Monitoring. Munin is used to monitor their resource usage. It's as simple as visiting "yoursite.com/status" to see what's going on.
  • Log Analysis. AWStats is used to track hits and types of requests. This information can be used to target bottlenecks.
  • Scalability Plan. - Use Munin to tell when to think about upgrading. When your growth trend will soon cross your resources trend, it's time to do something. - Move MySQL to a separate server. This frees up resources (CPU, disk, memory). What you want to run on this server depend on its capabilities. Maybe run a memcached server on it. - Move to a distributed memory cache using memcached. - Add a MySQL master/slave configuration. - If more webservers are needed us LVS on the front end as a load balancer.
  • Future Directions. Work on fault tolerance.

    Lessons Learned

  • It's possible to create a nice site fairly quickly with just a few people using commonly available low cost tools. And your system will be solid and powerful. No cut corners.
  • Use feedback from your system to know what needs optimizing and when it's time to scale.
  • Good documentation and an active community draw people. These are very attractive qualities for people making decisions about what to use. It's hard to go with a tool chain when it looks like you may get stuck in the future with no way out and no help. If you make tools make them dead easy to understand, learn, use, and deploy.
  • Stick with the familiar. It may not be optimal, it may not be the best, but it's more important that you get started and make progress. You don't want to delay releasing your site so you can learn a completely different tool chain that may make your life somewhat easier and in some projected future. The future is now.
  • Use what works for other people. The fact that Yahoo and Digg use PHP is a good recommendation. Certainly PHP is not the only way to build a site, but it does cut your risk level and help you sleep at night. It also means there's an active community that can help you when you have problems.

    Click to read more ...

  • Thursday
    Jul262007

    Product: AWStats a Log Analyzer

    AWStats is a free powerful and featureful tool that generates advanced web, streaming, ftp or mail server statistics, graphically. This log analyzer works as a CGI or from command line and shows you all possible information your log contains, in few graphical web pages. It uses a partial information file to be able to process large log files, often and quickly. It can analyze log files from all major server tools like Apache log files (NCSA combined/XLF/ELF log format or common/CLF log format), WebStar, IIS (W3C log format) and a lot of other web, proxy, wap, streaming servers, mail servers and some ftp servers.

    Click to read more ...

    Wednesday
    Jul252007

    Product: 3 PAR REMOTE COPY

    3PAR Remote Copy is a uniquely simple and efficient replication technology that allows customers to protect and share any application data affordably. Built upon 3PAR Thin Copy technology, Remote Copy lowers the total cost of storage by addressing the cost and complexity of remote replication. Common Uses of 3PAR Remote Copy: Affordable Disaster Recovery: Mirror data cost-effectively across town or across the world. Centralized Archive: Replicate data from multiple 3PAR InServs located in multiple data centers to a centralized data archive location. Resilient Pod Architecture: Mutually replicate tier 1 or 2 data to tier 3 capacity between two InServs (application pods). Remote Data Access: Replicate data to a remote location for sharing of data with remote users.

    Click to read more ...

    Wednesday
    Jul252007

    Product: NetApp MetroCluster Software

    NetApp MetroCluster Software Cost-effective is an integrated high-availability storage cluster and site failover capability. NetApp MetroCluster is an integrated high-availability and disaster recovery solution that can reduce system complexity and simplify management while ensuring greater return on investment. MetroCluster uses clustered server technology to replicate data synchronously between sites located miles apart, eliminating data loss in case of a disruption. Simple and powerful recovery process minimizes downtime, with little or no user action required. At one company I worked at they used the NetApp snap mirror feature to replicate data across long distances to multiple datacenters. They had a very fast backbone and it worked well. The issue with NetApp is always one of cost, but if you can afford it, it's a good option.

    Click to read more ...

    Wednesday
    Jul252007

    Paper: Designing Disaster Tolerant High Availability Clusters

    A very detailed (339 pages) paper on how to use HP products to create a highly available cluster. It's somewhat dated and obviously concentrates on HP products, but it is still good information. Table of contents: 1. Disaster Tolerance and Recovery in a Serviceguard Cluster 2. Building an Extended Distance Cluster Using ServiceGuard 3. Designing a Metropolitan Cluster 4. Designing a Continental Cluster 5. Building Disaster-Tolerant Serviceguard Solutions Using Metrocluster with Continuous Access XP 6. Building Disaster Tolerant Serviceguard Solutions Using Metrocluster with EMC SRDF 7. Cascading Failover in a Continental Cluster Evaluating the Need for Disaster Tolerance What is a Disaster Tolerant Architecture? Types of Disaster Tolerant Clusters Extended Distance Clusters Metropolitan Cluster Continental Cluster Continental Cluster With Cascading Failover Disaster Tolerant Architecture Guidelines Protecting Nodes through Geographic Dispersion Protecting Data through Replication Using Alternative Power Sources Creating Highly Available Networking Disaster Tolerant Cluster Limitations Managing a Disaster Tolerant Environment Using this Guide with Your Disaster Tolerant Cluster Products 2. Building an Extended Distance Cluster Using ServiceGuard Types of Data Link for Storage and Networking Two Data Center Architecture Two Data Center FibreChannel Implementations Advantages and Disadvantages of a Two-Data-Center Architecture Three Data Center Architectures Rules for Separate Network and Data Links Guidelines on DWDM Links for Network and Data 3. Designing a Metropolitan Cluster Designing a Disaster Tolerant Architecture for use with Metrocluster Products Single Data Center Two Data Centers and Third Location with Arbitrator(s) Additional EMC SRDF Configurations Setting up Hardware for 1 by 1 Configurations Setting up Hardware for M by N Configurations Worksheets Disaster Tolerant Checklist Cluster Configuration Worksheet Package Configuration Worksheet Next Steps 4. Designing a Continental Cluster Understanding Continental Cluster Concepts Mutual Recovery Configuration Application Recovery in a Continental Cluster Monitoring over a Wide Area Network Cluster Events Interpreting the Significance of Cluster Events How Notifications Work Alerts Alarms Creating Notifications for Failure Events Creating Notifications for Events that Indicate a Return of Service Performing Cluster Recovery Notes on Packages in a Continental Cluster How Serviceguard commands work in a Continentalcluster Designing a Disaster Tolerant Architecture for use with Continentalclusters Mutual Recovery Serviceguard Clusters Data Replication Highly Available Wide Area Networking Data Center Processes Continentalclusters Worksheets Preparing the Clusters Setting up and Testing Data Replication Configuring a Cluster without Recovery Packages Configuring a Cluster with Recovery Packages Building the Continentalclusters Configuration Preparing Security Files Creating the Monitor Package Editing the Continentalclusters Configuration File Checking and Applying the Continentalclusters Configuration Starting the Continentalclusters Monitor Package Validating the Configuration Documenting the Recovery Procedure Reviewing the Recovery Procedure Testing the Continental Cluster Testing Individual Packages Testing Continentalclusters Operations Switching to the Recovery Packages in Case of Disaster Receiving Notification Verifying that Recovery is Needed Using the Recovery Command to Switch All Packages How the cmrecovercl Command Works Forcing a Package to Start Restoring Disaster Tolerance Restore Clusters to their Original Roles Primary Packages Remain on the Surviving Cluster Primary Packages Remain on the Surviving Cluster using cmswitchconcl Newly Created Cluster Will Run Primary Packages Newly Created Cluster Will Function as Recovery Cluster for All Recovery Groups Maintaining a Continental Cluster Adding a Node to a Cluster or Removing a Node from a Cluster Adding a Package to the Continental Cluster Removing a Package from the Continental Cluster Changing Monitoring Definitions Checking the Status of Clusters, Nodes, and Packages Reviewing Messages and Log Files Deleting a Continental Cluster Configuration Renaming a Continental Cluster Checking Java File Versions Next Steps Support for Oracle RAC Instances in a Continentalclusters Environment Configuring the Environment for Continentalclusters to Support Oracle RAC Initial Startup of Oracle RAC Instance in a Continentalclusters Environment Failover of Oracle RAC Instances to the Recovery Site Failback of Oracle RAC Instances After a Failover 5. Building Disaster-Tolerant Serviceguard Solutions Using Metrocluster with Continuous Access XP Files for Integrating XP Disk Arrays with Serviceguard Clusters Overview of Continuous Access XP Concepts PVOLs and SVOLs Device Groups and Fence Levels Creating the Cluster Preparing the Cluster for Data Replication Creating the RAID Manager Configuration Defining Storage Units Configuring Packages for Disaster Recovery Completing and Running a Metrocluster Solution with Continuous Access XP Maintaining a Cluster that uses Metrocluster/CA XP/CA Device Group Monitor Completing and Running a Continental Cluster Solution with Continuous Access XP Setting up a Primary Package on the Primary Cluster Setting up a Recovery Package on the Recovery Cluster Setting up the Continental Cluster Configuration Switching to the Recovery Cluster in Case of Disaster Failback Scenarios Maintaining the Continuous Access XP Data Replication Environment 6. Building Disaster Tolerant Serviceguard Solutions Using Metrocluster with EMC SRDF Files for Integrating ServiceGuard with EMC SRDF Overview of EMC and SRDF Concepts Preparing the Cluster for Data Replication Installing the Necessary Software Building the Symmetrix CLI Database Determining Symmetrix Device Names on Each Node Building a Metrocluster Solution with EMC SRDF Setting up 1 by 1 Configurations Grouping the Symmetrix Devices at Each Data Center Setting up M by N Configurations Configuring Serviceguard Packages for Automatic Disaster Recovery Maintaining a Cluster that Uses Metrocluster/SRDF Managing Business Continuity Volumes R1/R2 Swapping Building a Continental Cluster Solution with EMC SRDF Setting up a Primary Package on the Primary Cluster Setting up a Recovery Package on the Recovery Cluster Setting up the Continental Cluster Configuration Switching to the Recovery Cluster in Case of Disaster Failback Scenarios Maintaining the EMC SRDF Data Replication Environment R1/R2 Swapping 7. Cascading Failover in a Continental Cluster Overview Symmetrix Configuration Using Template Files Data Storage Setup Setting Up Symmetrix Device Groups Setting up Volume Groups Testing the Volume Groups Primary Cluster Package Setup Recovery Cluster Package Setup Continental Cluster Configuration Data Replication Procedures Data Initialization Procedures Data Refresh Procedures in the Steady State Data Replication in Failover and Failback Scenarios

    Click to read more ...