« Sponsored Post: Apple, Couchbase, Farmerswife, VividCortex, Internap, SocialRadar, Campanja, Transversal, MemSQL, Scalyr, FoundationDB, AiScaler, Aerospike, AppDynamics, ManageEngine, Site24x7 | Main | Stuff The Internet Says On Scalability For January 30th, 2015 »

Marco Arment Uses Go Instead of PHP and Saves Money by Cutting the Number of Servers in Half

On the excellent Accidental Tech Podcast there's a running conversation about Marco Arment's (Tumblr, Instapaper) switch to Go, from a much loved PHP, to implement feed crawling for Overcast, his popular podcasting app for the iPhone.

In Episode 101 (at about 1:10) Marco said he halved the number of servers used for crawling feeds by switching to Go. The total savings was a few hundred dollars a month in server costs.

Why? Feed crawling requires lots of parallel networking requests and PHP is bad at that sort of thing, while Go is good at it. 

Amazingly, Marco wrote an article on how much Overcast earned in 2014. It earned $164,000 after Apple's 30%, but before other expenses. At this revenue level the savings, while not huge in absolute terms given the traffic of some other products Marco has worked on, was a good return on programming effort. 

How much effort? It took about two months to rewrite and debug the feed crawlers. In addition, lots of supporting infrastructure that tied into the crawling system had to be created, like the logging infrastructure, the infrastructure that says when a feed was last crawled, monitoring delays, knowing if there's queue congestion, and forcing a feed to be crawled immediately.

So while the development costs were high up front, as Overcast grows the savings will also grow over time as efficient code on fast servers can absorb more load without spinning up more servers.

Lots of good lessons here, especially for the lone developer:

Attack the hot spot. Go was not used for the entire project, much of which is still in PHP. Go was used to attack one hot spot, a major bottleneck that was a huge resource hog. As John Siracusa said on the show, "Put your effort towards the thing that gives you the biggest bang for the buck." 

Polyglot programming. Marco is showing the pattern of specializing functions over time. First, make it work, then see what's happening, then make smart changes. Use PHP where it works best, use Go where it works best,  and maybe in the future move to 3rd party products as scale, features and temperament dictate. 

Keep an open mind. Marco thinks for most web app problems other languages aren't better enough at solving problems to make them worth learning when compared to PHP. Go is a different sort of language and it's good at parallelism and concurrency, which makes it a valuable tool to have your tool belt for certain kinds of problems.

Festoon your tool belt. Marco also talked about how there's a benefit to just learning another language. He now has a valuable tool in his tool belt for certain kinds of problems, that he knows and trusts well enough to use again.

Reader Comments (4)

I heard that podcast and I disagreed with his approach at a high level for a couple reasons.

1) PubSubHubbub is better than polling
2) There are existing services (superfeedr) that you don't need to two months to write, that will have a lower operating cost going forward.

Just my 2 cents.

February 2, 2015 | Unregistered CommenterTom Schneider

"Why? Feed crawling requires lots of parallel networking requests and PHP is bad at that sort of thing, while Go is good at it. "

It's wrong. There are the solutions. ie: pthreads extension, or Facebook's Hack Language async feature which can be used with a few changes. And yes, PHP code is able to be run under Hack: <?php -> <?hh ..

Too many developers think that PHP is bad language. But until yet, I never saw that you can do X job with Y language, but can't do that job with PHP. PHP is not like in it's old days. PHP is based on C language, so if you can code in C you can write extensions to solve specific problems. So, it can be much more expensive to hire programmers for Go, and hard to find experienced workers, while many good PHP programmers out there who know PHP internals. Another thing about PHP, with HHVM; server costs can be minimizied. So, I think PHP is just good in many cases. But honestly, if I'm going to write a Service, probably I will go with "Go" and still PHP if I'm going to write a website (even for sites that handle heavy traffic).

The aboves are 'my' opinions.

February 2, 2015 | Unregistered Commenterno-author

One of the fun things from following the process on his Twitter feed is that picking up the language itself was apparently pretty quick--one day he was posting about finishing his first Go program, and not long after he was listing the features he'd added to his poller that day. Of course, just being able to do get some things done on first project doesn't mean you're immediately familiar/comfortable with the nooks and crannies of a tool. Still, it's worth remembering that sometimes it works out when you just dive in, and if it doesn't work out, it might be an acceptable risk.

Maybe related to how quickly he learned it, he flagged Go's relative lack of clever features as something he liked.

February 3, 2015 | Unregistered CommenterR

Using a compiled language instead of an interpreted one reduces resource consumption and/or runs faster.

See also:

News at 11.

February 3, 2015 | Unregistered CommenterChris Westin

PostPost a New Comment

Enter your information below to add a new comment.
Author Email (optional):
Author URL (optional):
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>