advertise
Friday
Feb122016

Stuff The Internet Says On Scalability For February 12th, 2016


Maybe this year's Mavericks can ride some gravitational waves?  

 

If you like this sort of Stuff then please consider offering your support on Patreon.

  • 3.96 Million: viewers streaming the Super Bowl; 1000 kilometers: roads made of solar panels in France; 500mg: amount of chlorophyll absorbing photons in a tree;

  • Quotable Quotes:
    • cmyr: soundcloud actually represents a very important cultural document; there is tons of music that has been created in the past 5+ years that exists exclusively there, and it would be a tremendous cultural loss if it were to disappear. 
    • aback: we will continue to endure substantial cultural losses for so long as people continue to believe that content can & should be distributed and consumed for free.
    • @dotemacs: - We’re moving to Java + Spring. - Why? - Cos of threads… and scaling… - OK … what exactly…? - I’m just going by what I was told.
    • @asolove: Chaos Monkey People: every week, one randomly-selected person must take the whole week off regardless of their current work. Org must adapt.
    • @waynejwerner: we almost started using them, but couldn't find examples of not Google / Netflix using containers in production
    • Oisin Hanrahan: Scaling is a never ending process of revisiting the same challenges at different volume points and learning from each and every one so that we get better and better. 
    • @nickcalyx: Q: What's the difference between USA and USB? A: One connects to all your devices & accesses your data, and the other is a hardware standard
    • @mathewi: In an alternate universe, Twitter never went public and is a profitable real-time news utility with an open API and multiple revenue streams
    • @etherealmind: “QOS in the Internet is like equipping fish with tricycles” - Geoff Huston - I laughed so hard. (podcast out later today)
    • Jef Akst: Plants may trick bacteria into attacking before the microbial population reaches a critical size, allowing the plants to successfully defend the weak invasion.
    • @CapgeminiIndia: Every uber car replaces 9 personal cars, digital innovation has a wider impact than you think. - Noshir Kaka #Nasscom_ILF
    • @EmperiorEric: CloudKit is fantastic, and its scaling tiers are not only amazing when free, but extremely cheap when not. But daily bandwidth scales slow.
    • Palmer Luckey: If a year from now we can have the people who buy into VR [Oculus] using their headsets every day, every week, coming back to it regularly, that’s what makes VR a huge success
    • Fister: all of the archives in the world could be stored in one box of seeds.
    • Annisa Cinderakasih: I really like the idea walking through the park/botanical garden which actually a ‘data library’, while I can listen to the narration of its data inside just by touching it.
    • @swardley: The time of war is upon us in gaming and many components of the gaming value chain are ripe for shifting from product to commodity (+utility) forms. 
    • @joyent: Q: "Are we at peak confusion in containers?" A: "5 more schedulers and we’re there." - @frazelledazzell from @docker at #ContainerSummit
    • @capotej: scala career timeline: year 1: dope, gonna write some terse code year 2: hmm, maybe i shouldnt use every feature year 3: java 8 looks nice
    • @skamille: OH: "I thought cloud native referred to engineers born after AWS launched?
    • AaronLasseigne: It's [Snapchat] a video game for her. She doesn't look at the photos in the morning, she just responds to keep the chain going. It's like grinding. She has a score, tries to improve it, get trophies, new equipment (I mean filters), etc.
    • @kentbye: Google will put out a Project Tango phone in 2016 with 3D depth sensors. It'll bootstrap consumer AR at mobile scale 
    • @pas256: Google Cloud Functions are completely different to AWS Lambda: exports.fn = function(context, data) vs exports.fn = function(event, context)
    • medicine.ai: The Next Miracle Drug is an Algorithm.
    • @viktorklang: "The secret to very responsive systems is to keep the utilization down." - @mjpt777 (from Queuing Theory)
    • Mark Anderson: The cloud and the Internet of Things will likely provide plenty of snooping opportunities for the agency and others like it.
    • @WhatTheFFacts: It would take you 10 years to view all the photos shared on Snapchat in the last hour.
    • @kevinmarks: government should not be an app, but a protocol. Unauthenticated read, POST with nuance
    • @noggin143: the cern openstack cloud at 155K cores has no proprietary extensions IMHO.
    • @beaucronin: The VR strategy chess game is getting very serious 
    • @HoustonTexasVR: Blows my mind @Amazon can buy a game dev engine & release it free, no royalties just to promote uptake of its web services / cloud platform.
    • Kieren McCarthy: We're going to use your toothbrush to snoop on you, says US spy boss
    • Steve Ranger: In a surveillance economy, privacy represents an opportunity for profit forgone.
    • sudovoodoo: Should say that we horizontally scale this [socketcluster.io] thing pretty heavily using the sc-redis module. Elasticache + ELB + 4 EC2 Instances = support for 5000+ person conferences :D
    • There are more Quotable Quotes in the full article.

  • Netflix has completed its 7 year long odyssey of moving all their operations to the cloud. Completing the Netflix Cloud Migration. It's really a love story. Netflix has grown their streaming membership 8x from 2008 and overall viewing has grown by three orders of magnitude. To support this growth they could not have racked servers fast enough in their own datacenter. Nor could they have grown world wide to support 130 new countries. Reliability is up, approaching four nines. Costs are down, costs per streaming start ended up being a fraction of those in the datacenter. Cloud elasticity is the driver for reduced costs. It's possible to continuously optimize instance type mix and to grow and shrink their footprint near-instantaneously without the need to maintain large capacity buffers. And on their voyage they've taken us all with them, teaching everyone what it means to operate at scale in the cloud. Netflix Open Source

  • 1 billion Apple devices are in active use around the world. Interesting metric, to talk about the number of devices instead of the number of users. Apple is saying we may not be able to grow users as fast as we used to, let's count devices instead, that we can grow, and BTW, we plan on making a lot more devices to sell to a loyal customer base. Also Wall Street, that's a pretty big market to sell accessories and services into. We're not dead yet.

  • Meta fatigue, there's no pill for it, but JavaScript fatigue fatigue offers tips against feeling overwhelmed: Don’t try to know everything; Wait for the critical mass; Stick to things you understand: don’t use more than 1–2 new technologies per project; Do exploratory toy projects.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Wednesday
Feb102016

How to build your Property Management System integration using Microservices

This is a guest post by Rafael Neves, Head of Enterprise Architecture at ALICE, a NY-based hospitality technology startup. While the domain is Property Management, it's also a good microservices intro.

In a fragmented world of hospitality systems, integration is a necessity. Your system will need to interact with different systems from different providers, each providing its own Application Program Interface (API). Not only that, but as you integrate with more hotel customers, the more instances you will need to connect and manage this connection. A Property Management System (PMS) is the core system of any hotel and integration is paramount as the industry moves to become more connected.

 

To provide software solutions in the hospitality industry, you will certainly need to establish a 2-way integration with the PMS providers. The challenge is building and managing these connections at scale, with multiple PMS instances across multiple hotels. There are several approaches you can leverage to implement these integrations. Here, I present one simple architectural design to building an integration foundation that will increase ROI as you grow. This approach is the use of microservices.

What are microservices? 

Click to read more ...

Tuesday
Feb092016

A Smallish List of Parse Migration Guides

Since Parse's big announcement it looks like the release of migration guides from various alternative services has died down. 

The biggest surprise is the rise of Parse's own open source Parse Server. Check out its commit velocity on GitHub. It seems to be on its way to becoming a vibrant and viable platform.

The immediate release of Parse Server with the announcement of the closing of Parse was surprising. How could it be out so soon? That's a lot of work. Some options came to mind. Maybe it's a version of an on-premise system they already had in the works? Maybe it's a version of the simulation software they use for internal testing? Or maybe they had enough advanced notice they could make an open source version of Parse? 

The winner is...

Charity Majors, formerly of Parse/Facebook, says in How to Survive an Acquisition, tells all:

Massive props to Kevin Lacker and those who saw the writing on the wall and did an amazing job preparing to open up the ecosystem.

That's impressive. It seems clear the folks at Parse weren't on board with Facebook's decision, but they certainly did everything possible to make the best out of a bad situation. It's even possible this closure could be a good thing for Parse in the long run if open source support continues to flourish.

Here's a list of different ways of getting from here to there...

Migration Guides

Click to read more ...

Monday
Feb082016

What's Next? The NFL's Magic Yellow Line Shows the Way to Augmented Reality

 

Update: Amazon just released Lumberyard, a free AAA game engine deeply integrated with AWS & Twitch.

What’s next? Mobile is entering its comforting middle age period of development. Conversational commerce is a thing, a good thing, but is it really a great thing?

What’s next may be what has been next for decades: Augmented reality (AR) (and VR). AR systems will be here sooner than you might think. A matter of years, not decades. Robert Scoble, for example, thinks Meta, an early startup in AR industry, will be bigger than the Macintosh. More on that in a later post. Magic Leap has no product and $1.3 billion in funding. Facebook has Oculus. Microsoft has HoloLens. Google may be releasing a VR system later this year. Apple is working on VR. Becoming the next iPhone is up for grabs.

AR is a Huge Opportunity for Programmers and Startups 

This is a technological revolution that will be bigger than mobile. Opportunities in mobile for developers have largely played out. Experience shows the earlier you get in on a revolution the better the opportunity will be. Do you want to be writing free iOS apps forever?

It’s so early we don’t really have an idea what AR is or what the market will be or what it means from a developer perspective. But if you watched the Super Bowl you saw an early example of the power of AR. It’s the benign looking, yet technically impressive, computer generated yellow first down line marker.

Augmented Reality is Already a Sports Reality

Click to read more ...

Friday
Feb052016

Stuff The Internet Says On Scalability For February 5th, 2016


We have an early entry for the best vacation photo of the century. 

 

If you like this sort of Stuff then please consider offering your support on Patreon.
  • 1 billion: WhatsApp users; 3.5 billion: Facebook users in 2030; $3.5 billion: art sold online; $150 billion: China's budget for making chips; 37.5MB: DNA information in a single sperm; 

  • Quotable Quotes:
    • @jeffiel: "But seriously developers, trust us next time your needs temporarily overlap our strategic interests. And here's a t-shirt."
    • @feross: Modern websites are the epitome of inefficiency. Using giant multi-MB javascript files to do what static HTML could do in 1999.
    • Rob Joyce (NSA): We put the time in …to know [that network] better than the people who designed it and the people who are securing it,' he said. 'You know the technologies you intended to use in that network. We know the technologies that are actually in use in that network. Subtle difference. You'd be surprised about the things that are running on a network vs. the things that you think are supposed to be there.
    • @MikeIsaac: i just realized how awkward Facebook's f8 conference is gonna be this year
    • @Nick_Craver: Stats correction: Stack Overflow did 157,370,800,409 redis ops in the past 30 days, almost always under 2% CPU:
    • @BenedictEvans: The global SMS system does around 20bn messages a day. WhatsApp is now doing 42bn. With 57 engineers.
    • @jaygoldberg: WhatsApp has the benefit of running on top of the world's data networks which employ a few more engineers... 
    • @anildash: It’s odd that developers think Twitter is so hostile while Facebook shuts down stuff like Parse & FBML + cuts back the Instagram & FB APIs.
    • @asynchio:  I use to think CEP = stateful business rules engine + inference + stream processing. Has it changed?
    • @Marco_Rasp: "SOA is about reuse, MicroServices about time to market." @samnewman #microxchg
    • @pfhllnts: "I predict quantum containers where Docker exists both inside and outside a container." @marcoceppi #fosdem
    • @viktorklang: Awesome story: 295x speedup with Akka Streams on same HW compared to Rails :) 
    • krinchan: Yes. Because a currency almost completely controlled by Chinese miners who are strangling the network at 1MB blocks, causing transaction times in excess of three hours at peak and just introduced the ability to arbitrarily reverse those transactions during the lag is totally going to handle DraftKings and FanDuel.
    • @mpesce: 1/The Apple AX series SOCs are more than powerful enough to run a Hololens-type device very effectively.
    • Matthew Yglesias: Amazon's leadership, from CEO Jeff Bezos on down, are deliberately redeploying every dollar of revenue Amazon earns into making the company bigger and bigger.
    • German forest ranger finds that trees have social networks: trees operate less like individuals and more as communal beings. Working together in networks and sharing resources, they increase their resistance to threats
    • @ValaAfshar: 11 years ago some guy named Mark Zuckerberg talks about his new company. He is now 4th richest person in the world. 
    • Bernard Marr: In China, the government is rolling out a social credit score that aggregates not only a citizen’s financial worthiness, but also how patriotic he or she is, what they post on social media, and who they socialize with
    • @Carnage4Life: Facebook is valued at $326 billion and worth more than Exxon Mobil. Remember when people freaked out at $15B value? 
    • @Nick_Craver: High levels of efficiency at scale aren't one thing; it's a thousand things. Many we haven't really shared in detail...and we should.
    • 2BuellerBells: Things to reinvent: Event loops (done!) Unix (In progress!) Erlang (est. 5 years)
    • @LusciousPear: I'm consistently seeing GETs from @googlecloud storage 2-5x faster than S3. niiiice
    • Kevin Old: The future looks mighty scalable.
    • @BenedictEvans: All curation grows until it requires search. All search grows until it requires curation.
    • @Carnage4Life: Google has 7 services with 1B monthly active users; Gmail, Search, Chrome, Android, Maps, YouTube and Google Play 
    • @jmhodges: That's 1.3 million unique domains in a single day. Yesterday. Let's Encrypt is doing a thing.
    • @danielbryantuk: "60% percent of app users rate performance/response time ahead of features" @grabnerandi  #OOP2016 
    • @tdeekens: Sometimes Monoliths don’t get enough respect. They’re part of our revenue system allowing us to build Microservices. They gave us a business
    • Searching for the Algorithms Underlying Life: Valiant’s self-stated goal is to find “mathematical definitions of learning and evolution which can address all ways in which information can get into systems.” If successful, the resulting “theory of everything”...would literally fuse life science and computer science together.
    • @mountain_ghosts: 1995: the information superhighway will mean anyone can do anything from anywhere 2015: must be willing to relocate to San Francisco

  • Fingerprinting made burglars put on gloves. CCTV made kids pull their hoods up. Spying made honest people use encryption. Forensics: What Bugs, Burns, Prints, DNA and More Tell Us About Crime.

  • So that's what bandwidth means. ucaetano: The bandwidth doesn't depend on the frequency you're occupying, but on the amount of spectrum available: you "usually" get in the order of 1 bps for every Hz of spectrum available for mobile: a 20Mz chunk of spectrum will give you ~20Mbps, no matter if it is 700MHz or 5 GHz. Higher frequencies have awful penetration and range, that's why today you define who wins in the mobile game by the amount of 700MHz and 800MHz spectrum they own. In other words, lower frequency spectrum is (within certain limits) always better.

  • Even spies have limits. Optic Nerve: millions of Yahoo webcam images intercepted by GCHQ. A British surveillance agency suffered the indignity of only saving images every five minutes from user feeds to reduce server load. My kingdom for a cloud! Why? They needed data to train their face recognition algorithms. That's what happens if you aren't Google.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Wednesday
Feb032016

A Case Study: WordPress Migration for Shift.ms

The case study presented involves a migration from custom database to WordPress. The company with the task is Valet and it has a vast portfolio of previously done jobs that included shifts from database to WordPress, multisite-to-multisite, and multisite to single site among others. The client is Shift.ms.

Problem

The client, Shift.ms, presented a taxing problem to the team. Shift.ms had a custom database that they needed migrated to WordPress. They had installed a WordPress/BuddyPress and wanted their data moved into this new installation. All this may seem rather simple. However, there was one problem; the client had some data in the newly installed WordPress that they intended to keep.

Challenges

The main problem was that the schema for the database and that of WordPress are very different in infrastructure. The following issues arose in an effort to deal with the problem:

Click to read more ...

Tuesday
Feb022016

Sponsored Post: Netflix, Macmillan Learning, Aerospike, TrueSight Pulse, LaunchDarkly, Robinhood, StatusPage.io, Redis Labs, InMemory.Net, VividCortex, MemSQL, Scalyr, AiScaler, AppDynamics, ManageEngine, Site24x7

Who's Hiring?

  • Macmillan Learning, a premier e-learning institute, is looking for VP of DevOps to manage the DevOps teams based in New York and Austin. This is a very exciting team as the company is committed to fully transitioning to the Cloud, using a DevOps approach, with focus on CI/CD, and using technologies like Chef/Puppet/Docker, etc. Please apply here.

  • DevOps Engineer at Robinhood. We are looking for an Operations Engineer to take responsibility for our development and production environments deployed across multiple AWS regions. Top candidates will have several years experience as a Systems Administrator, Ops Engineer, or SRE at a massive scale. Please apply here.

  • Senior Service Reliability Engineer (SRE): Drive improvements to help reduce both time-to-detect and time-to-resolve while concurrently improving availability through service team engagement.  Ability to analyze and triage production issues on a web-scale system a plus. Find details on the position here: https://jobs.netflix.com/jobs/434

  • Manager - Performance Engineering: Lead the world-class performance team in charge of both optimizing the Netflix cloud stack and developing the performance observability capabilities which 3rd party vendors fail to provide.  Expert on both systems and web-scale application stack performance optimization. Find details on the position here https://jobs.netflix.com/jobs/860482

  • Senior Devops Engineer - StatusPage.io is looking for a senior devops engineer to help us in making the internet more transparent around downtime. Your mission: help us create a fast, scalable infrastructure that can be deployed to quickly and reliably.

  • Software Engineer (DevOps). You are one of those rare engineers who loves to tinker with distributed systems at high scale. You know how to build these from scratch, and how to take a system that has reached a scalability limit and break through that barrier to new heights. You are a hands on doer, a code doctor, who loves to get something done the right way. You love designing clean APIs, data models, code structures and system architectures, but retain the humility to learn from others who see things differently. Apply to AppDynamics

  • Software Engineer (C++). You will be responsible for building everything from proof-of-concepts and usability prototypes to deployment- quality code. You should have at least 1+ years of experience developing C++ libraries and APIs, and be comfortable with daily code submissions, delivering projects in short time frames, multi-tasking, handling interrupts, and collaborating with team members. Apply to AppDynamics

Fun and Informative Events


  • Your event could be here. How cool is that?

Cool Products and Services

  • Aerospike Shows Fivefold Cost Advantage over Cassandra at Higher Performance in DataStax’s Own Benchmark. A recent NoSQL database performance test by DataStax concluded that Cassandra bested Couchbase, MongoDB and HBase. Since Aerospike wasn’t included in the evaluation, we ran the benchmark against Aerospike in the same test cases. The result? Aerospike dramatically outperformed Cassandra AND cost 5 times less. Read the details here

  • Dev teams are using LaunchDarkly’s Feature Flags as a Service to get unprecedented control over feature launches. LaunchDarkly allows you to cleanly separate code deployment from rollout. We make it super easy to enable functionality for whoever you want, whenever you want. See how it works.

  • TrueSight Pulse is SaaS IT performance monitoring with one-second resolution, visualization and alerting. Monitor on-prem, cloud, VMs and containers with custom dashboards and alert on any metric. Start your free trial with no code or credit card.

  • Turn chaotic logs and metrics into actionable data. Scalyr is a tool your entire team will love. Get visibility into your production issues without juggling multiple tools and tabs. Loved and used by teams at Codecademy, ReturnPath, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.

  • InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net

  • VividCortex measures your database servers’ work (queries), not just global counters. If you’re not monitoring query performance at a deep level, you’re missing opportunities to boost availability, turbocharge performance, ship better code faster, and ultimately delight more customers. VividCortex is a next-generation SaaS platform that helps you find and eliminate database performance problems at scale.

  • MemSQL provides a distributed in-memory database for high value data. It's designed to handle extreme data ingest and store the data for real-time, streaming and historical analysis using SQL. MemSQL also cost effectively supports both application and ad-hoc queries concurrently across all data. Start a free 30 day trial here: http://www.memsql.com/

  • aiScaler, aiProtect, aiMobile Application Delivery Controller with integrated Dynamic Site Acceleration, Denial of Service Protection and Mobile Content Management. Also available on Amazon Web Services. Free instant trial, 2 hours of FREE deployment support, no sign-up required. http://aiscaler.com

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below...

Click to read more ...

Tuesday
Feb022016

The Big List of Alternatives to Parse

Parse is not going away. It’s going to get better.
Ilya Sukhar — April 25th, 2013 on the Future of Parse

 

Parse is dead. The great diaspora has begun. The gold rush is on. There’s a huge opportunity for some to feed and grow on Parse’s 600,000 fleeing customers.

Where should you go? What should you do? By now you’ve transitioned through all five stages of grief and ready for stage six: doing something about it. Fortunately there are a lot of options and I’ve gathered as many resources as I can here in one place.

There is a Lot Pain Out There

Parse closing is a bigger deal than most shutterings. There’s even a petition: Don't Shut down Parse.com. That doesn’t happen unless you’ve managed to touch people. What could account for such an outpouring of emotion?

Parse and the massive switch to mobile computing grew up at the same time. Mobile is by definition personal. Many programmers capable of handling UI programming challenge were not as experienced with backend programming and Parse filled that void. When a childhood friend you grew to depend on dies, it hurts. That hurt is deep. It goes into the very nature of how you make stuff, how you grow, how you realize your dreams, how you make a living. That’s a very intimate connection.

For a trip through memory lane Our Incredible Journey is a tumblr chronicling many services that are no longer with us.

Some reactions from around the net:

maxado_zdl: F*ck you facebook!!!!!!!!!!!!!!!!!!!!!!!!

pacp_ec: Damn it Facebook only George R. R. Martin is allowed to kill my heroes

Mythul: I really hate facebook right now ! Thanks for screwing up my apps with your bad business model!

Mufro: Damn. We've been slowly migrating our smaller apps to Parse as we make annual updates. Now we're trying to figure out what we're gonna do... go back to the pain of rolling our own server backends out? This leaves a pretty big hole in the market IMO. I don't know of anyone who gets you off the ground as quickly and affordably as Parse does. It's been a joy to use their product, but I knew deep down it was too good to be true. I guess we'll have to take a look at AWS again, maybe Azure. We use Firebase in another project, so we might check that out too. This sucks though.

samwize7: When Facebook acquired Parse, I thought it is good news since they ain't profitable, and now they have a backing of a giant, who tried hard to woo developers. I built many mobile apps using Parse, and has always been a fan of how they build a product for developers. Their documentation is awesome, their free tier is generous, their SDK covers widely. Today, their announcement is a sad news. And once again, proves that we can't trust Facebook.

clev1: This literally just ruined my day....I've got 2 major projects near completion that I've been using Parse as a BaaS for. Anyone with experience know how difficult or a transition it is to switch to Firebase?

solumamantis: I just can't believe the service is being retired... I started using three months ago - my new app coming out soon is completely reliant on it..... I will have a look on Firebase, but honestly I think i will build my own Parse/Node.js version and manage it myself....

changingminds: What the f*ck. Wtf am I supposed to do with 120k users who currently use my app that uses parse? I gotta redo the entire f*cking backend? F*cking bullsh*t.

manooka: My entire startup relies on Parse. I developed the website and apps myself as this was perfect for me as a Front-end developer without having to worry about back-end servers/databases etc. This is SERIOUSLY bad news.

stuntmanmikey: I'm a full-stack developer who is part of a startup that depends on Parse. As the only developer, the amount of time we've saved NOT having to write a data access layer and web service layer has been a windfall for us. Now I'm left to either switch to a similar product (Firebase just doesn't have the same appeal to me) or implement the backend myself at great cost.

neckbeardfedoras: The thing is, most of the folks using Parse probably use it because they're not full stack or back end developers. Removal of Parse means more time or money spent on resources to manage a back end system.

Why did Facebook Shutdown Parse?

Click to read more ...

Monday
Feb012016

A Patreon Architecture Short

Patreon recently snagged $30 Million in funding. It seems the model of pledging $1 for individual feature releases or code changes won't support fast enough growth. CEO Jack Conte says: We need to bring in so many people so fast. We need to keep up with hiring and keep up with making all of the things.

Since HighScalability is giving Patreon a try I've naturally wondered how it's built. Modulo some serious security issues Patreon has always worked well. So I was interested to dig up this nugget in a thread on the funding round where the Director of Engineering at Patreon shares a little about how Patreon works:

  • Server is in Python using Flask and SQLAlchemy, 
  • Runs on AWS (EC2, RDS (MySQL), and some Redis, Celery, SQS, etc. to boot). 
  • A few microservices here and there in other languages too (e.g. real time chat server with Node & Firebase)
  • Web code is written in React (with some legacy code in Angular). We tend to use Redux for the non-component pieces, but are still trying out new React-compatible libraries here and there.
  • iOS and Android code are written in Objective-C and Java, respectively. 
  • We use Realm on both platforms for data storage
  • Most of the rest is pretty standard modern project stuff (CocoaPods for iOS, Gradle on Android, etc.)

For this time period it seems like a good set of technologies to use for the type of application Patreon is. It's interesting to see Angular as referred to as legacy code. React seems to be winning the framework wars.

The use of Realm is notable on the mobile platform as a common storage layer. Realm's simplicity is attractive.

The use of microservices may have helped Patreon dodge the Parse closing down bullet. Instead of trying to find one backend to rule them all they picked Firebase, a more targeted technology, to implement a specific feature. Service diversification is a great way to manage service failure risk.

Friday
Jan292016

Stuff The Internet Says On Scalability For January 29th, 2016

Hey, it's HighScalability time:


This is a trace of a Google search query. A single query might touch a couple thousand machines.

 

If you like this Stuff then please consider supporting me on Patreon.
  • 88: the too short life of Marvin Minsky; $18.4 billion: profit made by Apple in 3 months; 100M: hours of video watched on Facebook each day; 1.59 billion: Facebook users; $115B: size of game market by 2020; 12 years: Mars rover still going strong; 96.3m: barrels of oil produced per day; 570 Billion: object brighter than the Sun; 134 pounds: carried by drones;  $2.4 billion: AWS Q4 sales; 2.5 million: advertisers on the Facebook;

  • Quotable Quotes:
    • @ptaoussanis: Real-world scaling 101: be in the habit of routinely, objectively asking what parts of your system could stand to be simplified or removed
    • @Carnage4Life: Azure revenue up 140%. Search revenue from #BingAds up 21%. Microsoft is killing it in the cloud
    • @gabriel_boya: Scaling up a Cloud Service on @azure takes so many hours that your customers may be gone by the time your instances are allocated...
    • AJ007: Facebook is the only platform that lets advertisers target a mass audience with very fine demographic precision. Google you lose the demographics. Television, you lose the the precision.
    • Junaid Anwar: It is to be noted that clustering [node.js] yielded two times the performance as compared to the non-clustering case which shows that performance linearly increases with processing cores when clustering is used.
    • crash41301: Our company has been slowly shrinking the hundreds of services we have down to a handful of larger, automated tested services and the dev team (about 50) likes it much more.
    • @swardley: Compute is the activity, Architecture is the practice
    • van lessen: Self-Contained Systems (SCS) describe an architectural approach to build software systems, e.g. to split monoliths into multiple functionally separated, yet largely autonomous web applications. 
    • R. P. Feynman: What is the cause of management's fantastic faith in the machinery?
    • Steven Max Patterson: Facebook filters much from the raw newstream and gives me what it thinks I want with about 20% accuracy.
    • Brandon Butterworth~ a single mega data centre might simply represent a single, large potential point of failure
    • boggzPit: Damn it Facebook. Why did I ever believe you could handle being cool to developers?
    • Vadim Tkachenko: To recap an interesting point in that post: when using 48 cores with the server, the result was worse than with 12 cores. I wanted to understand the reason is was true, so I started digging. My primary suspicion was that Java (I never trust Java) was not good dealing with 100GB of memory.
    • Seth Lloyd: Our algorithm shows that you don't need a big quantum computer to kick some serious topological butt...You could find the topology of simple structures on a very simple quantum computer. 
    • Robert Scoble: When he was doing his thesis 20 years ago, it took him two years to analyze just 24 hours of data from farms (he pulls in data from satellites, Doppler radar and even drones). Today, his company does the same thing in seconds.
    • @jgrahamc: Devotees of microservices use 'monolith' as a derogatory term; wait 10 years and we'll be using 'spider's web' as a derogatory term.
    • @mweagle: I see your femtoservice, and pivot with a single source code point: “yoctoservice” :) #disrupt #unicorn #M&A
    • milesrout: The entire point of Docker is that you use it for everything. It's a universal application image format. That is the point. It's contained, secure, and childproof. That is the point. It's not just about scalability. If I could use a desktop operating system where all programs ran as docker containers, I'd do that too. That's what they're for.
    • Bill Wash: I will never pass up an opportunity to help out a colleague, and I’ll remember the days before I knew everything.
    • @CarlHasselskog:  my startup handles ~10 million uploaded files/day with two employees in total (entire company). That's largely thanks to you guys.
    • AJ Kohn: December saw more negative numbers with a 6.96% decrease, year over year, in desktop search volume. Every month in 2015 had lower desktop query volume than the same month in 2014. Every. Month.
    • Jerry Chen: Every startup has a different size unit of value. Bigger is not better, smaller is not better.
    • sacundim: No, the goal of normalization is to eliminate logical inconsistencies—data sets that entail two or more different answers to the same question. 
    • Jake Archibald: Streams can be used to do fun things like turn clouds to butts, transcode MPEG to GIF, but most importantly, they can be combined with service workers to become the fastest way to serve content.
    • Solomon Hykes: Computers do run only one unikernel at a time. It’s just that sometimes they are virtual computers. Remember that virtualization is increasingly hardware-assisted, and the software parts are mature. So for many use cases it’s reasonable to separate concerns and just assume that VMs are just a special type of computer.

  • Relying on a tool backed by a big company is no protection. Facebook is closing down Parse. This is a stunner because Parse was a popular and well made service, used by millions of now adrift mobile apps. What happened? This might be it: "Facebook also would have had to invest untold millions of dollars in capital and, more importantly, engineering talent, to get the Parse business fully off the ground to have a better chance at making a dent in competitors like Amazon, Microsoft and Google." How about Firebase? The Firebase founder responds: "We're not going anywhere. What makes us different? Firebase is very complementary to Google's other product offerings. Cloud for one, as well as Angular, Polymer, GCM, etc." The moral of the store is told by bsaul: "parse wasn't a core service for facebook, nor a relevant source of a revenue AND their API wasn't standard. Those points combined made it very risky for people to use it." 

  • The Internet will soon be eating a lot of Brotli, Google's new lossless compression algorithm that is making the Internet 17-25% faster. Support will be in Chrome and other browsers, but server side support may take longer. Why does it only work with https? Richard Coles: one reason why this is limited to https is to stop it being mangled by proxies, which has been a practical problem in the past with encodings.

  • Young Skynet is continuing its dastardly plan of self-creation by seeding deep learning both far and wide. Microsoft Open Sources Deep Learning, AI Toolkit On GitHub. Twitter released Distributed learning in TorchTeach Yourself Deep Learning with TensorFlow and Udacity.

  • While the Super Bowl will make a mess of local traffic, it's great for cell phone service. Verizon spent $70 million to triple Bay Area LTE capacity ahead of the Super Bowl. They have more than tripled its 4G LTE network capacity; Build 16 new area cell sites; Install 75 small cells; Boost capacity by adding 37 XLTE to existing sites; Complete preparations to deploy 14 mobile cell sites in high traffic locations.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...