advertise
Tuesday
Jan162018

Sponsored Post: Triplebyte, Exoscale, Symbiont, Loupe, Etleap, Aerospike, Stream, Scalyr, VividCortex, Domino Data Lab, MemSQL, Zohocorp

Who's Hiring? 

  • Triplebyte now hires software engineers for top tech companies and hundreds of the most exciting startups like Apple, Dropbox, Mixpanel, and Instacart. They identify your strengths from an online coding quiz and let you skip resume and recruiter screens at multiple companies at once. It's free, confidential, and background-blind. Apply here.

  • Symbiont is a New York-based financial technology company building new kinds of computer networks to connect independent financial institutions together and allow them to share business logic and data in real time. This involves developing a distributed system which is also decentralized, and which allows for the creation of smart contracts, self-executing cryptographic agreements among counterparties. To do so, we're using a lot of techniques in blockchain technology, as well as those from traditional distributed systems, programming language design and cryptography. We are hiring for a number of roles, from entry-level to expert, including Haskell Backend Engineer, Database Engineer, Product Engineer, Site Reliability Engineer (SRE), Programming Language Engineer and SecOps Engineer. To find out more, just e-mail us your resume

  • Need excellent people? Advertise your job here! 

Fun and Informative Events

  • Webinar: January 23, 11am GMT & again at 11am Pacific / 2pm Eastern. How Microservices is Disrupting FinTech; Featuring Guest Speakers from Forrester Research and Genesis Global. The world of Finance is being digitally disrupted across areas such as mobile payments, money transfers, loans, fundraising, and wealth management. Speed of change is critical, and software has become the tip of the spear in this disruption with a new model called microservices, an approach where large applications are broken down into small, loosely coupled and composable autonomous pieces. Join us as our guest speakers Randy Heffner, VP and Principal Analyst at Forrester Research, and Stephen Murphy, CEO at Genesis Global, explain how microservices improve speed of execution, key emerging practices for doing microservices well, and how microservices are enabling disruption in financial services. Register at: https://www.aerospike.com/lp/microservices-disrupting-fintech-webinar/

  • Advertise your event here!

Cool Products and Services

  • Exoscale GPU Cloud Servers. Powerful on-demand GPU. Perfect for your machine learning, artificial, and encoding workloads. GPU instances work exactly like other instances: they are billed by the minute and integrate seamlessly with your existing infrastructure. Tap the GPU's full power with direct passthrough access. Speed-up Tensorflow or any other Deep Learning, Big Data, AI, or Encoding workload. Start your GPU instances via our API or with your existing deployment management tools. Add parallel computational power to your stack with no effort. Get Started

  • .NET developers dealing with Errors in Production: You know the pain of troubleshooting errors with limited time, limited information, and limited tools. Managers want to know what’s wrong right away, users don’t want to provide log data, and you spend more time gathering information than you do fixing the problem. To fix all that, Loupe was built specifically as a .NET logging and monitoring solution. Loupe notifies you about any errors and tells you all the information you need to fix them. It tracks performance metrics, identifies which errors cause the greatest impact, and pinpoints the root causes. Learn more and try it free today.

  • Enterprise-Grade Database Architecture. The speed and enormous scale of today’s real-time, mission critical applications has exposed gaps in legacy database technologies. Read Building Enterprise-Grade Database Architecture for Mission-Critical, Real-Time Applications to learn: Challenges of supporting digital business applications or Systems of Engagement; Shortcomings of conventional databases; The emergence of enterprise-grade NoSQL databases; Use cases in financial services, AdTech, e-Commerce, online gaming & betting, payments & fraud, and telco; How Aerospike’s NoSQL database solution provides predictable performance, high availability and low total cost of ownership (TCO)

  • The Practical Guide to Managing Data Science at Scale. The ability to manage, scale, and accelerate an entire data science discipline increasingly separates successful organizations from those falling victim to hype and disillusionment. Download this practical guide for data science management, if you're currently, or aspiring to be, a data science manager. The paper demystifies and elevates the current state of data science management.

  • Etleap is a Redshift ETL tool that lets you bring all the data everyone wants into Redshift. It's easy enough for analysts to add and manage data connections on their own, without inundating IT/Engineering with requests for help. It takes just minutes to add new connections such as MySQL, Salesforce, S3, and many others, then you can "set it and forget it." Learn more about Redshift ETL with Etleap.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network. 

  • Build, scale and personalize your news feeds and activity streams with getstream.io. Try the API now in this 5 minute interactive tutorial. Stream is free up to 3 million feed updates so it's easy to get started. Client libraries are available for Node, Ruby, Python, PHP, Go, Java and .NET. Stream is currently also hiring Devops and Python/Go developers in Amsterdam. More than 400 companies rely on Stream for their production feed infrastructure, this includes apps with 30 million users. With your help we'd like to ad a few zeros to that number. Check out the job opening on AngelList.

  • Scalyr is a lightning-fast log management and operational data platform.  It's a tool (actually, multiple tools) that your entire team will love.  Get visibility into your production issues without juggling multiple tabs and different services -- all of your logs, server metrics and alerts are in your browser and at your fingertips. .  Loved and used by teams at Codecademy, ReturnPath, Grab, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.

  • MemSQL envisions a world of adaptable databases and flexible data workloads - your data anywhere in real time. Today, global enterprises use MemSQL as a real-time data warehouse to cost-effectively ingest data and produce industry-leading time to insight. MemSQL works in any cloud, on-premises, or as a managed service. Start a free 30 day trial here: memsql.com/download/.

  • Advertise your product or service here!

If you are interested in a sponsored post for an event, job, or product, please contact us for more information.

Click to read more ...

Tuesday
Jan162018

Monolith or Microservices: Which should you start with?

 

This is a guest Post by Jake Lumetta, Founder and CEO, ButterCMS, an API-first CMS. For more content like this, follow @ButterCMS on Twitter and subscribe to our blog.

Conventional wisdom for startups counsels starting with a monolith, but are there situations where you should start with microservices instead? Interviews with dozens of CTOs illuminated the key considerations when deciding whether to start with a monolith or microservices.

Challenging Conventional Wisdom

My good friend Darby Frey recently kicked off a greenfield project after assuming his new role as Sr. Platform Engineering Lead of Gamut. Despite starting out with monolith at his previous company Belly, he discovered that — in the right circumstances — starting with a monolith isn’t always the best way to go.

“As one does, I allowed much of my thinking in my early days [at my new company] to be influenced by my previous company,” Darby told me.

At Belly, Darby and his team broke down their monolith into a fairly large microservices architecture. They managed to get it to a good place, but only after months of trials and tribulations migrating to microservices.

With this experience fresh in his mind, he approached his new project at Gamut a bit more cautious of microservices.

“I was firmly a member of Team Monolith. [I thought] let’s build a single application and just pull things apart later if we start to feel pain,” he said.

While this was a greenfield project, Darby’s team was small, and he had aggressive timelines, so on the surface, a monolith seemed like the obvious choice.

“[But with this new project], I was anxious to not repeat the mistakes of the past.”

And with that, he found himself faced with a decision we all struggle with, should we start with a monolith or microservices and how do we decide?

Evaluating Pros and Cons

Facing A Monolith

Click to read more ...

Friday
Jan122018

Stuff The Internet Says On Scalability For January 12th, 2018

Hey, it's HighScalability time: 


Whiskey still? Chandelier? Sky city? Nope, it's IBM's 50-qubit quantum computer. (engadget)

 

If you like this sort of Stuff then please support me on Patreon. And I'd appreciate your recommending my new book—Explain the Cloud Like I'm 10—to anyone who needs to understand the cloud (who doesn't?). I think they'll like it. Now with twice the brightness and new chapters on Netflix and Cloud Computing.

 

  • 28.5 billion: PornHub visitors; 3 billion: computer chips have Spectre security hole; 75.8%: people incorrectly think private browsing is actually private; 380,000: streams needed to make minimum wage on Spotify; 30Gbps: throughput for servers in Azure using new network interface cards packing field-programmable gate arrays; 8 quadrillion calculations per second: new NOAA supercomputer; $2 billion: market cap for parody cryptocurrency dogecoin; $1 trillion: IoT spending by 2035; 100,000: IoT sensors monitor canal in China; 1,204: definitions for emo; 23 million: digits in largest prime number; 2.8%: decline in PC shipments; 

  • Quoteable Quotes:
    • @Lee_Holmes: We owe a debt of gratitude to the unsung heroes of Spectre and Meltdown: the thousands of engineers that spent their holidays working on OS patches, browser patches, cloud roll-outs, and more. Thank you.
    • Geoff Huston: While a small number of providers have made significant progress in public IPv6 deployments for their respective customer base, the overall majority of the Internet is still exclusively using IPv4. This is despite the fact that among that small set of networks that have deployed IPv6 are some of the largest ISPs in the Internet! 
    • Robert Sapolsky: But our frequent human tragedy is that the more we consume, the hungrier we get. More and faster and stronger. What was an unexpected pleasure yesterday is what we feel entitled to today, and what won’t be enough tomorrow.
    • Simon Wardley: As for losers — I’m sorry to say that one set of losers will be those who hold on to DevOps practices.
    • StackOverflow: Angular and React are by far the most popular across the board, no matter the technology used. It makes sense that they are the clear frontrunners, supported by two of the biggest and most influential companies in tech. Just looking at those two frameworks, Angular is more visited amongst C#, Java, and (to a degree) PHP developers, whereas React is more popular with Rails, Node.js, and Python developers.
    • Matthew Green: due to flaws in both Signal and WhatsApp (which I single out because I use them), it’s theoretically possible for strangers to add themselves to an encrypted group chat. However, the caveat is that these attacks are extremely difficult to pull off in practice, so nobody needs to panic. But both issues are very avoidable, and tend to undermine the logic of having an end-to-end encryption protocol in the first place. 
    • Taylor Lorenz: The data shows that despite its perception as a nascent social platform, Snapchat is much more of a chat app. And key features like Snap Maps have yet to gain widespread adoption among the app’s user base.
    • FittedCloud: In summary, converting C4/M4 instances may not be a trial task. But for those that are able to go through the analysis and convert could save significantly and improve application performance. At the pace with which AWS moves it is likely that next generation instances will continue to be available in the coming years so the sooner you upgrade the better prepared you are for the future generation instances.
    • Sridhar Rajagopalan: When we moved the Sense analytics chain to GCP, the data coverage metric went from below 80% to roughly 99.8% for one of our toughest customer use cases. Put another way, our data litter decreased from over 20%, or one in five, to approximately one in five hundred. That’s a decrease of a factor of approximately 100, or two orders of magnitude!
    • Simon Wardley: We have to be very, very careful on this point. Because people think “reduced waste” means “reduced IT spend”. And it certainly does not. We’ll see more efficiency and rapid development of higher order systems. But in terms of reducing IT spend, people said the same thing about EC2 in 2007, 2008. And they quickly learned about something called Jevons’ Paradox. What happens is that as we make something more efficient, we wind up consuming vastly more of that thing. So when people say “Oh, we’re going to spend so much less money with serverless!” nah — forget it. We’re just going to do more stuff.
    • Stephen Cass: 5G is likely to become the glue that binds many of our critical technologies together, which will put mobile carriers at the center of modern global civilization in a big way.
    • @SGgrc: The General Law of Cross-Task Information Leakage: “In any setting where short-term performance optimizations have global effect, a sufficiently clever task can infer the recent history of other tasks by observing its own performance.”
    • Quincy Larson: It turns out a LOT of developers got their first tech job in their 30s, 40s, and 50s.
    • @mjpt777: The approach of not caring about software performance because processors keep getting faster seems to be a bit broken right now.
    • Bill Joy: We sought “grand challenge” breakthroughs because they can lead to a cascade of positive effects and transformations far beyond their initial applications. The grand challenge approach works — dramatic improvements reducing energy, materials and food impact are possible. If we widely deploy such breakthrough innovations, we will take big steps toward a sustainable future.
    • @cocoaphony: I have made some snarky comments about "serverless" (since it…you know…runs on servers). And I have flailed around quite a bit trying to understand that world. But as it comes into focus, wow, oh my goodness, it really is quite amazing what I'm able to do with so little code.
    • More. There are more quotes. More stuff. More. More. More. Who can resist more?

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Tuesday
Jan092018

How Technology is Changing Music to fit Inside Streaming's New Box

 

Technology shapes and forms. A Roman war chariot could still ride to battle on a modern road. Songs are typically about 3 minutes long because a 78rpm record held about three minutes of sound per side. So it shouldn't be a surprise streaming—a new technology for distributing music—pounds songs, business models, and production methods into new shapes fitting a new medium.

How would artists respond to near zero digital production costs, zero marginal distribution costs, and streaming's pay per play business model? Uniquely, as described in great detail in the podcast WS More or Less: Why Albums are Getting Longer.

Michael Jackson's Thriller album had nine tracks and runs at just over 42 minutes. Chris Brown released Heartbreak on a Full Moon, an album with 45 tracks and runs well over 2 hours. Albums are getting longer.

Why? You get what you measure.

1500 plays or streams of a track from a single album counts as a record sale. So the longer the album the fewer people have to listen to the whole thing to increase record sales. Record sales are how chart rankings are determined. Doing better in the charts gets you more exposure, which gets you more streams, which gets you more exposure, which gets you more money. And so on.

Game the system, rule the world. Streaming almost doubles every year. 251 billion songs were streamed last year. Streaming is a game you want to win.

Hip Hop leads the way by adopting new production strategies based on modern digital technologies. Hip Hop releases more music per year by working as a worldwide team. Teams of producers and song writers collaborate together to produce more music than any one band could every hope to produce.

Increasing production is the new internet-driven digital production model. More people produce more content. More content in the streaming age means more money. We see this same evolution in games, video, and books. Kindle Unlimited, for example, has changed book publishing. A traditional band can't hope to keep up using traditional artisanal methods.

The structure of songs is also changing to game the system. For a song to count as a stream you must listen for 30 seconds. Artists are doing everything they can to get your attention for 30 seconds. After that, who cares?

It's no longer "get to the chorus don't bore us." What's happening is intros are becoming an audio emoji encoding of what to expect in the rest of the song. The example given in the podcast is Despacito, a hit song approaching 5 billion views on YouTube.

Despacito starts with a Puerto Rican guitar riff that let's you know it's a latin ballad, that's followed by shout-outs that tell you it will have some reggae tones, followed by crooned vocal melody that let's you know it's a pop song, followed by some electronic sounds that let's you know it's modern, and so on. Only after that does the song really start. It's like watching a movie trailer counting as watching the movie. In a Chainsmokers song the first 30 second verse never repeats, it just hooks you in. 

Is this the shape of things to come? Inevitably. We're seeing internet economics fundamentally alter everything. There's no escape.

WS More or Less: Why Albums are Getting Longer is a fun and insightful listen. Highly recommended.

Friday
Jan052018

Stuff The Internet Says On Scalability For January 5th, 2018

Hey, it's HighScalability time: 


All we know about how the universe works. The standard model and mind blowing video.

 

If you like this sort of Stuff then please support me on Patreon. And I'd appreciate your recommending my new book—Explain the Cloud Like I'm 10—to anyone who needs to understand the cloud (who doesn't?). I think they'll like it. Now with twice the brightness and new chapters on Netflix and Cloud Computing.

 

  • 15: datacenters not built because of Google's TPU; 5 billlion: items shipped by Amazon Prime; 600: free online courses; 1.6 million: React downloads per week; 140 milliseconds: time Elon Musk's massive backup battery took to respond to crisis at power plant; 16: world spanning Riot Games clusters; < $100 a kilowatt-hour: Lithium-ion battery packs by 2025; 23%: performance OS penalty fixing Intel bug; 200k: pending Bitcoin transactions; 123 million: household data leak from marketing analytics company Alteryx; .67: hashes per day mining Bitcoin with pencil and paper; $21 million: projected cost of redundant power for Atlanta Airport; 62: nuclear test films; 10x: more galaxies in the universe; 55%: DuckDuckGo growth; 49%: increase in node downloads; 144 terabits/second: submarine cable from Hong Kong to L.A.; $100 million/day: spending on apps and advertising on the App Store; 

  • Quotable Quotes:
    • @seldo: Two rounds of deep analysis of employee performance data at Google show that the top predictors of success are being a good communicator in a team where you feel emotionally safe
    • @emilyrauhala: I'm on the Tianjin to Beijing train and the automated announcement just warned us that breaking train rules will hurt our personal credit scores!
    • Laura M. Moy: Someone like me might be able to use cell-site location information to figure out where you live, where you work, where your kids go to school, whether you’re having an affair, and where you worship. 
    • Google: There has been speculation that the deployment of KPTI causes significant performance slowdowns. Performance can vary, as the impact of the KPTI mitigations depends on the rate of system calls made by an application. On most of our workloads, including our cloud infrastructure, we see negligible impact on performance.
    • @cloud_opinion: 2010: AWS: Cloud is real Other vendors: Lol, bookseller, real money is in virtualisation 2017: AWS: Serverless is real Other vendors: Lol, real money is in Containers
    • Peter Norvig: Understanding the brain is a fascinating problem but I think it’s important to keep it separate from the goal of AI which is solving problems. Each field can learn from the other, but if you conflate the two it’s like aiming at two mountain peaks at the same time—you usually end up in the valley between them.
    • @bcrypt: IMO all software engineers should be required to program a laser machine to give themselves LASIK
    • Errata Security: [Meltdown/Spectre] will force a redesign of CPUs and operating systems. While not a big news item for consumers, it's huge in the geek world. We'll need to redesign operating systems and how CPUs are made.
    • Errata Security: the CPUs have no bug. The results are correct, it's just that the timing is different. CPU designers will never fix the general problem of undetermined timing.
    • Linus Torvalds: I think somebody inside of Intel needs to really take a long hard look at their CPU's, and actually admit that they have issues instead of writing PR blurbs that say that everything works as designed.
    • @mjg59: Your workload is almost certainly not syscall bound. Stop worrying about microbenchmarks.
    • @BenedictEvans: 90% of adults on earth have a mobile phone now, and about 60% and growing have a smartphone.
    • @vambenepe: WRT serverless data processing, I'm feeling pretty good about Google Cloud. Redshift: how big a cluster you want? BigQuery: just send queries EMR: how big a cluster you want? Dataflow: just send pipelines Kinesis: how many shards you want? Pub/Sub: just send events
    • @mattiasgeniar: This is intense: AWS instances having 2x/3x the CPU load after the Intel patch got applied. Can your business afford a 3x increase in server spending? 😲
    • @rsthau: 1) Vulnerability to cache timing attacks wasn't part of "correctness" criteria when this stuff was designed. 2) [For Intel] doing the access check in parallel with sequential execution saves time, with no "observable" effect if you ignore 1).
    • Robert Sapolsky: When we stop fearing something, it isn’t because some amygdaloid neurons have lost their excitability. We don’t passively forget that something is scary. We actively learn that it isn’t anymore.
    • RescueTime: Our data showed that we do our most productive work (represented by the light blue blocks) between 10 and noon and then again from 2-5pm each day. However, breaking it down to the hour, we do our most productive work on Wednesdays at 3pm. 
    • ForHackernews: Measureable: 8-12% - Highly cached random memory, with buffered I/O, OLTP database workloads, and benchmarks with high kernel-to-user space transitions are impacted between 8-12%. Examples include Oracle OLTP (tpm), MariaBD (sysbench), Postgres(pgbench), netperf (< 256 byte), fio (random IO to NvME). Modest: 3-7% - Database analytics, Decision Support System (DSS), and Java VMs are impacted less than the “Measureable” category. These applications may have significant sequential disk or network traffic, but kernel/device drivers are able to aggregate requests to moderate level of kernel-to-user transitions. Examples include SPECjbb2005 w/ucode and SQLserver, and MongoDB.
    • There are so many more quotes. Huge piles of quotes. Buckets of 'em.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Monday
Dec182017

Explain the Cloud Like I'm 10

“Todd, can you explain ‘The Cloud’? What is it?” I was asked this question at lunch by Joe, a fellow tour member on a recent trip Linda (my wife) and I took to France.

It was not a question I was expecting on vacation, but with many years of experience as a programmer, a lot of it spent in cloud computing, it’s a question I should have been able to knock out of the park.

Except I didn’t. My answer stank. I hemmed and hawed. I stuttered and sputtered. I could tell that nothing I said was making any sense. I gave a horrible answer, and it has haunted me ever since.

While talking, I noticed a lot of other people at the table were interested in my answer as well. It seemed a lot of smart people were confused about the cloud.

When I got back home I did a lot of research. I was trying to redeem myself by finding the perfect book to recommend. I couldn't find one! So I decided to write Explain the Cloud Like I'm 10. It's the answer I wish I'd given Joe in France.

Here's what I came up with:

  • What Does the Cloud Look Like?
  • Why is it called a ‘cloud’ anyway?
  • If You Use Online Banking You Already Understand The Cloud
  • What is the Cloud?
  • What is a Service?
  • What is a Cloud Service?
  • Facebook Messenger is a Cloud Service. How does it Work?
  • What isn’t a Cloud Service?
  • What does Stuff Mean?
  • What does Stored in the Cloud Mean?
  • What is a Device?
  • What is a Program or App?
  • What is the Internet?
  • What is a Datacenter?
  • What is Cloud Computing?
  • The Good, the Bad and the Ugly of Cloud Services
  • Kindle: Amazon’s Cloud Service for Reading Ebooks
  • iCloud: Apple’s Cloud Service for Syncing Data
  • Google Maps: A Cloud Service for Navigation
  • Cloud DVR: TV in the Cloud
  • Netflix: What Happens When You Press Play?
  • Fighting the Ecosystem Wars in the Proactive Cloud
  • Does stormy weather affect cloud computing?
  • If I say Something is in the cloud, do you know what it means?

Beginners will find the cloud explained from the beginning. Little prior knowledge is assumed. I use lots of pictures, lots of examples, and many somewhat questionable analogies. We build our way up to understanding the cloud idea by idea.

At the end, we'll use what we've learned to understand the inner-workings of Kindle, iCloud, Google Maps, Nest, and Cloud DVRs. You will understand the cloud and what it means for something to be in the cloud.

I also think even intermediate level people will learn something. I cover a lot. You can read an example chapter at Netflix: What Happens When You Press Play? 

If you are someone or know someone who could use a good book on the cloud, then I'd appreciate giving Explain the Cloud Like I'm 10 a try. And please write a review. That would really help.

Thanks!

Friday
Dec152017

Stuff The Internet Says On Scalability For December 15th, 2017

Hey, wake up, it's HighScalability time:

 

Merry Christmas and Happy New Year everyone! I'll be off until the new year. Here's hoping all your gifts were selected using machine learning.

 

If you like this sort of Stuff then please support me on Patreon. And I'd appreciate your recommending my new book—Explain the Cloud Like I'm 10—to anyone who needs to understand the cloud (who doesn't?). I think they'll like it. Now with twice the brightness and new chapters on Netflix and Cloud Computing.

 

  • 157 terabytes: per second raw data output of the Square Kilometre Array; $11 million: made by a 6 year old on YouTube; 14TB: helium hard drive; 1: year education raises IQ 1-5 points; 10: seconds mining time to pay for wifi; 110 TFLOPS: Nvidia Launches $3,000 Titan V; 400: lines of JavaScript injected by Comcast; 20 million: requests per second processed by Netflix to personalize artwork; 270: configuration parameters in postgresql.conf; hundreds: eyes in scallop from a unique mirroring system; $72 billion: record DRAM revenue; 20: rockets landed by SpaceX; 

  • Quotable Quotes:
    • Bill Walton: Mirai was originally developed to help them corner the Minecraft market, but then they realized what a powerful tool they built. Then it just became a challenge for them to make it as large as possible.
    • Stephen Andriole: The entire world of big software design, development, deployment and support is dead. Customers know it, big software vendors know it and next generation software architects know it. The implications are far-reaching and likely permanent. Business requirements, governance, cloud delivery and architecture are the assassins of old "big" software and the liberators of new "small" software. In 20 years very few of us will recognize the software architectures of the 20th century or how software in the cloud enables ever-changing business requirements.
    • Melanie Johnston-Hollitt: There is not yet compute available that can process the data we want to collect and use to understand the universe. 
    • Brandon Liverence: Credit and debit card transaction data shows, at these businesses, the average customer in the top 20 percent spent 8x as much as the average customer from the bottom 80 percent.
    • @evonbuelow: After looking at the source code for a series of k8s components & operators, I'm struck by how go (#golang) is used more as a declarative construct than a set of procedural steps encoding sophisticated logic.
    • apandhi: I had a run-in with CoinHive this weekend so I did a bit of research. Most modern computers can do about 30/h a second. Coinhive currently pays out 0.00009030 XMR ($0.02 USD) per 1M hashes. For a 10 second pause, they'd mine 300 hashes (about $0.000006 USD). To make $1 USD, they'd need to have ~166,666.66 people connect to their in store WIFI.
    • @matt_healy: Went from zero clue about #aws codepipeline and friends yesterday, to setting up an automatic Lambda and API gateway deployment with every git push in production today. Awesome!
    • lgierth: Pubsub is probably one of the lesser known features of IPFS right now, given that it's still marked as experimental. We're researching more efficient tree-forming and message routing algorithms, but generally the interface is pretty stable by now. Pubsub is supported in both go-ipfs and js-ipfs. A shining example of pubsub in use is PeerPad, a collaborative text editor exchanging CRDTs over IPFS/Pubsub
    • Manish Rai Jain: Given these advancements, Amazon Neptune’s design is pre-2000. Single server vertically scaled, asynchronously replicated, lack of transactions — all this screams outdated.
    • There are more quotes. So many more. More. More. More. Yep, there's even more.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Monday
Dec112017

Netflix: What Happens When You Press Play?

 

This article is a chapter from my new book Explain the Cloud Like I'm 10. The first release was written specifically for cloud newbies. I've made some updates and added a few chapters—Netflix: What Happens When You Press Play? and What is Cloud Computing?—that level it up to a couple ticks past beginner. I think even fairly experienced people might get something out of it.

So if you are looking for a good introduction to the cloud or know someone who is, please take a look. I think you'll like it. I'm pretty proud of how it turned out. 

I pulled this chapter together from dozens of sources that were at times somewhat contradictory. Facts on the ground change over time and depend who is telling the story and what audience they're addressing. I tried to create as coherent a narrative as I could. If there are any errors I'd be more than happy to fix them. Keep in mind this article is not a technical deep dive. It's a big picture type article. For example, I don't mention the word microservice even once :-)

 

Netflix seems so simple. Press play and video magically appears. Easy, right? Not so much.

 

Given our discussion in the What is Cloud Computing? chapter, you might expect Netflix to serve video using AWS. Press play in a Netflix application and video stored in S3 would be streamed from S3, over the internet, directly to your device. 

A completely sensible approach…for a much smaller service. 

But that’s not how Netflix works at all. It’s far more complicated and interesting than you might imagine.

To see why let’s look at some impressive Netflix statistics for 2017.

  • Netflix has more than 110 million subscribers.
  • Netflix operates in more than 200 countries. 
  • Netflix has nearly $3 billion in revenue per quarter.
  • Netflix adds more than 5 million new subscribers per quarter.
  • Netflix plays more than 1 billion hours of video each week. As a comparison, YouTube streams 1 billion hours of video every day while Facebook streams 110 million hours of video every day.
  • Netflix played 250 million hours of video on a single day in 2017.
  • Netflix accounts for over 37% of peak internet traffic in the United States.
  • Netflix plans to spend $7 billion on new content in 2018. 

What have we learned? 

Netflix is huge. They’re global, they have a lot of members, they play a lot of videos, and they have a lot of money.

Another relevant factoid is Netflix is subscription based. Members pay Netflix monthly and can cancel at any time. When you press play to chill on Netflix, it had better work. Unhappy members unsubscribe.

Netflix operates in two clouds: AWS and Open Connect.

How does Netflix keep their members happy? With the cloud of course. Actually, Netflix uses two different clouds: AWS and Open Connect. 

Both clouds must work together seamlessly to deliver endless hours of customer-pleasing video.

The three parts of Netflix: client, backend, CDN.

You can think of Netflix as being divided into three parts: the client, the backend, and the CDN. 

The client is the user interface on any device used to browse and play Netflix videos. It could be an app on your iPhone, a website on your desktop computer, or even an app on your Smart TV. Netflix controls each and every client for each and every device. 

Everything that happens before you hit play happens in the backend, which runs in AWS. That includes things like preparing all new incoming video and handling requests from all apps, websites, TVs, and other devices.

Everything that happens after you hit play is handled by Open Connect. Open Connect is Netflix’s custom global content delivery network (CDN). When you press play the video is served from Open Connect. Don’t worry; we’ll talk about what this means later.

Interestingly, at Netflix they don’t actually say hit play on video, they say clicking start on a title. Every industry has its own lingo.

By controlling all three areas—client, backend, CDN— Netflix has achieved complete vertical integration. 

Netflix controls your video viewing experience from beginning to end. That’s why it just works when you click play from anywhere in the world. You reliably get the content you want to watch when you want to watch it. 

Let’s see how Netflix makes that happen.

In 2008 Netflix Started Moving to AWS

Click to read more ...

Friday
Dec082017

Stuff The Internet Says On Scalability For December 8th, 2017

Hey, it's HighScalability time: 


AWS Geek creates spectacular visual summaries.

 

If you like this sort of Stuff then please support me on Patreon. And please recommend my new book—Explain the Cloud Like I'm 10—to those looking to understand the cloud. I think they'll like it.


  • 127 terabytes: per year growth in blockchain if bitcoin wins; 4: hours from tabula rasa to chess god; 1.4 billion: Slack jobs per day; 400: hyperscale data centers worldwide by 2018; 9.8X: Machine Learning Engineer job growth; 14%: Ethereum transactions are for Cryptokitties; 80: seconds per hash on 55 year old IBM 1401 mainframe; $110 billion: app stores spending in 2018; 25: years since first text message; 4,000: AWS code pushes per day; two elephants: of space dust hits earth every day; 

  • Quotable Quotes:
    • @DavidBrin: Now that's what I call engineering! [Voyager 1] Thrusters that haven't been used in 37 years - still reliable!
    • drkoalamanSo despite not supporting other cryptos the majority of my time on the DNM's I think its officially time to step away from bitcoin, at least for the time being. Went to do a direct deal today with a vendor, realized my $250 purchase would end up costing me $315 or so with fees and would still take probably 24 hours to get to him. As of this morning the lowest electrum fee was approx $32 to send coin.... and people reporting at the highest level still not having coin move 12-16 hours later. Vendors are loving this surge but its creating a sellers market and backlogging the blockchain and fees are just crazy... Not to mention not knowing if your $250 will be worth $300 when it gets to the vendor or a random drop in BTC causing it to be less...
    • Alex Lindsay: 30 years ago you couldn’t get cash on Sunday. Now you can send cash on your watch.
    • @prestonjbyrne: “We’re launching on Ethereum” == “100% uptime, unless someone makes a cat app, at which point all bets are off”
    • @GossiTheDog: So I got somebody to talk, without names, about one of the big S3 bucket leaks. A developer set a bucket to open by mistake. They had open S3 bucket monitoring scripts running and got warning emails, which nobody did anything with - nobody had ownership of S3 buckets.
    • @jaksprats: reInvent 2017 Amazon Time Sync Service … Prediction: by reInvent 2018 either they build their own Spanner or they acquire CockroachDB
    • @PatrickMcFadin: 4/ I don’t think we’ll see many more big AWS database announcements after this year. What they have is “good enough” for them and the consensus is they are moving to AI and “everything Alexa” quickly.
    • Eric Horvitz~ in 50 summers, the aviation industry went from canvas flopping on a beach to the Boeing 707...And what is this thing called consciousness, that we use the word consciousness to refer to. Where does that come from? What are these subjective states? We have no idea. We have theories and reflections, but they are not really based in any scientific theories just yet. However, is it possible in 50 summers, we have a whole new world. We have big surprises. We understand how minds work.
    • @martinkl: Google Realtime API is shutting down … — It’s so risky to rely on proprietary services for building apps.
    • @jeremiahg: Equifax’s stock price isn’t recovering post-breach as expected. If the stock remains flat over the next two months, it’ll been interesting to discuss why — what made them different.
    • @xmal: A possible solution to the Fermi Paradox is that any sufficiently advanced civilization is dedicating all its resources to bitcoin mining.
    • @rbhar90: The AI future where megacorporations control enormous datasets and near infinite compute letting them machine learn to predict our every action terrifies me. At NIPS, it's clear this future is nearer rather than further.
    • Sue Hartley: This plant has built this little structure. It's sort of like a barracks for the ant army. And they live inside. When herbevores arrive the ant army comes out and attacks the herbivores...The plant can spend up to 20% of its resources housing and feeding its army.
    • Netflix: With great elasticity comes great responsibility.
    • Michael Widmann: There’s a change in the (NATO) mindset to accept that computers, just like aircraft and ships, have an offensive capability. I need to do a certain mission and I have an air asset, I also have a cyber asset. What fits best for the me to get the effect I want?
    • There are so many more quotes. More. More. More. More...

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Tuesday
Dec052017

Sponsored Post: Symbiont, Loupe, Etleap, Aerospike, Stream, Scalyr, VividCortex, Domino Data Lab, MemSQL, Zohocorp

Who's Hiring? 

  • Symbiont is a New York-based financial technology company building new kinds of computer networks to connect independent financial institutions together and allow them to share business logic and data in real time. This involves developing a distributed system which is also decentralized, and which allows for the creation of smart contracts, self-executing cryptographic agreements among counterparties. To do so, we're using a lot of techniques in blockchain technology, as well as those from traditional distributed systems, programming language design and cryptography. We are hiring for a number of roles, from entry-level to expert, including Haskell Backend Engineer, Database Engineer, Product Engineer, Site Reliability Engineer (SRE), Programming Language Engineer and SecOps Engineer. To find out more, just e-mail us your resume

  • Need excellent people? Advertise your job here! 

Fun and Informative Events

  • Webinar: January 23, 11am GMT & again at 11am Pacific / 2pm Eastern. How Microservices is Disrupting FinTech; Featuring Guest Speakers from Forrester Research and Genesis Global. The world of Finance is being digitally disrupted across areas such as mobile payments, money transfers, loans, fundraising, and wealth management. Speed of change is critical, and software has become the tip of the spear in this disruption with a new model called microservices, an approach where large applications are broken down into small, loosely coupled and composable autonomous pieces. Join us as our guest speakers Randy Heffner, VP and Principal Analyst at Forrester Research, and Stephen Murphy, CEO at Genesis Global, explain how microservices improve speed of execution, key emerging practices for doing microservices well, and how microservices are enabling disruption in financial services. Register at: https://www.aerospike.com/lp/microservices-disrupting-fintech-webinar/

  • Advertise your event here!

Cool Products and Services

  • .NET developers dealing with Errors in Production: You know the pain of troubleshooting errors with limited time, limited information, and limited tools. Managers want to know what’s wrong right away, users don’t want to provide log data, and you spend more time gathering information than you do fixing the problem. To fix all that, Loupe was built specifically as a .NET logging and monitoring solution. Loupe notifies you about any errors and tells you all the information you need to fix them. It tracks performance metrics, identifies which errors cause the greatest impact, and pinpoints the root causes. Learn more and try it free today.

  • Enterprise-Grade Database Architecture. The speed and enormous scale of today’s real-time, mission critical applications has exposed gaps in legacy database technologies. Read Building Enterprise-Grade Database Architecture for Mission-Critical, Real-Time Applications to learn: Challenges of supporting digital business applications or Systems of Engagement; Shortcomings of conventional databases; The emergence of enterprise-grade NoSQL databases; Use cases in financial services, AdTech, e-Commerce, online gaming & betting, payments & fraud, and telco; How Aerospike’s NoSQL database solution provides predictable performance, high availability and low total cost of ownership (TCO)

  • The Practical Guide to Managing Data Science at Scale. The ability to manage, scale, and accelerate an entire data science discipline increasingly separates successful organizations from those falling victim to hype and disillusionment. Download this practical guide for data science management, if you're currently, or aspiring to be, a data science manager. The paper demystifies and elevates the current state of data science management.

  • Etleap is a Redshift ETL tool that lets you bring all the data everyone wants into Redshift. It's easy enough for analysts to add and manage data connections on their own, without inundating IT/Engineering with requests for help. It takes just minutes to add new connections such as MySQL, Salesforce, S3, and many others, then you can "set it and forget it." Learn more about Redshift ETL with Etleap.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network. 

  • Build, scale and personalize your news feeds and activity streams with getstream.io. Try the API now in this 5 minute interactive tutorial. Stream is free up to 3 million feed updates so it's easy to get started. Client libraries are available for Node, Ruby, Python, PHP, Go, Java and .NET. Stream is currently also hiring Devops and Python/Go developers in Amsterdam. More than 400 companies rely on Stream for their production feed infrastructure, this includes apps with 30 million users. With your help we'd like to ad a few zeros to that number. Check out the job opening on AngelList.

  • Scalyr is a lightning-fast log management and operational data platform.  It's a tool (actually, multiple tools) that your entire team will love.  Get visibility into your production issues without juggling multiple tabs and different services -- all of your logs, server metrics and alerts are in your browser and at your fingertips. .  Loved and used by teams at Codecademy, ReturnPath, Grab, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.

  • MemSQL envisions a world of adaptable databases and flexible data workloads - your data anywhere in real time. Today, global enterprises use MemSQL as a real-time data warehouse to cost-effectively ingest data and produce industry-leading time to insight. MemSQL works in any cloud, on-premises, or as a managed service. Start a free 30 day trial here: memsql.com/download/.

  • Advertise your product or service here!

If you are interested in a sponsored post for an event, job, or product, please contact us for more information.

Click to read more ...

Page 1 ... 4 5 6 7 8 ... 224 Next 10 Entries »