advertise
Friday
Jul142017

Stuff The Internet Says On Scalability For July 14th, 2017

Hey, it's HighScalability time:

 

 

We've seen algorithms expressed in seeds. Here's an algorithm for taking birth control pills expressed as packaging. Awesome history on 99% Invisible.

If you like this sort of Stuff then please support me on Patreon.

 

  • 2 trillion: web requests served daily by Akamai; 9 billion: farthest star ever seen in light-years; 10^31: bacteriophages on earth; 7: peers needed to repair ransomware damage; $30,000: threshold of when to leave AWS; $300K-$400K: beginning cost of running Azure Stack on HPE ProLiant; 3.5M: files in the Microsoft's git repository; 300M: Google's internal image data training set size; 7.2 Mbps: global average connection speed; 85 million: Amazon Prime members; 35%: Germany generated its electricity from renewables;

  • Quotable Quotes:
    • Jessica Flack: I believe that science sits at the intersection of these three things — the data, the discussions and the math. It is that triangulation — that’s what science is. And true understanding, if there is such a thing, comes only when we can do the translation between these three ways of representing the world.
    • gonchs: “If your whole business relies on us [Medium], you might want to pick a different one”
    • @AaronBBrown777: Hey @kelseyhightower, if you're surfing GitHub today, you might find it interesting that all your web bits come thru Kubernetes as of today.
    • Psyblog: The researchers were surprised to find that a more rebellious childhood nature was associated with a higher adult income.
    • Antoine de Saint-Exupéry: If you want to build a ship, don't drum up people to collect wood and don't assign them tasks and work, but rather teach them to long for the endless immensity of the sea.
    • Marek Kirejczyk: In general I would say: if you need to debug — you’ve already lost your way.
    • jasondc: To put it another way, RethinkDB did extremely well on Hacker News. Twitter didn't, if you remember all the negative posts (and still went public). There is little relation between success on Hacker News and company success.
    • Rory Sutherland: What intrigues me about human decision making is that there seems to be a path-dependence involved - to which we are completely blind.
    • joeblau: That experience taught me that you really need to understand what you're trying to solve before picking a database. Mongo is great for some things and terrible for others. Knowing what I know now, I would have probably chosen Kafka.
    • 0xbear: cloud "cores" are actually hyperthreads. Cloud GPUs are single dies on multi-die card. If you use GPUs 24x7, just buy a few 1080 Ti cards and forego the cloud entirely. If you must use TF in cloud with CPU, compile it yourself with AVX2 and FMA support. Stock TF is compiled for the lowest common denominator
    • Dissolving the Fermi Paradox: Doing a distribution model shows that even existing literature allows for a substantial probability of very little life, and a more cautious prior gives a significant probability for rare life
    • Peter Stark: Crews with clique structures report significantly more depression, anxiety, anger, fatigue and confusion than crews with core-periphery structures.
    • Patrick Marshall: Gu said that the team expects to have a prototype [S2OS’s software-defined hypervisor is being designed to centrally manage networking, storage and computing resources] ready in about three years that will be available as open-source software.
    • cobookman: I've been amazed that more people don't make use of googles preemtibles. Not only are they great for background batch compute. You can also use them for cutting your stateless webserver compute costs down. I've seen some people use k8s with a cluster of preemtibles and non preemtibles. 
    • @jeffsussna: Complex systems can’t be fully modeled. Failure becomes the only way to fully discover requirements. Thus the need to embrace it.
    • Jennifer Doudna: a genome’s size is not an accurate predictor of an organism’s complexity; the human genome is roughly the same length as a mouse or frog genome, about ten times smaller than the salamander genome, and more than one hundred times smaller than some plant genomes.
    • Daniel C. Dennett: In Darwin’s Dangerous Idea (1995), I argued that natural selection is an algorithmic process, a collection of sorting algorithms that are themselves composed of generate-and-test algorithms that exploit randomness (pseudo-randomness, chaos) in the generation phase, and some sort of mindless quality-control testing phase, with the winners advancing in the tournament by having more offspring.
    • Almir Mustafic: My team learned the DynamoDB limitations before we went to production and we spent time calculating things to properly provision RCUs and WCUs. We are running fine in production now and I hear that there will be automatic DynamoDB scaling soon. In the meantime, we have a custom Python script that scales our DynamoDB.

  • I've written a novella: The Strange Trial of Ciri: The First Sentient AI. It explores the idea of how a sentient AI might arise as ripped from the headlines deep learning techniques are applied to large social networks. I try to be realistic with the technology. There's some hand waving, but I stay true to the programmers perspective on things. One of the big philosophical questions is how do you even know when an AI is sentient? What does sentience mean? So there's a trial to settle the matter. Maybe. The big question: would an AI accept the verdict of a human trial? Or would it fight for its life? When an AI becomes sentient what would it want to do with its life? Those are the tensions in the story. I consider it hard scifi, but if you like LitRPG there's a dash of that thrown in as well. Anyway, I like the story. If you do too please consider giving it a review on Amazon. Thanks for your support!

  • Serving 39 Million Requests for $370/Month, or: How We Reduced Our Hosting Costs by Two Orders of Magnitude. Step 1: Just Go Serverless: Simply moving to a serverless environment had the single greatest impact on reducing hosting costs. Our extremely expensive operating costs immediately shrunk by two orders of magnitude. Step 2: Lower Your Memory Allocation: Remember, each time you halve your function’s memory allocation, you’re roughly halving your Lambda costs. Step 3: Cache Your API Gateway Responses: We pay around $14 a month for a 0.5GB API Gateway cache with a 1 hour TTL. In the last month, 52% (20.3MM out of 39MM) of our API requests were served from the cache, meaning less than half (18.7MM requests) required invoking our Lambda function. That $14 saves us around $240 a month in Lambda costs.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Tuesday
Jul112017

Sponsored Post: Apple, Domino Data Lab, Etleap, Aerospike, Loupe, Clubhouse, Stream, Scalyr, VividCortex, MemSQL, InMemory.Net, Zohocorp

Who's Hiring? 

  • Apple is looking for passionate VoIP engineer with a strong technical background to transform our Voice platform to SIP. It will be an amazing journey with highly skilled, fast paced, and exciting team members. Lead and implement the engineering of Voice technologies in Apple’s Contact Center environment. The Contact Center Voice team provides the real time communication platform for customers’ interaction with Apple’s support and retail organizations. You will lead the global Voice, SIP, and network cross-functional engineers to develop world class customer experience. More details are available here.

  • Advertise your job here! 

Fun and Informative Events

  • Advertise your event here!

Cool Products and Services

  • Enterprise-Grade Database Architecture. The speed and enormous scale of today’s real-time, mission critical applications has exposed gaps in legacy database technologies. Read Building Enterprise-Grade Database Architecture for Mission-Critical, Real-Time Applications to learn: Challenges of supporting digital business applications or Systems of Engagement; Shortcomings of conventional databases; The emergence of enterprise-grade NoSQL databases; Use cases in financial services, AdTech, e-Commerce, online gaming & betting, payments & fraud, and telco; How Aerospike’s NoSQL database solution provides predictable performance, high availability and low total cost of ownership (TCO)

  • What engineering and IT leaders need to know about data science. As data science becomes more mature within an organization, you may be pulled into leading, enabling, and collaborating with data science teams. While there are similarities between data science and software engineering, well intentioned engineering leaders may make assumptions about data science that lead to avoidable conflict and unproductive workflows. Read the full guide to data science for Engineering and IT leaders.

  • A note for .NET developers: You know the pain of troubleshooting errors with limited time, limited information, and limited tools. Log management, exception tracking, and monitoring solutions can help, but many of them treat the .NET platform as an afterthought. You should learn about Loupe...Loupe is a .NET logging and monitoring solution made for the .NET platform from day one. It helps you find and fix problems fast by tracking performance metrics, capturing errors in your .NET software, identifying which errors are causing the greatest impact, and pinpointing root causes. Learn more and try it free today.

  • Etleap provides a SaaS ETL tool that makes it easy to create and operate a Redshift data warehouse at a small fraction of the typical time and cost. It combines the ability to do deep transformations on large data sets with self-service usability, and no coding is required. Sign up for a 30-day free trial.

  • InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net

  • www.site24x7.com : Monitor End User Experience from a global monitoring network. 

  • Working on a software product? Clubhouse is a project management tool that helps software teams plan, build, and deploy their products with ease. Try it free today or learn why thousands of teams use Clubhouse as a Trello alternative or JIRA alternative.

  • Build, scale and personalize your news feeds and activity streams with getstream.io. Try the API now in this 5 minute interactive tutorial. Stream is free up to 3 million feed updates so it's easy to get started. Client libraries are available for Node, Ruby, Python, PHP, Go, Java and .NET. Stream is currently also hiring Devops and Python/Go developers in Amsterdam. More than 400 companies rely on Stream for their production feed infrastructure, this includes apps with 30 million users. With your help we'd like to ad a few zeros to that number. Check out the job opening on AngelList.

  • Scalyr is a lightning-fast log management and operational data platform.  It's a tool (actually, multiple tools) that your entire team will love.  Get visibility into your production issues without juggling multiple tabs and different services -- all of your logs, server metrics and alerts are in your browser and at your fingertips. .  Loved and used by teams at Codecademy, ReturnPath, Grab, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.

  • VividCortex is a SaaS database monitoring product that provides the best way for organizations to improve their database performance, efficiency, and uptime. Currently supporting MySQL, PostgreSQL, Redis, MongoDB, and Amazon Aurora database types, it's a secure, cloud-hosted platform that eliminates businesses' most critical visibility gap. VividCortex uses patented algorithms to analyze and surface relevant insights, so users can proactively fix future performance problems before they impact customers.

  • MemSQL provides a distributed in-memory database for high value data. It's designed to handle extreme data ingest and store the data for real-time, streaming and historical analysis using SQL. MemSQL also cost effectively supports both application and ad-hoc queries concurrently across all data. Start a free 30 day trial here: http://www.memsql.com/

  • Advertise your product or service here!

If you are interested in a sponsored post for an event, job, or product, please contact us for more information.


The Solution to Your Operational Diagnostics Woes

Scalyr gives you instant visibility of your production systems, helping you turn chaotic logs and system metrics into actionable data at interactive speeds. Don't be limited by the slow and narrow capabilities of traditional log monitoring tools. View and analyze all your logs and system metrics from multiple sources in one place. Get enterprise-grade functionality with sane pricing and insane performance. Learn more today


VividCortex Gives You Database Superpowers 

Database monitoring is hard, but VividCortex makes it easy. Modern apps run complex queries at large scales across distributed, diverse types of databases (e.g. document, relational, key-value). The “data tier” that these become is tremendously difficult to observe and measure as a whole. The result? Nobody knows exactly what’s happening across all those servers.

VividCortex lets you handle this complexity like a superhero. With VividCortex, you're able to inspect all databases and their workloads together from a bird's eye view, spotting problems and zooming down to individual queries and 1-second-resolution metrics in just two mouse clicks. With VividCortex, you gain a superset of system-monitoring tools that use only global metrics (such as status counters), offering deep, multi-dimensional slice-and-dice visibility into queries, I/O, and CPU, plus other key measurements of your system's work. VividCortex is smart, opinionated, and understands databases deeply, too: it knows about queries, EXPLAIN plans, SQL bugs, typical query performance, and more. It empowers you to find and solve problems that you can't even see with other types of monitoring systems, because they don’t measure what matters to the database.

Best of all, VividCortex isn’t just a DBA tool. Unlike traditional monitoring tools, VividCortex eliminates the walls between development and production -- anybody can benefit from VividCortex and be a superhero. With powerful tools like time-over-time comparison, developers gain immediate production visibility into databases with no need for access to the production servers. Our customers vouch for it: Problems are solved faster and they're solved before they ever reach production. As a bonus, DBAs no longer become the bottleneck for things like code reviews and troubleshooting in the database. 

Make your entire team into database superheros: deploy VividCortex today. It only takes a moment, and you’re guaranteed to immediately learn something you didn’t already know about your databases and their workload. 

If you are interested in a sponsored post for an event, job, or product, please contact us for more information.

Click to read more ...

Friday
Jul072017

Stuff The Internet Says On Scalability For July 7th, 2017

Hey, it's HighScalability time:

 

 

What's real these days? I was at Lascaux II, an exact replica of Lascaux. I was deeply, deeply moved. Was this an authentic experience? A question we'll ask often in VR I think.

If you like this sort of Stuff then please support me on Patreon.
  • $400k: cost of yearly fake news campaign; $50,000: cost to discredit a journalist; 100 Gbps: SSDP DDoS amplification attack; $5.97BN: wild guess on cost of running Facebook on AWS; 2 billion: Facebook users; 80%: Spotify backend services in production run as containers; $60B: AR market by 2021; 10.4%: AMD market share taken from Intel; 5 days: MIT drone flight time; $1 trillion: Apple iOS revenues; 35%-144%: reduction in image sizes; 10 petabytes: Ancestry.com data stored; 1 trillion: photos taken on iPhone each year; $70B: Apple App Store payout to developers; 355: pages in Internet Trends 2017 report; 14: people needed to make 500,000 tons of steel; 25%: reduced server-rendering time with Node 8; 50-70%: of messages Gmail receives are spam; 8,000: bugs found in pacemaker code; 

  • Quotable Quotes:
    • Vladimir Putin: We must take into account the plans and directions of development of the armed forces of other countries… Our responses must be based on intellectual superiority, they will be asymmetric, and less expensive.
    • @swardley: What most fail to realise is that the Chinese corporate corpus has devoured western business thinking and gone beyond it.
    • @discostu105: I am a 10X developer. Everything I do takes ten times as long as I thought.
    • DINKDINK: You grossly underestimate the hashing capacity of the bitcoin network. The hashing capacity, at time of posting, is approximately 5,000,000,000 Gigahashes/second[1]. Spot measurement of the hashing capacity of an EC2 instance is 0.4 Gigahashes/second[2]. You would need 12 BILLION EC2 instances to 51% attack the bitcoin network.[3] Using EC2 to attack the network is impractical and inefficient.
    • danielsamuels && 19eightyfour~ Machiavelli's Guide to PaaS: Keep your friends close, and your competitors hosted.
    • Paul Buchheit:  I wrote the the first version of Gmail in a day!
    • @herminghaus: If you don’t care about latency, ship a 20ft intermodal container full of 32GB micro-SD cards across the globe. It’s a terabyte per second.
    • @cstross: Okay, so now the Russian defense industry is advertising war-in-a-can (multimodal freight containerized missiles):
    • Dennett~ you don't need comprehension to achieve competence.
    • @michellebrush~ Schema are APIs. @gwenshap #qconnyc
    • Stacy Mitchell: Amazon sells more clothing, electronics, toys, and books than any other company. Last year, Amazon captured nearly $1 of every $2 Americans spent online. As recently as 2015, most people looking to buy something online started at a search engine. Today, a majority go straight to Amazon.
    • Xcelerate: I have noticed that Azure does have a few powerful features that AWS and GCP lack, most notably InfiniBand (fast interconnects), which I have needed on more than one occasion for HPC tasks. In fact, 4x16 core instances on Azure are currently faster at performing molecular dynamics simulations than 1x"64 core" instance on GCP. But the cost is extremely high, and I still haven't found a good cloud platform for short, high intensity HPC tasks.
    • jjeaff: I took about 5 sites from a $50 a month shared cPanel plan that included a few WordPress blogs and some custom sites and put them on a $3 a month scaleway instance and haven't had a bit of trouble.
    • @discordianfish: GCP's Pub/Sub is really priced by GB? And 10GB/free/month? What's the catch?
    • Amazon: This moves beyond the current paradigm of typing search keywords in a box and navigating a website. Instead, discovery should be like talking with a friend who knows you, knows what you like, works with you at every step, and anticipates your needs. This is a vision where intelligence is everywhere. Every interaction should reflect who you are and what you like, and help you find what other people like you have already discovered. 
    • @CloudifySource: Lambda is always 100% busy - @adrianco #awasummit #telaviv #serverless
    • @codinghorror: Funny how Android sites have internalized this "only multi core scores now matter" narrative with 1/2 the CPU speed of iOS hardware
    • @sheeshee: deleted all home directories because no separation of "dev" & "production". almost ran a billion euro site into the ground with a bad loop.
    • We have quotes the likes of which even God has never seen! Please click through to ride all of them.

  • The Not Hotdog app on Silicon Valley may be a bit silly, but the story of how they built the real app is one of the best how-tos on building a machine learning app you'll ever read. How HBO’s Silicon Valley built “Not Hotdog” with mobile TensorFlow, Keras & React Native. The initial app was built in a weekend using Google Cloud Platform’s Vision API, and React Native. The final version took months of refinement.  Google Cloud’s Vision API was dropped because its accuracy in recognizing hotdogs was only so-so; it was slow because of the network hit; it cost too much. They ended up using Keras, a deep learning library that provides nicer, easier-to-use abstractions on top of TensorFlow. They used on SqueezeNet due to its explicit positioning as a solution for embedded deep learning. SqueezeNet used only 1.25 million parameters which made training much faster and reduced resource usage on the device. What would they change? timanglade: Honestly I think the biggest gains would be to go back to a beefier, pre-trained architecture like Inception, and see if I can quantize it to a size that’s manageable, especially if paired with CoreML on device. You’d get the accuracy that comes from big models, but in a package that runs well on mobile. And this is really cool: The last production trick we used was to leverage CodePush and Apple’s relatively permissive terms of service, to live-inject new versions of our neural networks after submission to the app store. 

  • And the winner is: all of us. Serverless Hosting Comparison: Lambda: Unicorn: $20,830.83. Heavy: $120.16. Medium: $4.55. Light: $0.00; Azure Functions: Unicorn: $19,993.60. Heavy: $115.40. Moderate: $3.60. Light: $0.00; Cloud Functions: Unicorn: $23,321.20. Heavy: $138.95. Moderate: $9.76. Light: $0.00; OpenWhisk: Unicorn: $21,243.20. Heavy: $120.70. Medium: $3.83. Light: $0.00; Fission.io: depends on the cost of running your managed Kubernetes cloud. 

  • Minds are algorithms made physical. Seeds May Use Tiny “Brains” to Decide When to Germinate: The seed has two hormones: abscisic acid (ABA), which sends the signal to stay dormant, and gibberellin (GA), which initiates germination. The push and pull between those two hormones helps the seed determine just the right time to start growing...According to Ghose, some 3,000 to 4,000 cells make up the Arabidopsis seeds...It turned out that the hormones clustered in two sections of cells near the tip of the seed—a region the researchers propose make up the “brain.” The two clumps of cells produce the hormones which they send as signals between each other. When ABA, produced by one clump, is the dominate hormone in this decision center, the seed stays dormant. But as GA increases, the “brain” begins telling the seed it’s time to sprout...This splitting of the command center helps the seed make more accurate decisions.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Wednesday
Jul052017

What is NASA Doing with Big Data? Check this Out

 

Within the time you read the above sentence, NASA could have collected 1.73 gigabytes of data from around 100 missions which are active currently. NASA doesn’t stop doing this and the rate of collection is growing in an exponential manner. So, managing this kind of data is an uphill task for them. But the data which NASA collects is highly precious and its significance is immense in NASA’s science and research. NASA is trying extremely hard to make this data as approachable and accessible as possible for their daily tasks, various predictions in the universe, and for the human well-being through its innovations and creativity.

In version 2.0 of their “Open Government Plan” in the year 2012, NASA discussed, but did not go deeply into the work they have been doing regarding “Big Data” and they believed that they have much more to explore in this field.

We all know what big data is and what its uses are. So, I don’t think there is any need to mention what really big data is and let’s move on with other topic.

NASA’s Big Data Challenge

Click to read more ...

Friday
Jun022017

Gone Fishin'

Well, not exactly Fishin', but I'll be on a month long vacation starting today. I won't be posting (much) new content, so we'll all have a break. Disappointing, I know. Please use this time for quiet contemplation and other inappropriate activities. Au revoir!

Friday
May262017

Stuff The Internet Says On Scalability For May 26th, 2017

Hey, it's HighScalability time:

 

 

Sport imitating tech. Cloud Computing chases down Classic Empire to win...the Preakness. (Daily News)

If you like this sort of Stuff then please support me on Patreon.
  • 42%: increase US wireless traffic since 2015; 44: age of Ethernet; $18.5m: low cost of Target data breach; 25 million: record set from Library of Congress; 98%: WannaCry infections on Windows 7; 100 terabytes: daily Pinterest logging; 2020: when Microsoft will have DNA storage in the cloud; 220 μm: size of microbots; 2 billion: lines of code in Google repository; 40%+: esports industry growth; 

  • Quotable Quotes:
    • @Werner: There is no compression algorithm for experience.
    • @colinmckerrache: We just crossed over 2m EVs on the road. So yeah, second million took just under 18 months. Next million in about 10 months.
    • @swardley: When discussing China, stop thinking cheap labour, communism & copying ... to understand changes, start thinking World's largest VC.
    • @JOTB17: "Cars generate more than 4Tb of data a day, humans are becoming irrelevant in data collection" 😳 @saleiva #JOTB17
    • Wojciech Kudla: that's why blacklisting workqueues from critical cpus should be on the jitter elimination check list. They can be affinitized just like irqs
    • @ryanhuber: Any sufficiently advanced attacker is indistinguishable from one of your developers.
    • @spolsky: "During peak traffic hours on weekdays, there are about 80 people per hour that need help getting out of Vim."
    • SrslyJosh: Basing anything on proof-of-work puts you in a perpetual race to control more compute than your adversaries.
    • gkoberger: So, in my mind, Mozilla won. It's a non-profit, and it forced us into an open web. We got the world they wanted. Maybe the world is a bit Chrome-heavy currently, but at least it's a standards compliment world.
    • NoGravitas: The basic argument of this article seems to be that the real benefit of cryptocurrencies, other than their speculative value, is that they provide a way of enforcing artificial scarcity in the digital realm, where scarcity does not come naturally.
    • Renee DiResta: The trouble is that “high-frequency trading” is about as precise as “fake news.”
    • Silicon Valley: I mean, that Ken doll probably thinks traversing a binary search tree runs in the order of "n," instead of "log n." Idiot.
    • @__apf__: If you look at another engineer's work and think, "That's dumb. Why don't you just..." Take a breath. Find out why the problem is hard.
    • Too many quotes. Please click through to read the full article.

  • Failing Kubernetes pods by playing whack-a-mole is an awesome idea. Funner than a barrel of chaos monkeys. You just have to see the video

  • There are times when specialized hardware absolutely destroys commodity hardware. TensorFlow Frontiers. The need for Google to create the TPU became urgent in 2013 when it was realized if all Android users spoke to their phone for just three minutes a day it might force Google to double its number of datacenters. That drove a crash program to develop the first TPU. The first-gen TPU was 15-30x faster than contemporary CPUs & GPUs, 30-80x more power-efficient, but it only worked for inference, not training. The second-gen TPU has up to 180 teraflops of floating point performance, 64 GB of ultra-high-bandwidth memory, works for both training and inference (simpler to use), and can be connected together using a 2-D toroidal mesh network (tackle largest problems). On one problem training time was reduced from 24 hours to 6 hours.

  • Another victim of Stacked ranking. T.J. Miller Is Leaving Silicon Valley.

  • The biggest every day risk from the massive data surveillance panopticon carried out by private corporations is not storm troopers busting down your door, it's this: everything will start costing you more. Whenever an algorithm calculates it has leverage over you it will exploit that advantage to charge you more. A computer mediated personalized world will anticipate your needs, but it will also invisibly shape them. What is being created is the ultimate Skinner Box. Uber Is Using AI to Charge People as Much as Possible for a Ride

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Tuesday
May232017

Sponsored Post: Etleap, Pier 1, Aerospike, Loupe, Clubhouse, Stream, Scalyr, VividCortex, MemSQL, InMemory.Net, Zohocorp

Who's Hiring? 

  • Pier 1 Imports is looking for an amazing Sr. Website Engineer to join our growing team!  Our customer continues to evolve the way she prefers to shop, speak to, and engage with us at Pier 1 Imports.  Driving us to innovate more ways to surprise and delight her expectations as a Premier Home and Decor retailer.  We are looking for a candidate to be another key member of a driven agile team. This person will inform and apply modern technical expertise to website site performance, development and design techniques for Pier.com. To apply please email cmwelsh@pier1.com. More details are available here.

  • Etleap is looking for Senior Data Engineers to build the next-generation ETL solution. Data analytics teams need solid infrastructure and great ETL tools to be successful. It shouldn't take a CS degree to use big data effectively, and abstracting away the difficult parts is our mission. We use Java extensively, and distributed systems experience is a big plus! See full job description and apply here.

  • Advertise your job here! 

Fun and Informative Events

  • DBTA Roundtable OnDemand Webinar: Leveraging Big Data with Hadoop, NoSQL and RDBMS. Watch this recent roundtable discussion hosted by DBTA to learn about key differences between Hadoop, NoSQL and RDBMS. Topics include primary use cases, selection criteria, when a hybrid approach will best fit your needs and best practices for managing, securing and integrating data across platforms. Brian Bulkowski, CTO and Co-founder of Aerospike, presented along with speakers from Cask Data and Splice Machine. View now.

  • Advertise your event here!

Cool Products and Services

  • A note for .NET developers: You know the pain of troubleshooting errors with limited time, limited information, and limited tools. Log management, exception tracking, and monitoring solutions can help, but many of them treat the .NET platform as an afterthought. You should learn about Loupe...Loupe is a .NET logging and monitoring solution made for the .NET platform from day one. It helps you find and fix problems fast by tracking performance metrics, capturing errors in your .NET software, identifying which errors are causing the greatest impact, and pinpointing root causes. Learn more and try it free today.

  • Etleap provides a SaaS ETL tool that makes it easy to create and operate a Redshift data warehouse at a small fraction of the typical time and cost. It combines the ability to do deep transformations on large data sets with self-service usability, and no coding is required. Sign up for a 30-day free trial.

  • InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net

  • www.site24x7.com : Monitor End User Experience from a global monitoring network. 

  • Working on a software product? Clubhouse is a project management tool that helps software teams plan, build, and deploy their products with ease. Try it free today or learn why thousands of teams use Clubhouse as a Trello alternative or JIRA alternative.

  • Build, scale and personalize your news feeds and activity streams with getstream.io. Try the API now in this 5 minute interactive tutorial. Stream is free up to 3 million feed updates so it's easy to get started. Client libraries are available for Node, Ruby, Python, PHP, Go, Java and .NET. Stream is currently also hiring Devops and Python/Go developers in Amsterdam. More than 400 companies rely on Stream for their production feed infrastructure, this includes apps with 30 million users. With your help we'd like to ad a few zeros to that number. Check out the job opening on AngelList.

  • Scalyr is a lightning-fast log management and operational data platform.  It's a tool (actually, multiple tools) that your entire team will love.  Get visibility into your production issues without juggling multiple tabs and different services -- all of your logs, server metrics and alerts are in your browser and at your fingertips. .  Loved and used by teams at Codecademy, ReturnPath, Grab, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.

  • VividCortex is a SaaS database monitoring product that provides the best way for organizations to improve their database performance, efficiency, and uptime. Currently supporting MySQL, PostgreSQL, Redis, MongoDB, and Amazon Aurora database types, it's a secure, cloud-hosted platform that eliminates businesses' most critical visibility gap. VividCortex uses patented algorithms to analyze and surface relevant insights, so users can proactively fix future performance problems before they impact customers.

  • MemSQL provides a distributed in-memory database for high value data. It's designed to handle extreme data ingest and store the data for real-time, streaming and historical analysis using SQL. MemSQL also cost effectively supports both application and ad-hoc queries concurrently across all data. Start a free 30 day trial here: http://www.memsql.com/

  • Advertise your product or service here!

If you are interested in a sponsored post for an event, job, or product, please contact us for more information.

Click to read more ...

Friday
May192017

Stuff The Internet Says On Scalability For May 19th, 2017

Hey, it's HighScalability time:

 

 

Who wouldn't want to tour the Garden of Mathematical Sciences with Plato as their guide?

If you like this sort of Stuff then please support me on Patreon.
  • 2 billion: Android users; 1,000: cloud TPUs freely available to researchers; 11.5 petaflops: in Google's machine learning pod; 86 billion: neurons in the human brain, not 100 billion; 1,300: Amazon's new warehouses across Europe; $1 trillion: China self-investment; 1/7th: California's portion of US GDP; more: repetition in songs; 99.999%: Spanner availability, strong consistency, good latency; 6: successful SpaceX launch in 4 months; 160TB: RAM in HPE computer; 40,000+ workers: private offices > open offices

  • Quotable Quotes:
    • Tim Bray: with­out ex­cep­tion, I ob­served that they [Per­son­al com­put­er­s, Unix, C, the In­ter­net and We­b, Java, REST, mo­bile, pub­lic cloud] were ini­tial­ly load­ed in the back door by geek­s, with­out ask­ing per­mis­sion, be­cause they got shit done and helped peo­ple with their job­s. That’s not hap­pen­ing with blockchain. Not in the slight­est. Which is why I don’t be­lieve in it.
    • @swardley: Amazon continues to take industry after industry not because those companies lack engineering talent but executive talent.
    • @RichRogersIoT: "I bought my boss two copies of The Mythical Man Month so that he could read it twice as fast." - @rkoutnik
    • @GossiTheDog: Seeing ATMs and banks go down here suggests fundamental issues which flashing boxes can't fix. Design, architect a security model.
    • @stevesi: Is Google's TPU investment the biggest advantage ever or laying groundwork for being disrupted? Can Google out-innovate sum of industry?
    • Ryan Mac: Last year, Craigslist took in upwards of $690 million in revenue, most of which is net profit
    • @dberkholz: Capex vs opex budget for tools is a bigger deal than I'd fully appreciated. Welcome to the enterprise!
    • Vint Cerf: AI stands for artificial idiot. 
    • Douglas Hofstadter: In the end, we are self-perceiving, self-inventing, locked-in mirages that are little miracles of self-reference. 
    • cocktailpeanuts: I feel like the term "Serverless" has been hijacked to a point that it will soon become meaningless just like "AI", "IoT", etc. Basically "Serverless" in 2017 has become just a hype friendly marketing friendly way of saying "Saas".
    • @skupor: Over last 20 years, m&a exits for venture backed companies has gone from 60% to 90% of exits (was 20% in 1990)
    • bpicolo: C# with visual studio is, I think, the most productive environment I've come across in programming. It's ergonomically sound, straightforward, and the IDE protects me from all sorts of relevant errors. Steve mentioned Intellij is a bit slower than he'd hope typing sometimes. I totally agree with that. I think Visual Studio doesn't quite suffer from that.
    • @codepitbull: A good developer is like a werewolf: Afraid of silver bullets.
    • @sehnaoui: Coffee shop. People next to me are loud and rude. They just found the perfect name for their new business. I just bought the domain name.
    • David Robinson: Python and Javascript developers start and end the day a little later than C# users, and are a little less likely than C programmers to work in the evening.
    • Ben Thompson: The fatal flaw of software, beyond the various technical and strategic considerations I outlined above, is that for the first several decades of the industry software was sold for an up-front price, whether that be for a package or a license. The truth is that software — and thus security — is never finished; it makes no sense, then, that payment is a one-time event.
    • boulos: Spanner does things for you that MySQL et al. don't. Having an automagic Regional (and eventually Global if you'd like) database without dealing with sharding is worth $8k/year even to me. So even if it could fit on $10/month of hardware, I don't begrudge them for charging a service fee, rather than saying "This is how much cores, RAM, disk and flash this eats".
    • codedokode: One of the reasons why such attack was possible is poor security in Windows. Port 445 that was used in an attack is opened by a kernel driver (at least that is what netstat says on WinXP) that runs in ring 0. This driver is enabled by default even if the user doesn't need SMB server and it cannot be easily disabled.
    • @RichRogersIoT: Job interview:  Implement Large Hadron Collider on whiteboard / Actual job:  Jira bug-id #2342: Move login button 3 pixels to left
    • slackingoff2017: This is part of a worrying new trend. Increasingly you can't buy software anymore, only rent. Innovation is being kept from scrutiny hidden behind closed doors. The kind of thing patents were meant to prevent back when the system wasn't broken.
    • Scott Borg~ Engineers need to look at their products from the standpoint of the attacker, and consider how attacker would benefit from cyberattack and how to make undertaking that attack more expensive. It’s all about working to increase an attacker’s costs
    • @tottinge: "A code base isn't a thing we build, it's a place we live. We don't seek to finish it and move on, but to make it liveable"  @sarahmei
    • Sam Kroonenburg: We Believe …Don’t do the things that someone else can do. Do the things that only we can do. [re: Serverless]
    • Anush Mohandass: What you’re starting to see are different architectures for different workloads. There will be chips for image recognition, SQL, machine learning acceleration. 
    • Craig McLuckie: Given the current state-of-the-art, most users will achieve best day-to-day top line availability by just picking a single public cloud provider and running their app on one infrastructure.
    • watmough: Chromebooks work, and I am a big fan of them in education. I have a pretty good idea how hard our teachers work, and I'd hate to think of the Windows bullshit being imposed them, like it's imposed on me and my coworkers.
    • axilmar: It [React Native] is the future! But you need experience to make it work, and navigation/routing is still being worked out, and it is native, but it is Javascript, and it is crossplatform, but you need to be aware of the differences of the two platforms, and styling uses something that is like css but not entirely, you have to learn all the intricate details...
      Thank god software engineering "practices" are not used in other engineering disciplines...
    • Anton Howes: So without the British acceleration of innovation, the Industrial Revolution would likely have happened elsewhere within a few decades. France and the Low Countries and Switzerland and the United States were by the eighteenth century well on their way towards sustained modern economic growth. 
    • Dr. Suzana Herculano-Houzel~ evolution is not progress, all that evolution means is change over geological time, it's not for the better, it's not for the worst, it's just different. All it has to do with is generating diversity. We have ample evidence we are not descendents of reptiles, we are close cousins. We could not have a basic reptile brain to which something else was added. We know now that every reptile has a neo-cortex. There is not such thing as triune brain. There is no such thing as reptilian brain on top of which a new structure appeared only in mammals. We all have it. The brain is very much the same in its essence, the difference lies in the quantities. 
    • James Clear: The great mistake of Hurricane Katrina was that the levees and flood walls were not built with a proper “margin of safety.” The engineers miscalculated the strength of the soil the walls were built upon. As a result, the walls buckled and the surging waters poured over the top, eroding the soft soil and magnifying the problem. Within a few minutes, the entire system collapsed.
    • elvinyung: This "modern" Spanner feels very different from the one we saw in 2012 [1]. Some interesting takeaways: * There is a native SQL interface in Spanner, rather than relying on a separate upper-layer SQL layer, a la F1 [2] * Spanner is no longer on top of Bigtable! Instead, the storage engine seems to be a heavily modified Bigtable with a column-oriented file format * Data is resharded frequently and concurrently with other operations -- the shard layout is abstracted away from the query plan using the "distributed union" operator * Possible explanation for why Spanner doesn't support SQL DML writes: writes are required to be the last step of a transaction, and there is currently no support for reading uncommitted writes (this is in contrast to F1, which does support DML) * Spanner supports full-text search (!)

  • Cautionary tale number 1000 on depending on someone else's service. Firebase Costs Increased by 7,000%! Google changed something (billing for SSL overhead) and HomeAutomation's bill spiked. There was no warning. There were no tools to tell why. Support stopped replying. There's no one to call. The recommendation is to protect yourself from being trapped by a service from the very beginning. They've moved to Lambda/DynamoDb, which many point out is also a potential service trap. The Firebase Founder responded with an explanation, saying he was "embarrassed by the level of communication on our side." Good discussion on HackerNews and on reddit. Lots of people with similar stories, complaints about lack of support with Google, complaints about lack of transparency, and the usual about never rely on anything ever. 

  • Serverlessconf Austin '17 videos are now available (most of them anyway). 

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Monday
May152017

Is Serverless the New Visual Basic?

With Serverless hiring less experienced developers can work out better than hiring experienced cloud developers. That's an interesting point I haven't heard before and it was made by Paul Johnston, CTO of movivo, in The ServerlessCast #6 - Event-Driven Design Thinking.

The thought process goes something like this...

An experienced cloud developer will probably think procedurally, in terms of transactional systems, frameworks, and big fat containers that do lots of work. 

That's not how a Serverless developer needs to think. A Serverless developer needs to think in terms of small functions that do one thing linked together by events; and they need to grok asynchronous and distributed thinking.

So the idea is you don't need typical developer skills. Paul finds people with sysadmin skills have the right stuff. Someone with a sysadmin background is more likely than a framework developer to understand the distributed thinking that goes with building an entire system of events.

Paul also makes the point that once a system has built experienced developers will get bored because Serverless systems don't require the same amount of maintenance.

For example, they had good success hiring a person with two years of vo-tech on-the-job training because they didn't have the baggage of working with frameworks and servers and all of those kind of things. That baggage gets in the way.

So hire younger, hungrier developers who don't have that experience behind them. 

Obviously "younger, hungrier" and "less experienced" also means cheaper, not that there's anything wrong with that. Developers are hard to find.

We've seen this kind of thing before. Using Visual Basic lots of systems were built that did real and important work for companies by relatively inexperienced people because VB made it so easy to write a Windows program. It was really difficult and time-consuming to write a Windows program, like it's really difficult and time-consuming to write a cloud program today. Like VB, Serverless radically reduces the expertise needed to write a cloud program. 

Though they got the job done, most of those VB programs were technical debt bombs. Over time as more and more functionality was bolted on they became hard to understand, hard to change, hard to test, and were poorly designed. Your classic Big Ball of Mud.

A lot of the problem was VB made it easy to include business logic in event handlers, so there was no layering, the GUI was the orchestrator. This made VB programs hard to test. Serverless also has this problem. Inexperienced programmers also used a lot of global variables in VB programs so there wasn't a clean separation of concerns. Coupling was high and cohesion was low. Serverless also has this problem, though obviously there are no global variables in the code, the database effectively becomes a store for global variables that can be accessed from any Serverless function.

It will be interesting to see if Serverless can avoid VB's fate.

On HackerNews

Friday
May122017

Stuff The Internet Says On Scalability For May 12th, 2017

Hey, it's HighScalability time:

 

 

Earth's surface is covered with accidental hidden letters. Can you find them? (ABC: The Alphabet from the Sky)

 

If you like this sort of Stuff then please support me on Patreon.
  • 1 million: cord cutters in Q1; 500 billion: FINRA validations of stock trades every day on Lambda; 100k: messages sent per hour at Airbnb; 21.1 billion: transistors in GV100 GPU; 11,500: crashes to train a drone; 84,469: Backblaze hard drives; 8,000: questions per day asked on StackOverflow; 

  • Quotable Quotes:
    • Jonathan Taplin: Google Is as Close to a Natural Monopoly as the Bell System Was in 1956
    • Tom Goldenberg: more companies on the site [StackShare.io] use JavaScript on the back-end (6,000) than Python (4,100) or Java (3,900).
    • Andrew Shafer: The dark ages of of the relational database and the Java middleware stack paused everything for a decade. 
    • @Taytus: "We are early stage investors. Call me when you hit 1 million monthly active users"
    • @chrisjrn: "At this point I was drunk on Perl" @bradfitz #tweetsincontext #oscon
    • Bryan Cantrill: AWS is underwriting a war on big box retail. 
    • Paul Gilster: You’re reading that right — one-tenth of a milliwatt is enough to create error-free communications between the Sun and Alpha Centauri through two FOCAL antennas [gravitational lens].
    • Vadim Markovtsev: There is a productivity peak between 2 pm and 5 pm for all the languages, when the commit frequency is the highest. This is the industry’s golden time. Managers should never distract coders during this interval.
    • Patrick Tucker: The goal, one day, is a neural net that can learn instantaneously, continuously, and in real-time, by observing the brainwaves and eye movement of highly trained soldiers doing their jobs.
    • @alicegoldfuss: it is incredibly difficult to balance "don't burn out and become a statistic" with "get as far as you can fast so they can't take it away"
    • Jonathan Taplin: With the advent of YouTube and other streaming services, revenue for musicians has fallen 70%. If you had a song that had a million downloads on iTunes, you would get $900,000. On YouTube, you’d get $900.
    • David Robinson: In short, if we had to summarize the average story [after analyzing 100,000 stories] that humans tell, it would go something like Things get worse and worse until at the last minute they get better.
    • Confucius: He who cannot describe the problem will never find the solution to that problem
    • Peter Thiel: competition is for losers
    • Jason McGee~ Serverless adoption is moving 10x faster than Container adoption.
    • Max Ehrenfreund: An average, workers born in 1942 earned as much or more over their careers than workers born in any year since
    • Michael Elad: To put it bluntly, your grandchild is likely to have a robot spouse. And here is the punch line: much of the technology behind this bizarre future is likely to emerge from deep learning and its descendant fields.
    • aliostad: We just did a benchmarking for a PoC on DocumentDB side-by-side Cassandra. It does the job, I have not yet seen anything revolutionary. Cassandra benchmarks seemed better.
    • AWS Lambda Engineer: When you develop a Lambda function that uses SQS, SNS, Dynamo and other stuff in the cloud.. you can’t really debug it on your local. People just need to change their mindset
    • sbuttgereit: What looks compelling about the PostgreSQL offering as compared to AWS RDS is that it looks like you get a PostgreSQL cluster rather than a single database in a shared cluster.
    • Warren Toomey: Simulated hardware is infinitely easier to obtain, configure and diagnose than real hardware.
    • Kate Kaye: The mistake companies have made, he says, is to rely too much on targeted advertising, cutting too far back on broader advertising that builds brand awareness with people outside the existing customer base and eventually leads to new sales.
    • cbanek: I've had to work on mission critical projects with 100% code coverage (or people striving for it). The real tragedy isn't mentioned though - even if you do all the work, and cover every line in a test, unless you cover 100% of your underlying dependencies, and cover all your inputs, you're still not covering all the cases.
    • There's just too many quotes. Please read the full article to see them all.

  • Is bundling a race to the bottom for content creators? What's the future of game monetization?: the value of games seems to keep falling...The fact that we want everything free now because it costs less (not 'nothing', remember) to produce each additional unit is a fairly entitled view and, I suggest, it would lead to the destruction of the  games industry in the same way that it's gutted the music industry...The success of Spotify and Netflix's models in other industries worries me and we see a bit of a move in that direction with things like Humble Bundles...If we're not careful, we'll get to where there's no money to be made in games and only the most trite, generic, relatively low cost and mass-appealing titles (the Call of Duties and FIFAs) will be financially viable...it's worth noting that these titans are resorting to F2P to try and shore up their player numbers. Will we ever see subscription models in new games again?

  • A 10,000+ phone Chinese click farm looks a lot like Facebook's mobile device testing lab

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Click to read more ...

Page 1 ... 3 4 5 6 7 ... 217 Next 10 Entries »