BigData
This is a guest post by Alban Perillat-Merceroz, from the Analytics team at Teads. In part one, we described our Analytics data ingestion pipeline, with BigQuery sitting as our data warehouse. However, having our analytics events in BigQuery is not enough. Most importantly, data needs to be served to our
architecture partitioning scaling
The original contributor to this article was David Yu, Distributed Systems Engineer at Optimizely. Our mission at Optimizely is to help decision makers turn data into action. This requires us to move data with speed and reliability. We track billions of user events, such as page views, clicks and custom
Strategy
This is a guest repost by Suresh Kondamudi from CleverTap. Dealing with large datasets is often daunting. With limited computing resources, particularly memory, it can be challenging to perform even basic tasks like counting distinct elements, membership check, filtering duplicate elements, finding minimum, maximum, top-n elements, or set operations like
AWS
While at Netflix and LinkedIn Siddharth "Sid" Anand wrote some great articles for HS. Sid is back, this time as a Data Architect at Agari. Original article is here. Data-rich companies (e.g. LinkedIn, Facebook, Google, and Twitter) have historically built custom data pipelines over bare metal in
BigData
This is a guest post by Matt Hunt, who leads open source projects for Bloomberg LP R&D. “Big Data” systems continue to attract substantial funding, attention, and excitement. As with many new technologies, they are neither a panacea, nor even a good fit for many common uses. Yet
Database
“You just can't have it all” is a phrase that most of us are accustomed to hearing and that many still believe to be true when discussing the speed, scale and cost of processing data. To reach high speed data processing, it is necessary to utilize more memory
BigData
Data needs the ability to combine with other data in new ways to reach maximum value. So data needs to have the equivalent of sex. That's why I used sex in the title of my previous article, Data Doesn't Need To Be Free, But It Does
BigData
How do we pay for the services we want to create and use? That is the question. Systems like Twitter, Instagram, Pinterest and all the other services you love are not cheap to build at scale. Grow now and figure out your business model later as the VC funding disappears,
Example
This is a guest repost by Pete Soderling, Founder at Hakka Labs, creating a community where software engineers come to grow. In response to a recent post from MongoHQ entitled “You don’t have big data," I would generally agree with many of the author’s points. However, regardless
Java
We have written agents deployed/distributed across the network. Agents sends data every 15 Secs may be even 5 secs. Working on a service/system to which all agent can post data/tuples with marginal payload. Upto 5% drop rate is acceptable. Ultimately the data will be segregated and stored
Example
This is a guest post by Ivan de Prado and Pere Ferrera, founders of Datasalt, the company behind Pangool and Splout SQL Big Data open-source projects. The amount of payments performed using credit cards is huge. It is clear that there is inherent value in the data that can be
Example
“Data is everywhere, never be at a single location. Not scalable, not maintainable.” –Alex Szalay While Galileo played life and death doctrinal games over the mysteries revealed by the telescope, another revolution went unnoticed, the microscope gave up mystery after mystery and nobody yet understood how subversive would be what