Setting up Spark Dynamic Allocation on MapR

Read this article on my new blog

Apache Spark can use various cluster manager to execute application (Stand Alone, YARN, Apache Mesos). When you install Apache Spark on MapR you can submit application in a Stand Alone mode or using YARN.

This article focuses on YARN and Dynamic Allocation, a feature that lets Spark add or remove executors dynamically based on the workload. You can find more information about this feature in this presentation from Databricks:

• Dynamic Allocation in Spark

Let’s see how to configure Spark and YARN to use dynamic allocation (that is disabled by default).

Save MapR Streams messages into MapR DB JSON

Read this article on my new blog

In this article you will learn how to create a MapR Streams Consumer that saves all the messages into a MapR-DB JSON Table.

Getting Started with MapR Streams

Read this article on my new blog

You can find a new tutorial that explains how to deploy an Apache Kafka application to MapR Streams, the tutorial is available here:

• Getting Started with MapR Streams

MapR Streams is a new distributed messaging system for streaming event data at scale, and it’s integrated into the MapR converged platform. MapR Streams uses the Apache Kafka API, so if you’re already familiar with Kafka, you’ll find it particularly easy to get started with MapR Streams.