Going Beyond Cluster Monitoring with Apache Ambari and Driven

More organizations are successfully moving their Hadoop big data applications to production and leveraging the power of their big data. A recent study by DNV GL – Business Assurance and research institute GFK Eurisko concluded that some organizations are starting to see results in the areas of greater efficiency, improved decision making, and more effective customer engagement. The level of growth big data can provide means these applications have graduated from experiments to first class citizens that enterprises need to ensure are running at peak performance and that any failures are minimized. Because of how complex the Hadoop environment can become, DevOps teams and Development Managers have a set of nightmarish challenges ahead of them, regardless of what platform they build upon or what tools they use to monitor performance (like Apache Ambari).

Expectations for management and DevOps teams are extremely high, and they must proactively troubleshoot technical issues to prevent any downtime. Big data applications are pulling data from multiple data sources across from many systems using various technologies that makes troubleshooting a monumental challenge to tackle.

To achieve peak operational performance, the right tools must be considered to have the right visibility and give teams the necessary data to press on. With a whole slew of big data monitoring solutions to choose from, how do you go about picking which ones will work for your organization?

Much like Cloudera Manager, organizations will often turn to Apache Ambari for cluster performance monitoring and management. However, as organizations scale up their Hadoop use, it because increasingly untenable to only have visibility into the cluster performance and have no visibility into how the applications running on the cluster(s) are performing. For a deeper analysis at the application level, DRIVEN is the way to strengthen troubleshooting beyond the cluster.

Is Monitoring the Cluster Enough?

Apache Ambari will tell you when something has failed, but it won’t help you understand why it failed. When exploring how to optimize your big data applications, looking at the cluster is only one piece of the puzzle. Figuring out why an application failed is running slow is the other piece that is simply missing from cluster monitoring solutions like Apache Ambari. The ability to quickly identify problem applications and why they failed without diving into log files can help DevOps fix issues with better efficiency.

Eventually, Driven can help DevOps teams empower their customers with self-service so they can see the status of the applications and do some triage without having to reach out to the operations teams. By answering questions like “Why is my application running slow today?” or “Did my application complete?”, Driven can enable dev teams to answer the majority of the common questions themselves, reducing the inquiries coming into the operations team.

Distribution Compatibility of DRIVEN & Apache Ambari

In most enterprises, multiple distributions are often used as different departments start independent projects. Apache Ambari would, for example, be used to monitor a Hortonworks distribution. It is not until these organizations move to production does the complexity using multiple monitoring solutions across different distributions becomes a hindrance to scaling your big data implementation.

With this issue in mind, DRIVEN was designed to be “technology agnostic,” so it can be installed on any distribution so you can monitor the applications running on all your clusters, regardless of distribution, within a single solution. Being able to see what is running on all your clusters is key to managing multi-tenant, multi-technology deployments. As you will get a macro level view of who and what are consuming the most resources, which applications are the largest and how long they take to complete and when the peak utilization times are on your clusters so you know when you need to limit submission of ad-hoc jobs.

Is There Anything Apache Ambari Can Do That Driven Can’t Do?

To be clear, DRIVEN is not meant to replace Apache Ambari. DRIVEN is not a cluster monitoring or deployment solution but it is designed to work with existing solutions in place, like Apache Ambari.

DRIVEN is meant for application monitoring and Apache Ambari for monitoring the cluster. DRIVEN is the best way to know how your big data applications running on your cluster are performing and Apache Ambari is an excellent solution to know how healthy your cluster is and how it is performing.

Hadoop Performance Management: What’s More Important… The Cluster Or The App?

Apache Ambari is an excellent choice if all you need is to manage the cluster. But for a more comprehensive big data application monitoring and management strategy, monitoring the cluster is just not enough. In tandem with DRIVEN’s application performance management tools, a granular approach is going to give better answers and help you optimize your Hadoop big data applications.

For more information about how DRIVEN compares to Apache Ambari, please download our free Hadoop Performance Monitoring Tools Comparison Guide.