livy is a REST server of Spark. You can see the talk of the Spark Summit 2016, Microsoft uses livy for HDInsight with Jupyter notebook and sparkmagic. Jupyter notebook is one of the most popular notebook OSS within data scientists. Using sparkmagic + Jupyter notebook, data scientists can execute ad-hoc Spark job easily.
Why livy is good? According to the official document, livy has features like:
Have long running SparkContexts that can be used for multiple Spark jobs, by multiple clients Share cached RDDs or Dataframes across multiple jobs and clients Multiple SparkContexts can be managed simultaneously, and they run on the cluster (YARN/Mesos) instead of the Livy Server for good fault tolerance and concurrency Jobs can be submitted as precompiled jars, snippets of code, or via Java/Scala client API Ensure security via secure authenticated communication Apache License, 100% open source Why livy + sparkmagic?