Skip to contents

Working with Livy (Spark)

Source: vignettes/spark-with-livy.Rmd
spark-with-livy.Rmd

Apache Spark processes data using compute that runs in Fabric. Livy is the service that lets an R program submit Spark code and receive its status and output. The Spark code runs on Fabric; it does not run in your local R process.

Use Spark when a transformation is too large for one computer, needs a Spark-specific library or format, or already exists as a Spark application. For a small table read, SQL or a direct Lakehouse reader is usually simpler and starts faster. This guide begins with one small statement, then moves to reusable sessions, complete applications, and shared environments.

Before the first call

You need a Fabric workspace on supported capacity, a Lakehouse with a Livy endpoint, and the tenant admin setting for the Livy API enabled.

A delegated caller needs all four of these Microsoft Entra scopes:

  • Lakehouse.Execute.All
  • Lakehouse.Read.All
  • Code.AccessFabric.All
  • Code.AccessStorage.All

With delegated sign-in and audience = NULL, ‘fabricQueryR’ requests those four scopes. The following Code.* scopes are optional; add one only when the Spark code needs the corresponding Azure service at runtime:

Optional scope Runtime access
Code.AccessAzureKeyvault.All Azure Key Vault secrets, keys, or certificates
Code.AccessAzureDataLake.All Azure Data Lake Storage Gen1
Code.AccessAzureDataExplorer.All Azure Data Explorer (Kusto)
Code.AccessSQL.All Azure SQL

An explicit audience replaces the defaults rather than extending them. For example, include all four required scopes when adding Azure SQL access. The Lakehouse $livy_query() method below calls fabric_livy_query():

livy_scopes <- paste0(
  paste0("https", "://api.fabric.microsoft.com/"),
  c(
    "Lakehouse.Execute.All",
    "Lakehouse.Read.All",
    "Code.AccessFabric.All",
    "Code.AccessStorage.All",
    "Code.AccessSQL.All"
  )
)

# `$livy_query()` is the object interface to `fabric_livy_query()`
result <- lakehouse$livy_query(
  code = "SELECT * FROM external_sql_table",
  kind = "sql",
  audience = livy_scopes
)

Client-credentials authentication instead uses the single Fabric/Power BI .default audience selected by the package. Microsoft currently documents service-principal (SPN) tokens for session jobs. Add that principal to the workspace as a Contributor, and confirm that the tenant’s Livy settings allow it; the workspace role alone does not guarantee service-side acceptance.

Microsoft’s current batch guide is internally inconsistent: its introduction says Azure SPN is unsupported, while its authentication section gives a certificate-based SPN example. ‘fabricQueryR’ can acquire and send an app-only token, but it cannot override a service-side identity restriction. Verify unattended batch authentication in the target tenant and use a delegated user if Fabric rejects the SPN.

The delegated user must be a Contributor in the workspace containing the Livy endpoint and data-source items.

Then, you can discover the Lakehouse item which you want to use Livy with:

library(fabricQueryR)

workspaces <- fabric_workspaces()
matches <- Filter(
  \(x) identical(x$displayName, "Analytics workspace"),
  workspaces
)
stopifnot(length(matches) == 1L)
workspace <- matches[[1L]]
lakehouse <- workspace$lakehouses()[[1L]]

$lakehouses() is the workspace method for fabric_lakehouses().

The discovered FabricLakehouse is a read-only R6 object. Its fields expose the service metadata. $livy_query(), $livy_session(), and $livy_batch_submit() correspond to fabric_livy_query(), fabric_livy_session(), and fabric_livy_batch_submit().

If discovery cannot retrieve the endpoint in your environment, copy the session-job connection string from Lakehouse settings > Livy endpoint and pass that URL instead.

Run one piece of Spark code

$livy_query() (fabric_livy_query()) is the simplest method. It starts a temporary session, runs one statement, waits, and closes the session:

result <- lakehouse$livy_query(
  kind = "sql",
  code = "SELECT 1 AS id, 'hello from Spark' AS message"
)

result$output$parsed

The returned output$parsed value is usually a tibble for tabular results, an R object for JSON, or character output for printed text.

Choose the language that matches the code

The kind argument tells Livy how to interpret code:

Microsoft recommends the latest generally available runtime for production, currently Fabric Runtime 2.0 (Spark 4.1). SparkR still works, but SparkR is deprecated upstream in Spark 4.x and may be removed in a future Spark release. Microsoft Fabric distributes sparklyr for R-first workloads in notebooks and Spark job definitions. Microsoft does not currently document sparklyr over an item-scoped Livy session. Choose PySpark or Spark SQL when the remote workload must be independent of the current SparkR runtime bridge.

kind Code language
"sql" Spark SQL
"sparkr" R: SparkR; an experimental sparklyr bridge is described below
"pyspark" Python with Spark
"spark" Scala

For example, SparkR code can use the active Spark session and Lakehouse:

This again uses $livy_query() (fabric_livy_query()):

result <- lakehouse$livy_query(
  kind = "sparkr",
  code = paste(
    "df <- sql('SELECT * FROM orders LIMIT 100')",
    "printSchema(df)",
    "showDF(df, numRows = 10)",
    sep = "\n"
  )
)

sparklyr is not another Livy kind. The following is an experimental adaptation of Fabric’s documented notebook/Spark-job "synapse" connection; Microsoft does not document it for item-scoped Livy, and ‘fabricQueryR’ does not currently live-test it. If you evaluate that bridge in your tenant, sparklyr code submitted through Livy still uses kind = "sparkr":

result <- lakehouse$livy_query(
  kind = "sparkr",
  code = paste(
    "library(sparklyr)",
    "spark_version <- sparkR.version()",
    "config <- spark_config()",
    paste0(
      "sc <- spark_connect(master = 'yarn', version = spark_version, ",
      "spark_home = '/opt/spark', method = 'synapse', config = config)"
    ),
    "orders <- dplyr::tbl(sc, 'orders')",
    "print(dplyr::collect(head(orders, 10)))",
    "spark_disconnect(sc)",
    sep = "\n"
  )
)

Even when it works, this removes only the application-level dependency on SparkR’s DataFrame verbs. It remains dependent on the R interpreter and SparkR JVM bridge, so validate it after every runtime upgrade and do not treat this example as a supported production contract.

Reuse a session for several statements

Starting Spark can take time. Use $livy_session() (fabric_livy_session()) when sequential statements need to share variables or cached data:

answer <- local({
  session <- lakehouse$livy_session()
  on.exit(session$close(), add = TRUE)

  session$wait()
  session$run("shared_value = 40", kind = "pyspark")
  answer <- session$run("print(shared_value + 2)", kind = "pyspark")
  answer
})
answer$output$parsed

Always close a session explicitly. R object cleanup does not make a network request, so forgetting $close() can leave Spark running. The returned FabricLivySession lifecycle methods ($wait(), $run(), and $close()) do not have separate free-function wrappers.

A standard session is right for one R process running a sequence. High concurrency is an advanced option for several isolated workloads that may share underlying compute; it is not needed for a few statements in order.

Submit a complete application file

Use $livy_batch_submit() (fabric_livy_batch_submit()) when the work is a repeatable Python, R, or Java/Scala script stored in OneLake or ADLS:

batch <- lakehouse$livy_batch_submit(
  file = paste0(
    "abfss://", workspace$id,
    "@onelake.dfs.fabric.microsoft.com/",
    lakehouse$id,
    "/Files/jobs/daily_transform.py"
  ),
  name = "daily-transform",
  wait = TRUE,
  timeout = 1800
)

batch$result()

The application file must already be available through an ABFS or ABFSS path. Percent-encode spaces and other URL-reserved characters in file path segments; raw spaces and whitespace in the workspace or filesystem authority are invalid. Upload the file with $onelake_upload() (fabric_onelake_upload()) first when necessary. With wait = FALSE, the function returns a FabricLivyBatch object immediately; call its $wait(), $result(), or $logs() methods later.

The FabricLivyBatch lifecycle methods ($wait(), $result(), $logs(), $status(), and $cancel()) do not have separate free-function wrappers.

If a session, statement, or batch wait times out, the fabric_livy_timeout_error condition keeps the exact live object in its handle field. You can inspect $status() or request $cancel() through that handle in the current R process. The kind-specific session, statement, or batch field contains safe metadata for logging; after serialization, a handle intentionally no longer carries its credential.

Use an Environment for repeatable configuration

A published Fabric Environment can hold Spark settings and libraries shared by several runs. Discover it and pass its ID when the workload depends on that configuration:

environment <- workspace$environments()[[1L]]

result <- lakehouse$livy_query(
  kind = "pyspark",
  code = "print(spark.version)",
  environment_id = environment$id
)

$environments() is the workspace method for fabric_environments(). The Lakehouse $livy_query() method calls fabric_livy_query().