Skip to contents

Submit a Microsoft Fabric Livy batch job

Source: R/fabric_livy_batch.R
fabric_livy_batch_submit.Rd

Runs a complete Python, R, or Java/Scala Spark application stored in OneLake or ADLS. Use this for repeatable scripts and unattended processing; use fabric_livy_session() when several interactive statements should share variables and Spark state

Usage

fabric_livy_batch_submit(
  livy_url,
  file,
  name = NULL,
  class_name = NULL,
  args = NULL,
  jars = NULL,
  files = NULL,
  py_files = NULL,
  archives = NULL,
  conf = NULL,
  environment_id = NULL,
  target_lakehouse_id = NULL,
  tags = NULL,
  driver_memory = NULL,
  driver_cores = NULL,
  executor_memory = NULL,
  executor_cores = NULL,
  num_executors = NULL,
  tenant_id = Sys.getenv("FABRICQUERYR_TENANT_ID"),
  client_id = Sys.getenv("FABRICQUERYR_CLIENT_ID", unset =
    "04b07795-8ddb-461a-bbee-02f9e1bf7b46"),
  token = NULL,
  auth_args = list(),
  audience = NULL,
  verbose = TRUE,
  wait = FALSE,
  timeout = 1200,
  poll_interval = 5,
  cancel_on_timeout = TRUE
)

Arguments

livy_url

A copied Livy connection URL, Livy API base URL, or enriched Lakehouse object. Copy the batch-job URL from Lakehouse settings > Livy endpoint, or use an item from fabric_lakehouses()

file

Absolute ABFS/ABFSS URI of the main Python, R, or Java/Scala application file. It must contain a filesystem/container, host, and non-root path, without a password, port, query, fragment, backslash, or dot path segment. After uploading a script under a Lakehouse's Files/ area, its Properties dialog can copy this path. Spaces in path segments must be percent-encoded as %20; raw spaces and authority whitespace are invalid

name

Optional readable job name shown in Fabric monitoring

class_name

Main class for a Java/Scala application; leave NULL for Python or R scripts

args

Optional character vector of command-line arguments passed to the application

jars

Optional JAR dependency URIs

files

Optional supporting-file URIs copied to the job

py_files

Optional Python dependency URIs, such as .py or .zip files

archives

Optional archive URIs that Spark should unpack

conf

Optional named list of Spark settings or application-specific values

environment_id

Optional GUID of a published Fabric Environment whose libraries and Spark settings should be used

target_lakehouse_id

Optional Lakehouse GUID made available as spark.targetLakehouse. Use this when the application needs an explicit default Lakehouse context

tags

Optional named list of string labels for monitoring

driver_memory, executor_memory

Optional Spark memory values such as "4g". Leave NULL to use Fabric defaults

driver_cores, executor_cores, num_executors

Optional Spark resource counts. Larger values consume more capacity; leave NULL unless the workload has been sized deliberately

tenant_id

Microsoft Entra tenant ID. Defaults to FABRICQUERYR_TENANT_ID

client_id

Microsoft Entra application/client ID. Defaults to FABRICQUERYR_CLIENT_ID, then the Azure CLI application ID

token

Optional access token or token-provider function. Leave NULL to let 'fabricQueryR' use its normal sign-in flow for a Microsoft Fabric host. A custom livy_url requires an explicitly supplied token or provider. HTTPS validation does not prove ownership or token audience; use a custom host only when your organization controls it, with a credential issued for its intended audience

auth_args

Additional sign-in options passed to AzureAuth::get_azure_token()

audience

Optional sign-in scopes. For delegated sign-in, NULL requests the four required Livy scopes listed below. An explicit vector replaces those defaults, so include every required scope plus any optional Code.Access* scope the Spark code needs. Client credentials require one .default audience

verbose

Logical. Show submission and lifecycle messages

wait

Logical. FALSE returns immediately so other R work can continue; TRUE waits for a terminal state before returning the same object

timeout

Maximum seconds to wait when wait = TRUE

poll_interval

Seconds between status checks when waiting

cancel_on_timeout

Logical. When waiting at submission time, request cancellation if the local timeout expires. Defaults to TRUE, so a timed out call does not normally leave Spark compute running unattended. The structured timeout condition contains the live FabricLivyBatch object in handle, for status checks or cancellation in the current R process, and stable public metadata in batch. A serialized handle intentionally loses its in-process credential

Value

A FabricLivyBatch 'R6' object. Inspect its $state, call $result() for structured metadata and logs, and call $wait() later when submitting with wait = FALSE

Before you submit

Fabric needs a workspace on supported capacity and a Lakehouse. The application file must already be accessible through an ABFS/ABFSS URI; this function does not upload a local script. Use fabric_onelake_upload() first when needed

Delegated sign-in requires Lakehouse.Execute.All, Lakehouse.Read.All, Code.AccessFabric.All, and Code.AccessStorage.All. Add Code.AccessAzureKeyvault.All, Code.AccessAzureDataLake.All, Code.AccessAzureDataExplorer.All, or Code.AccessSQL.All only when Spark accesses that Azure service at runtime. The signed-in identity also needs an appropriate workspace role

Microsoft's current batch guide is internally inconsistent about service principals: its introduction says SPN is unsupported, while its authentication section provides a certificate-based SPN example. This package can acquire and send a client-credentials token, but cannot make the Fabric service accept that identity. Until Microsoft clarifies the contract, verify unattended batch authentication in the target tenant and use a delegated user when the service rejects an SPN. A Contributor role alone is not a guarantee of batch SPN support

See also

Microsoft Fabric batch jobs

Examples

if (FALSE) { # \dontrun{
# Discover the Lakehouse and Python file used by this batch
workspace <- fabric_workspaces()[[1L]]
lakehouse <- fabric_lakehouses(workspace)[[1L]]
scripts <- fabric_onelake_list(
  workspace,
  lakehouse,
  path = "Files/jobs"
)
script <- scripts[grepl("[.]py$", scripts$path), ][1L, ]
script_uri <- paste0(
  "abfss://", workspace$id, "@onelake.dfs.fabric.microsoft.com/",
  lakehouse$id, "/", script$path[[1L]]
)

# Submit the discovered script and wait for its Spark application to finish
batch <- fabric_livy_batch_submit(
  lakehouse,
  file = script_uri,
  wait = TRUE,
  cancel_on_timeout = TRUE
)
batch$result()
} # }