Google Cloud Native is in preview. Google Cloud Classic is fully supported.
Google Cloud Native v0.32.0 published on Wednesday, Nov 29, 2023 by Pulumi
google-native.dataproc/v1beta2.getJob
Explore with Pulumi AI
Google Cloud Native is in preview. Google Cloud Classic is fully supported.
Google Cloud Native v0.32.0 published on Wednesday, Nov 29, 2023 by Pulumi
Gets the resource representation for a job in a project.
Using getJob
Two invocation forms are available. The direct form accepts plain arguments and either blocks until the result value is available, or returns a Promise-wrapped result. The output form accepts Input-wrapped arguments and returns an Output-wrapped result.
function getJob(args: GetJobArgs, opts?: InvokeOptions): Promise<GetJobResult>
function getJobOutput(args: GetJobOutputArgs, opts?: InvokeOptions): Output<GetJobResult>
def get_job(job_id: Optional[str] = None,
project: Optional[str] = None,
region: Optional[str] = None,
opts: Optional[InvokeOptions] = None) -> GetJobResult
def get_job_output(job_id: Optional[pulumi.Input[str]] = None,
project: Optional[pulumi.Input[str]] = None,
region: Optional[pulumi.Input[str]] = None,
opts: Optional[InvokeOptions] = None) -> Output[GetJobResult]
func LookupJob(ctx *Context, args *LookupJobArgs, opts ...InvokeOption) (*LookupJobResult, error)
func LookupJobOutput(ctx *Context, args *LookupJobOutputArgs, opts ...InvokeOption) LookupJobResultOutput
> Note: This function is named LookupJob
in the Go SDK.
public static class GetJob
{
public static Task<GetJobResult> InvokeAsync(GetJobArgs args, InvokeOptions? opts = null)
public static Output<GetJobResult> Invoke(GetJobInvokeArgs args, InvokeOptions? opts = null)
}
public static CompletableFuture<GetJobResult> getJob(GetJobArgs args, InvokeOptions options)
public static Output<GetJobResult> getJob(GetJobArgs args, InvokeOptions options)
fn::invoke:
function: google-native:dataproc/v1beta2:getJob
arguments:
# arguments dictionary
The following arguments are supported:
getJob Result
The following output properties are available:
- Done bool
- Indicates whether the job is completed. If the value is false, the job is still in progress. If true, the job is completed, and status.state field will indicate if it was successful, failed, or cancelled.
- Driver
Control stringFiles Uri - If present, the location of miscellaneous control files which may be used as part of job setup and handling. If not present, control files may be placed in the same location as driver_output_uri.
- Driver
Output stringResource Uri - A URI pointing to the location of the stdout of the job's driver program.
- Hadoop
Job Pulumi.Google Native. Dataproc. V1Beta2. Outputs. Hadoop Job Response - Optional. Job is a Hadoop job.
- Hive
Job Pulumi.Google Native. Dataproc. V1Beta2. Outputs. Hive Job Response - Optional. Job is a Hive job.
- Job
Uuid string - A UUID that uniquely identifies a job within the project over time. This is in contrast to a user-settable reference.job_id that may be reused over time.
- Labels Dictionary<string, string>
- Optional. The labels to associate with this job. Label keys must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt). Label values may be empty, but, if present, must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt). No more than 32 labels can be associated with a job.
- Pig
Job Pulumi.Google Native. Dataproc. V1Beta2. Outputs. Pig Job Response - Optional. Job is a Pig job.
- Placement
Pulumi.
Google Native. Dataproc. V1Beta2. Outputs. Job Placement Response - Job information, including how, when, and where to run the job.
- Presto
Job Pulumi.Google Native. Dataproc. V1Beta2. Outputs. Presto Job Response - Optional. Job is a Presto job.
- Pyspark
Job Pulumi.Google Native. Dataproc. V1Beta2. Outputs. Py Spark Job Response - Optional. Job is a PySpark job.
- Reference
Pulumi.
Google Native. Dataproc. V1Beta2. Outputs. Job Reference Response - Optional. The fully qualified reference to the job, which can be used to obtain the equivalent REST path of the job resource. If this property is not specified when a job is created, the server generates a job_id.
- Scheduling
Pulumi.
Google Native. Dataproc. V1Beta2. Outputs. Job Scheduling Response - Optional. Job scheduling configuration.
- Spark
Job Pulumi.Google Native. Dataproc. V1Beta2. Outputs. Spark Job Response - Optional. Job is a Spark job.
- Spark
RJob Pulumi.Google Native. Dataproc. V1Beta2. Outputs. Spark RJob Response - Optional. Job is a SparkR job.
- Spark
Sql Pulumi.Job Google Native. Dataproc. V1Beta2. Outputs. Spark Sql Job Response - Optional. Job is a SparkSql job.
- Status
Pulumi.
Google Native. Dataproc. V1Beta2. Outputs. Job Status Response - The job status. Additional application-specific status information may be contained in the type_job and yarn_applications fields.
- Status
History List<Pulumi.Google Native. Dataproc. V1Beta2. Outputs. Job Status Response> - The previous job status.
- Submitted
By string - The email address of the user submitting the job. For jobs submitted on the cluster, the address is username@hostname.
- Yarn
Applications List<Pulumi.Google Native. Dataproc. V1Beta2. Outputs. Yarn Application Response> - The collection of YARN applications spun up by this job.Beta Feature: This report is available for testing purposes only. It may be changed before final release.
- Done bool
- Indicates whether the job is completed. If the value is false, the job is still in progress. If true, the job is completed, and status.state field will indicate if it was successful, failed, or cancelled.
- Driver
Control stringFiles Uri - If present, the location of miscellaneous control files which may be used as part of job setup and handling. If not present, control files may be placed in the same location as driver_output_uri.
- Driver
Output stringResource Uri - A URI pointing to the location of the stdout of the job's driver program.
- Hadoop
Job HadoopJob Response - Optional. Job is a Hadoop job.
- Hive
Job HiveJob Response - Optional. Job is a Hive job.
- Job
Uuid string - A UUID that uniquely identifies a job within the project over time. This is in contrast to a user-settable reference.job_id that may be reused over time.
- Labels map[string]string
- Optional. The labels to associate with this job. Label keys must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt). Label values may be empty, but, if present, must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt). No more than 32 labels can be associated with a job.
- Pig
Job PigJob Response - Optional. Job is a Pig job.
- Placement
Job
Placement Response - Job information, including how, when, and where to run the job.
- Presto
Job PrestoJob Response - Optional. Job is a Presto job.
- Pyspark
Job PySpark Job Response - Optional. Job is a PySpark job.
- Reference
Job
Reference Response - Optional. The fully qualified reference to the job, which can be used to obtain the equivalent REST path of the job resource. If this property is not specified when a job is created, the server generates a job_id.
- Scheduling
Job
Scheduling Response - Optional. Job scheduling configuration.
- Spark
Job SparkJob Response - Optional. Job is a Spark job.
- Spark
RJob SparkRJob Response - Optional. Job is a SparkR job.
- Spark
Sql SparkJob Sql Job Response - Optional. Job is a SparkSql job.
- Status
Job
Status Response - The job status. Additional application-specific status information may be contained in the type_job and yarn_applications fields.
- Status
History []JobStatus Response - The previous job status.
- Submitted
By string - The email address of the user submitting the job. For jobs submitted on the cluster, the address is username@hostname.
- Yarn
Applications []YarnApplication Response - The collection of YARN applications spun up by this job.Beta Feature: This report is available for testing purposes only. It may be changed before final release.
- done Boolean
- Indicates whether the job is completed. If the value is false, the job is still in progress. If true, the job is completed, and status.state field will indicate if it was successful, failed, or cancelled.
- driver
Control StringFiles Uri - If present, the location of miscellaneous control files which may be used as part of job setup and handling. If not present, control files may be placed in the same location as driver_output_uri.
- driver
Output StringResource Uri - A URI pointing to the location of the stdout of the job's driver program.
- hadoop
Job HadoopJob Response - Optional. Job is a Hadoop job.
- hive
Job HiveJob Response - Optional. Job is a Hive job.
- job
Uuid String - A UUID that uniquely identifies a job within the project over time. This is in contrast to a user-settable reference.job_id that may be reused over time.
- labels Map<String,String>
- Optional. The labels to associate with this job. Label keys must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt). Label values may be empty, but, if present, must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt). No more than 32 labels can be associated with a job.
- pig
Job PigJob Response - Optional. Job is a Pig job.
- placement
Job
Placement Response - Job information, including how, when, and where to run the job.
- presto
Job PrestoJob Response - Optional. Job is a Presto job.
- pyspark
Job PySpark Job Response - Optional. Job is a PySpark job.
- reference
Job
Reference Response - Optional. The fully qualified reference to the job, which can be used to obtain the equivalent REST path of the job resource. If this property is not specified when a job is created, the server generates a job_id.
- scheduling
Job
Scheduling Response - Optional. Job scheduling configuration.
- spark
Job SparkJob Response - Optional. Job is a Spark job.
- spark
RJob SparkRJob Response - Optional. Job is a SparkR job.
- spark
Sql SparkJob Sql Job Response - Optional. Job is a SparkSql job.
- status
Job
Status Response - The job status. Additional application-specific status information may be contained in the type_job and yarn_applications fields.
- status
History List<JobStatus Response> - The previous job status.
- submitted
By String - The email address of the user submitting the job. For jobs submitted on the cluster, the address is username@hostname.
- yarn
Applications List<YarnApplication Response> - The collection of YARN applications spun up by this job.Beta Feature: This report is available for testing purposes only. It may be changed before final release.
- done boolean
- Indicates whether the job is completed. If the value is false, the job is still in progress. If true, the job is completed, and status.state field will indicate if it was successful, failed, or cancelled.
- driver
Control stringFiles Uri - If present, the location of miscellaneous control files which may be used as part of job setup and handling. If not present, control files may be placed in the same location as driver_output_uri.
- driver
Output stringResource Uri - A URI pointing to the location of the stdout of the job's driver program.
- hadoop
Job HadoopJob Response - Optional. Job is a Hadoop job.
- hive
Job HiveJob Response - Optional. Job is a Hive job.
- job
Uuid string - A UUID that uniquely identifies a job within the project over time. This is in contrast to a user-settable reference.job_id that may be reused over time.
- labels {[key: string]: string}
- Optional. The labels to associate with this job. Label keys must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt). Label values may be empty, but, if present, must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt). No more than 32 labels can be associated with a job.
- pig
Job PigJob Response - Optional. Job is a Pig job.
- placement
Job
Placement Response - Job information, including how, when, and where to run the job.
- presto
Job PrestoJob Response - Optional. Job is a Presto job.
- pyspark
Job PySpark Job Response - Optional. Job is a PySpark job.
- reference
Job
Reference Response - Optional. The fully qualified reference to the job, which can be used to obtain the equivalent REST path of the job resource. If this property is not specified when a job is created, the server generates a job_id.
- scheduling
Job
Scheduling Response - Optional. Job scheduling configuration.
- spark
Job SparkJob Response - Optional. Job is a Spark job.
- spark
RJob SparkRJob Response - Optional. Job is a SparkR job.
- spark
Sql SparkJob Sql Job Response - Optional. Job is a SparkSql job.
- status
Job
Status Response - The job status. Additional application-specific status information may be contained in the type_job and yarn_applications fields.
- status
History JobStatus Response[] - The previous job status.
- submitted
By string - The email address of the user submitting the job. For jobs submitted on the cluster, the address is username@hostname.
- yarn
Applications YarnApplication Response[] - The collection of YARN applications spun up by this job.Beta Feature: This report is available for testing purposes only. It may be changed before final release.
- done bool
- Indicates whether the job is completed. If the value is false, the job is still in progress. If true, the job is completed, and status.state field will indicate if it was successful, failed, or cancelled.
- driver_
control_ strfiles_ uri - If present, the location of miscellaneous control files which may be used as part of job setup and handling. If not present, control files may be placed in the same location as driver_output_uri.
- driver_
output_ strresource_ uri - A URI pointing to the location of the stdout of the job's driver program.
- hadoop_
job HadoopJob Response - Optional. Job is a Hadoop job.
- hive_
job HiveJob Response - Optional. Job is a Hive job.
- job_
uuid str - A UUID that uniquely identifies a job within the project over time. This is in contrast to a user-settable reference.job_id that may be reused over time.
- labels Mapping[str, str]
- Optional. The labels to associate with this job. Label keys must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt). Label values may be empty, but, if present, must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt). No more than 32 labels can be associated with a job.
- pig_
job PigJob Response - Optional. Job is a Pig job.
- placement
Job
Placement Response - Job information, including how, when, and where to run the job.
- presto_
job PrestoJob Response - Optional. Job is a Presto job.
- pyspark_
job PySpark Job Response - Optional. Job is a PySpark job.
- reference
Job
Reference Response - Optional. The fully qualified reference to the job, which can be used to obtain the equivalent REST path of the job resource. If this property is not specified when a job is created, the server generates a job_id.
- scheduling
Job
Scheduling Response - Optional. Job scheduling configuration.
- spark_
job SparkJob Response - Optional. Job is a Spark job.
- spark_
r_ Sparkjob RJob Response - Optional. Job is a SparkR job.
- spark_
sql_ Sparkjob Sql Job Response - Optional. Job is a SparkSql job.
- status
Job
Status Response - The job status. Additional application-specific status information may be contained in the type_job and yarn_applications fields.
- status_
history Sequence[JobStatus Response] - The previous job status.
- submitted_
by str - The email address of the user submitting the job. For jobs submitted on the cluster, the address is username@hostname.
- yarn_
applications Sequence[YarnApplication Response] - The collection of YARN applications spun up by this job.Beta Feature: This report is available for testing purposes only. It may be changed before final release.
- done Boolean
- Indicates whether the job is completed. If the value is false, the job is still in progress. If true, the job is completed, and status.state field will indicate if it was successful, failed, or cancelled.
- driver
Control StringFiles Uri - If present, the location of miscellaneous control files which may be used as part of job setup and handling. If not present, control files may be placed in the same location as driver_output_uri.
- driver
Output StringResource Uri - A URI pointing to the location of the stdout of the job's driver program.
- hadoop
Job Property Map - Optional. Job is a Hadoop job.
- hive
Job Property Map - Optional. Job is a Hive job.
- job
Uuid String - A UUID that uniquely identifies a job within the project over time. This is in contrast to a user-settable reference.job_id that may be reused over time.
- labels Map<String>
- Optional. The labels to associate with this job. Label keys must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt). Label values may be empty, but, if present, must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt). No more than 32 labels can be associated with a job.
- pig
Job Property Map - Optional. Job is a Pig job.
- placement Property Map
- Job information, including how, when, and where to run the job.
- presto
Job Property Map - Optional. Job is a Presto job.
- pyspark
Job Property Map - Optional. Job is a PySpark job.
- reference Property Map
- Optional. The fully qualified reference to the job, which can be used to obtain the equivalent REST path of the job resource. If this property is not specified when a job is created, the server generates a job_id.
- scheduling Property Map
- Optional. Job scheduling configuration.
- spark
Job Property Map - Optional. Job is a Spark job.
- spark
RJob Property Map - Optional. Job is a SparkR job.
- spark
Sql Property MapJob - Optional. Job is a SparkSql job.
- status Property Map
- The job status. Additional application-specific status information may be contained in the type_job and yarn_applications fields.
- status
History List<Property Map> - The previous job status.
- submitted
By String - The email address of the user submitting the job. For jobs submitted on the cluster, the address is username@hostname.
- yarn
Applications List<Property Map> - The collection of YARN applications spun up by this job.Beta Feature: This report is available for testing purposes only. It may be changed before final release.
Supporting Types
HadoopJobResponse
- Archive
Uris This property is required. List<string> - Optional. HCFS URIs of archives to be extracted in the working directory of Hadoop drivers and tasks. Supported file types: .jar, .tar, .tar.gz, .tgz, or .zip.
- Args
This property is required. List<string> - Optional. The arguments to pass to the driver. Do not include arguments, such as -libjars or -Dfoo=bar, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
- File
Uris This property is required. List<string> - Optional. HCFS (Hadoop Compatible Filesystem) URIs of files to be copied to the working directory of Hadoop drivers and distributed tasks. Useful for naively parallel tasks.
- Jar
File Uris This property is required. List<string> - Optional. Jar file URIs to add to the CLASSPATHs of the Hadoop driver and tasks.
- Logging
Config This property is required. Pulumi.Google Native. Dataproc. V1Beta2. Inputs. Logging Config Response - Optional. The runtime log config for job execution.
- Main
Class This property is required. string - The name of the driver's main class. The jar file containing the class must be in the default CLASSPATH or specified in jar_file_uris.
- Main
Jar File Uri This property is required. string - The HCFS URI of the jar file containing the main class. Examples: 'gs://foo-bucket/analytics-binaries/extract-useful-metrics-mr.jar' 'hdfs:/tmp/test-samples/custom-wordcount.jar' 'file:///home/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar'
- Properties
This property is required. Dictionary<string, string> - Optional. A mapping of property names to values, used to configure Hadoop. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site and classes in user code.
- Archive
Uris This property is required. []string - Optional. HCFS URIs of archives to be extracted in the working directory of Hadoop drivers and tasks. Supported file types: .jar, .tar, .tar.gz, .tgz, or .zip.
- Args
This property is required. []string - Optional. The arguments to pass to the driver. Do not include arguments, such as -libjars or -Dfoo=bar, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
- File
Uris This property is required. []string - Optional. HCFS (Hadoop Compatible Filesystem) URIs of files to be copied to the working directory of Hadoop drivers and distributed tasks. Useful for naively parallel tasks.
- Jar
File Uris This property is required. []string - Optional. Jar file URIs to add to the CLASSPATHs of the Hadoop driver and tasks.
- Logging
Config This property is required. LoggingConfig Response - Optional. The runtime log config for job execution.
- Main
Class This property is required. string - The name of the driver's main class. The jar file containing the class must be in the default CLASSPATH or specified in jar_file_uris.
- Main
Jar File Uri This property is required. string - The HCFS URI of the jar file containing the main class. Examples: 'gs://foo-bucket/analytics-binaries/extract-useful-metrics-mr.jar' 'hdfs:/tmp/test-samples/custom-wordcount.jar' 'file:///home/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar'
- Properties
This property is required. map[string]string - Optional. A mapping of property names to values, used to configure Hadoop. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site and classes in user code.
- archive
Uris This property is required. List<String> - Optional. HCFS URIs of archives to be extracted in the working directory of Hadoop drivers and tasks. Supported file types: .jar, .tar, .tar.gz, .tgz, or .zip.
- args
This property is required. List<String> - Optional. The arguments to pass to the driver. Do not include arguments, such as -libjars or -Dfoo=bar, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
- file
Uris This property is required. List<String> - Optional. HCFS (Hadoop Compatible Filesystem) URIs of files to be copied to the working directory of Hadoop drivers and distributed tasks. Useful for naively parallel tasks.
- jar
File Uris This property is required. List<String> - Optional. Jar file URIs to add to the CLASSPATHs of the Hadoop driver and tasks.
- logging
Config This property is required. LoggingConfig Response - Optional. The runtime log config for job execution.
- main
Class This property is required. String - The name of the driver's main class. The jar file containing the class must be in the default CLASSPATH or specified in jar_file_uris.
- main
Jar File Uri This property is required. String - The HCFS URI of the jar file containing the main class. Examples: 'gs://foo-bucket/analytics-binaries/extract-useful-metrics-mr.jar' 'hdfs:/tmp/test-samples/custom-wordcount.jar' 'file:///home/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar'
- properties
This property is required. Map<String,String> - Optional. A mapping of property names to values, used to configure Hadoop. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site and classes in user code.
- archive
Uris This property is required. string[] - Optional. HCFS URIs of archives to be extracted in the working directory of Hadoop drivers and tasks. Supported file types: .jar, .tar, .tar.gz, .tgz, or .zip.
- args
This property is required. string[] - Optional. The arguments to pass to the driver. Do not include arguments, such as -libjars or -Dfoo=bar, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
- file
Uris This property is required. string[] - Optional. HCFS (Hadoop Compatible Filesystem) URIs of files to be copied to the working directory of Hadoop drivers and distributed tasks. Useful for naively parallel tasks.
- jar
File Uris This property is required. string[] - Optional. Jar file URIs to add to the CLASSPATHs of the Hadoop driver and tasks.
- logging
Config This property is required. LoggingConfig Response - Optional. The runtime log config for job execution.
- main
Class This property is required. string - The name of the driver's main class. The jar file containing the class must be in the default CLASSPATH or specified in jar_file_uris.
- main
Jar File Uri This property is required. string - The HCFS URI of the jar file containing the main class. Examples: 'gs://foo-bucket/analytics-binaries/extract-useful-metrics-mr.jar' 'hdfs:/tmp/test-samples/custom-wordcount.jar' 'file:///home/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar'
- properties
This property is required. {[key: string]: string} - Optional. A mapping of property names to values, used to configure Hadoop. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site and classes in user code.
- archive_
uris This property is required. Sequence[str] - Optional. HCFS URIs of archives to be extracted in the working directory of Hadoop drivers and tasks. Supported file types: .jar, .tar, .tar.gz, .tgz, or .zip.
- args
This property is required. Sequence[str] - Optional. The arguments to pass to the driver. Do not include arguments, such as -libjars or -Dfoo=bar, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
- file_
uris This property is required. Sequence[str] - Optional. HCFS (Hadoop Compatible Filesystem) URIs of files to be copied to the working directory of Hadoop drivers and distributed tasks. Useful for naively parallel tasks.
- jar_
file_ uris This property is required. Sequence[str] - Optional. Jar file URIs to add to the CLASSPATHs of the Hadoop driver and tasks.
- logging_
config This property is required. LoggingConfig Response - Optional. The runtime log config for job execution.
- main_
class This property is required. str - The name of the driver's main class. The jar file containing the class must be in the default CLASSPATH or specified in jar_file_uris.
- main_
jar_ file_ uri This property is required. str - The HCFS URI of the jar file containing the main class. Examples: 'gs://foo-bucket/analytics-binaries/extract-useful-metrics-mr.jar' 'hdfs:/tmp/test-samples/custom-wordcount.jar' 'file:///home/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar'
- properties
This property is required. Mapping[str, str] - Optional. A mapping of property names to values, used to configure Hadoop. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site and classes in user code.
- archive
Uris This property is required. List<String> - Optional. HCFS URIs of archives to be extracted in the working directory of Hadoop drivers and tasks. Supported file types: .jar, .tar, .tar.gz, .tgz, or .zip.
- args
This property is required. List<String> - Optional. The arguments to pass to the driver. Do not include arguments, such as -libjars or -Dfoo=bar, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
- file
Uris This property is required. List<String> - Optional. HCFS (Hadoop Compatible Filesystem) URIs of files to be copied to the working directory of Hadoop drivers and distributed tasks. Useful for naively parallel tasks.
- jar
File Uris This property is required. List<String> - Optional. Jar file URIs to add to the CLASSPATHs of the Hadoop driver and tasks.
- logging
Config This property is required. Property Map - Optional. The runtime log config for job execution.
- main
Class This property is required. String - The name of the driver's main class. The jar file containing the class must be in the default CLASSPATH or specified in jar_file_uris.
- main
Jar File Uri This property is required. String - The HCFS URI of the jar file containing the main class. Examples: 'gs://foo-bucket/analytics-binaries/extract-useful-metrics-mr.jar' 'hdfs:/tmp/test-samples/custom-wordcount.jar' 'file:///home/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar'
- properties
This property is required. Map<String> - Optional. A mapping of property names to values, used to configure Hadoop. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site and classes in user code.
HiveJobResponse
- Continue
On Failure This property is required. bool - Optional. Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries.
- Jar
File Uris This property is required. List<string> - Optional. HCFS URIs of jar files to add to the CLASSPATH of the Hive server and Hadoop MapReduce (MR) tasks. Can contain Hive SerDes and UDFs.
- Properties
This property is required. Dictionary<string, string> - Optional. A mapping of property names and values, used to configure Hive. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site.xml, /etc/hive/conf/hive-site.xml, and classes in user code.
- Query
File Uri This property is required. string - The HCFS URI of the script that contains Hive queries.
- Query
List This property is required. Pulumi.Google Native. Dataproc. V1Beta2. Inputs. Query List Response - A list of queries.
- Script
Variables This property is required. Dictionary<string, string> - Optional. Mapping of query variable names to values (equivalent to the Hive command: SET name="value";).
- Continue
On Failure This property is required. bool - Optional. Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries.
- Jar
File Uris This property is required. []string - Optional. HCFS URIs of jar files to add to the CLASSPATH of the Hive server and Hadoop MapReduce (MR) tasks. Can contain Hive SerDes and UDFs.
- Properties
This property is required. map[string]string - Optional. A mapping of property names and values, used to configure Hive. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site.xml, /etc/hive/conf/hive-site.xml, and classes in user code.
- Query
File Uri This property is required. string - The HCFS URI of the script that contains Hive queries.
- Query
List This property is required. QueryList Response - A list of queries.
- Script
Variables This property is required. map[string]string - Optional. Mapping of query variable names to values (equivalent to the Hive command: SET name="value";).
- continue
On Failure This property is required. Boolean - Optional. Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries.
- jar
File Uris This property is required. List<String> - Optional. HCFS URIs of jar files to add to the CLASSPATH of the Hive server and Hadoop MapReduce (MR) tasks. Can contain Hive SerDes and UDFs.
- properties
This property is required. Map<String,String> - Optional. A mapping of property names and values, used to configure Hive. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site.xml, /etc/hive/conf/hive-site.xml, and classes in user code.
- query
File Uri This property is required. String - The HCFS URI of the script that contains Hive queries.
- query
List This property is required. QueryList Response - A list of queries.
- script
Variables This property is required. Map<String,String> - Optional. Mapping of query variable names to values (equivalent to the Hive command: SET name="value";).
- continue
On Failure This property is required. boolean - Optional. Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries.
- jar
File Uris This property is required. string[] - Optional. HCFS URIs of jar files to add to the CLASSPATH of the Hive server and Hadoop MapReduce (MR) tasks. Can contain Hive SerDes and UDFs.
- properties
This property is required. {[key: string]: string} - Optional. A mapping of property names and values, used to configure Hive. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site.xml, /etc/hive/conf/hive-site.xml, and classes in user code.
- query
File Uri This property is required. string - The HCFS URI of the script that contains Hive queries.
- query
List This property is required. QueryList Response - A list of queries.
- script
Variables This property is required. {[key: string]: string} - Optional. Mapping of query variable names to values (equivalent to the Hive command: SET name="value";).
- continue_
on_ failure This property is required. bool - Optional. Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries.
- jar_
file_ uris This property is required. Sequence[str] - Optional. HCFS URIs of jar files to add to the CLASSPATH of the Hive server and Hadoop MapReduce (MR) tasks. Can contain Hive SerDes and UDFs.
- properties
This property is required. Mapping[str, str] - Optional. A mapping of property names and values, used to configure Hive. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site.xml, /etc/hive/conf/hive-site.xml, and classes in user code.
- query_
file_ uri This property is required. str - The HCFS URI of the script that contains Hive queries.
- query_
list This property is required. QueryList Response - A list of queries.
- script_
variables This property is required. Mapping[str, str] - Optional. Mapping of query variable names to values (equivalent to the Hive command: SET name="value";).
- continue
On Failure This property is required. Boolean - Optional. Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries.
- jar
File Uris This property is required. List<String> - Optional. HCFS URIs of jar files to add to the CLASSPATH of the Hive server and Hadoop MapReduce (MR) tasks. Can contain Hive SerDes and UDFs.
- properties
This property is required. Map<String> - Optional. A mapping of property names and values, used to configure Hive. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site.xml, /etc/hive/conf/hive-site.xml, and classes in user code.
- query
File Uri This property is required. String - The HCFS URI of the script that contains Hive queries.
- query
List This property is required. Property Map - A list of queries.
- script
Variables This property is required. Map<String> - Optional. Mapping of query variable names to values (equivalent to the Hive command: SET name="value";).
JobPlacementResponse
- Cluster
Labels This property is required. Dictionary<string, string> - Optional. Cluster labels to identify a cluster where the job will be submitted.
- Cluster
Name This property is required. string - The name of the cluster where the job will be submitted.
- Cluster
Uuid This property is required. string - A cluster UUID generated by the Dataproc service when the job is submitted.
- Cluster
Labels This property is required. map[string]string - Optional. Cluster labels to identify a cluster where the job will be submitted.
- Cluster
Name This property is required. string - The name of the cluster where the job will be submitted.
- Cluster
Uuid This property is required. string - A cluster UUID generated by the Dataproc service when the job is submitted.
- cluster
Labels This property is required. Map<String,String> - Optional. Cluster labels to identify a cluster where the job will be submitted.
- cluster
Name This property is required. String - The name of the cluster where the job will be submitted.
- cluster
Uuid This property is required. String - A cluster UUID generated by the Dataproc service when the job is submitted.
- cluster
Labels This property is required. {[key: string]: string} - Optional. Cluster labels to identify a cluster where the job will be submitted.
- cluster
Name This property is required. string - The name of the cluster where the job will be submitted.
- cluster
Uuid This property is required. string - A cluster UUID generated by the Dataproc service when the job is submitted.
- cluster_
labels This property is required. Mapping[str, str] - Optional. Cluster labels to identify a cluster where the job will be submitted.
- cluster_
name This property is required. str - The name of the cluster where the job will be submitted.
- cluster_
uuid This property is required. str - A cluster UUID generated by the Dataproc service when the job is submitted.
- cluster
Labels This property is required. Map<String> - Optional. Cluster labels to identify a cluster where the job will be submitted.
- cluster
Name This property is required. String - The name of the cluster where the job will be submitted.
- cluster
Uuid This property is required. String - A cluster UUID generated by the Dataproc service when the job is submitted.
JobReferenceResponse
- Job
Id This property is required. string - Optional. The job ID, which must be unique within the project. The ID must contain only letters (a-z, A-Z), numbers (0-9), underscores (_), or hyphens (-). The maximum length is 100 characters.If not specified by the caller, the job ID will be provided by the server.
- Project
This property is required. string - Optional. The ID of the Google Cloud Platform project that the job belongs to. If specified, must match the request project ID.
- Job
Id This property is required. string - Optional. The job ID, which must be unique within the project. The ID must contain only letters (a-z, A-Z), numbers (0-9), underscores (_), or hyphens (-). The maximum length is 100 characters.If not specified by the caller, the job ID will be provided by the server.
- Project
This property is required. string - Optional. The ID of the Google Cloud Platform project that the job belongs to. If specified, must match the request project ID.
- job
Id This property is required. String - Optional. The job ID, which must be unique within the project. The ID must contain only letters (a-z, A-Z), numbers (0-9), underscores (_), or hyphens (-). The maximum length is 100 characters.If not specified by the caller, the job ID will be provided by the server.
- project
This property is required. String - Optional. The ID of the Google Cloud Platform project that the job belongs to. If specified, must match the request project ID.
- job
Id This property is required. string - Optional. The job ID, which must be unique within the project. The ID must contain only letters (a-z, A-Z), numbers (0-9), underscores (_), or hyphens (-). The maximum length is 100 characters.If not specified by the caller, the job ID will be provided by the server.
- project
This property is required. string - Optional. The ID of the Google Cloud Platform project that the job belongs to. If specified, must match the request project ID.
- job_
id This property is required. str - Optional. The job ID, which must be unique within the project. The ID must contain only letters (a-z, A-Z), numbers (0-9), underscores (_), or hyphens (-). The maximum length is 100 characters.If not specified by the caller, the job ID will be provided by the server.
- project
This property is required. str - Optional. The ID of the Google Cloud Platform project that the job belongs to. If specified, must match the request project ID.
- job
Id This property is required. String - Optional. The job ID, which must be unique within the project. The ID must contain only letters (a-z, A-Z), numbers (0-9), underscores (_), or hyphens (-). The maximum length is 100 characters.If not specified by the caller, the job ID will be provided by the server.
- project
This property is required. String - Optional. The ID of the Google Cloud Platform project that the job belongs to. If specified, must match the request project ID.
JobSchedulingResponse
- Max
Failures Per Hour This property is required. int - Optional. Maximum number of times per hour a driver may be restarted as a result of driver terminating with non-zero code before job is reported failed.A job may be reported as thrashing if driver exits with non-zero code 4 times within 10 minute window.Maximum value is 10.
- Max
Failures Total This property is required. int - Optional. Maximum number of times in total a driver may be restarted as a result of driver exiting with non-zero code before job is reported failed. Maximum value is 240.
- Max
Failures Per Hour This property is required. int - Optional. Maximum number of times per hour a driver may be restarted as a result of driver terminating with non-zero code before job is reported failed.A job may be reported as thrashing if driver exits with non-zero code 4 times within 10 minute window.Maximum value is 10.
- Max
Failures Total This property is required. int - Optional. Maximum number of times in total a driver may be restarted as a result of driver exiting with non-zero code before job is reported failed. Maximum value is 240.
- max
Failures Per Hour This property is required. Integer - Optional. Maximum number of times per hour a driver may be restarted as a result of driver terminating with non-zero code before job is reported failed.A job may be reported as thrashing if driver exits with non-zero code 4 times within 10 minute window.Maximum value is 10.
- max
Failures Total This property is required. Integer - Optional. Maximum number of times in total a driver may be restarted as a result of driver exiting with non-zero code before job is reported failed. Maximum value is 240.
- max
Failures Per Hour This property is required. number - Optional. Maximum number of times per hour a driver may be restarted as a result of driver terminating with non-zero code before job is reported failed.A job may be reported as thrashing if driver exits with non-zero code 4 times within 10 minute window.Maximum value is 10.
- max
Failures Total This property is required. number - Optional. Maximum number of times in total a driver may be restarted as a result of driver exiting with non-zero code before job is reported failed. Maximum value is 240.
- max_
failures_ per_ hour This property is required. int - Optional. Maximum number of times per hour a driver may be restarted as a result of driver terminating with non-zero code before job is reported failed.A job may be reported as thrashing if driver exits with non-zero code 4 times within 10 minute window.Maximum value is 10.
- max_
failures_ total This property is required. int - Optional. Maximum number of times in total a driver may be restarted as a result of driver exiting with non-zero code before job is reported failed. Maximum value is 240.
- max
Failures Per Hour This property is required. Number - Optional. Maximum number of times per hour a driver may be restarted as a result of driver terminating with non-zero code before job is reported failed.A job may be reported as thrashing if driver exits with non-zero code 4 times within 10 minute window.Maximum value is 10.
- max
Failures Total This property is required. Number - Optional. Maximum number of times in total a driver may be restarted as a result of driver exiting with non-zero code before job is reported failed. Maximum value is 240.
JobStatusResponse
- Details
This property is required. string - Optional Job state details, such as an error description if the state is ERROR.
- State
This property is required. string - A state message specifying the overall job state.
- State
Start Time This property is required. string - The time when this state was entered.
- Substate
This property is required. string - Additional state information, which includes status reported by the agent.
- Details
This property is required. string - Optional Job state details, such as an error description if the state is ERROR.
- State
This property is required. string - A state message specifying the overall job state.
- State
Start Time This property is required. string - The time when this state was entered.
- Substate
This property is required. string - Additional state information, which includes status reported by the agent.
- details
This property is required. String - Optional Job state details, such as an error description if the state is ERROR.
- state
This property is required. String - A state message specifying the overall job state.
- state
Start Time This property is required. String - The time when this state was entered.
- substate
This property is required. String - Additional state information, which includes status reported by the agent.
- details
This property is required. string - Optional Job state details, such as an error description if the state is ERROR.
- state
This property is required. string - A state message specifying the overall job state.
- state
Start Time This property is required. string - The time when this state was entered.
- substate
This property is required. string - Additional state information, which includes status reported by the agent.
- details
This property is required. str - Optional Job state details, such as an error description if the state is ERROR.
- state
This property is required. str - A state message specifying the overall job state.
- state_
start_ time This property is required. str - The time when this state was entered.
- substate
This property is required. str - Additional state information, which includes status reported by the agent.
- details
This property is required. String - Optional Job state details, such as an error description if the state is ERROR.
- state
This property is required. String - A state message specifying the overall job state.
- state
Start Time This property is required. String - The time when this state was entered.
- substate
This property is required. String - Additional state information, which includes status reported by the agent.
LoggingConfigResponse
- Driver
Log Levels This property is required. Dictionary<string, string> - The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
- Driver
Log Levels This property is required. map[string]string - The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
- driver
Log Levels This property is required. Map<String,String> - The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
- driver
Log Levels This property is required. {[key: string]: string} - The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
- driver_
log_ levels This property is required. Mapping[str, str] - The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
- driver
Log Levels This property is required. Map<String> - The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
PigJobResponse
- Continue
On Failure This property is required. bool - Optional. Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries.
- Jar
File Uris This property is required. List<string> - Optional. HCFS URIs of jar files to add to the CLASSPATH of the Pig Client and Hadoop MapReduce (MR) tasks. Can contain Pig UDFs.
- Logging
Config This property is required. Pulumi.Google Native. Dataproc. V1Beta2. Inputs. Logging Config Response - Optional. The runtime log config for job execution.
- Properties
This property is required. Dictionary<string, string> - Optional. A mapping of property names to values, used to configure Pig. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site.xml, /etc/pig/conf/pig.properties, and classes in user code.
- Query
File Uri This property is required. string - The HCFS URI of the script that contains the Pig queries.
- Query
List This property is required. Pulumi.Google Native. Dataproc. V1Beta2. Inputs. Query List Response - A list of queries.
- Script
Variables This property is required. Dictionary<string, string> - Optional. Mapping of query variable names to values (equivalent to the Pig command: name=[value]).
- Continue
On Failure This property is required. bool - Optional. Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries.
- Jar
File Uris This property is required. []string - Optional. HCFS URIs of jar files to add to the CLASSPATH of the Pig Client and Hadoop MapReduce (MR) tasks. Can contain Pig UDFs.
- Logging
Config This property is required. LoggingConfig Response - Optional. The runtime log config for job execution.
- Properties
This property is required. map[string]string - Optional. A mapping of property names to values, used to configure Pig. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site.xml, /etc/pig/conf/pig.properties, and classes in user code.
- Query
File Uri This property is required. string - The HCFS URI of the script that contains the Pig queries.
- Query
List This property is required. QueryList Response - A list of queries.
- Script
Variables This property is required. map[string]string - Optional. Mapping of query variable names to values (equivalent to the Pig command: name=[value]).
- continue
On Failure This property is required. Boolean - Optional. Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries.
- jar
File Uris This property is required. List<String> - Optional. HCFS URIs of jar files to add to the CLASSPATH of the Pig Client and Hadoop MapReduce (MR) tasks. Can contain Pig UDFs.
- logging
Config This property is required. LoggingConfig Response - Optional. The runtime log config for job execution.
- properties
This property is required. Map<String,String> - Optional. A mapping of property names to values, used to configure Pig. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site.xml, /etc/pig/conf/pig.properties, and classes in user code.
- query
File Uri This property is required. String - The HCFS URI of the script that contains the Pig queries.
- query
List This property is required. QueryList Response - A list of queries.
- script
Variables This property is required. Map<String,String> - Optional. Mapping of query variable names to values (equivalent to the Pig command: name=[value]).
- continue
On Failure This property is required. boolean - Optional. Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries.
- jar
File Uris This property is required. string[] - Optional. HCFS URIs of jar files to add to the CLASSPATH of the Pig Client and Hadoop MapReduce (MR) tasks. Can contain Pig UDFs.
- logging
Config This property is required. LoggingConfig Response - Optional. The runtime log config for job execution.
- properties
This property is required. {[key: string]: string} - Optional. A mapping of property names to values, used to configure Pig. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site.xml, /etc/pig/conf/pig.properties, and classes in user code.
- query
File Uri This property is required. string - The HCFS URI of the script that contains the Pig queries.
- query
List This property is required. QueryList Response - A list of queries.
- script
Variables This property is required. {[key: string]: string} - Optional. Mapping of query variable names to values (equivalent to the Pig command: name=[value]).
- continue_
on_ failure This property is required. bool - Optional. Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries.
- jar_
file_ uris This property is required. Sequence[str] - Optional. HCFS URIs of jar files to add to the CLASSPATH of the Pig Client and Hadoop MapReduce (MR) tasks. Can contain Pig UDFs.
- logging_
config This property is required. LoggingConfig Response - Optional. The runtime log config for job execution.
- properties
This property is required. Mapping[str, str] - Optional. A mapping of property names to values, used to configure Pig. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site.xml, /etc/pig/conf/pig.properties, and classes in user code.
- query_
file_ uri This property is required. str - The HCFS URI of the script that contains the Pig queries.
- query_
list This property is required. QueryList Response - A list of queries.
- script_
variables This property is required. Mapping[str, str] - Optional. Mapping of query variable names to values (equivalent to the Pig command: name=[value]).
- continue
On Failure This property is required. Boolean - Optional. Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries.
- jar
File Uris This property is required. List<String> - Optional. HCFS URIs of jar files to add to the CLASSPATH of the Pig Client and Hadoop MapReduce (MR) tasks. Can contain Pig UDFs.
- logging
Config This property is required. Property Map - Optional. The runtime log config for job execution.
- properties
This property is required. Map<String> - Optional. A mapping of property names to values, used to configure Pig. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site.xml, /etc/pig/conf/pig.properties, and classes in user code.
- query
File Uri This property is required. String - The HCFS URI of the script that contains the Pig queries.
- query
List This property is required. Property Map - A list of queries.
- script
Variables This property is required. Map<String> - Optional. Mapping of query variable names to values (equivalent to the Pig command: name=[value]).
PrestoJobResponse
This property is required. List<string>- Optional. Presto client tags to attach to this query
- Continue
On Failure This property is required. bool - Optional. Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries.
- Logging
Config This property is required. Pulumi.Google Native. Dataproc. V1Beta2. Inputs. Logging Config Response - Optional. The runtime log config for job execution.
- Output
Format This property is required. string - Optional. The format in which query output will be displayed. See the Presto documentation for supported output formats
- Properties
This property is required. Dictionary<string, string> - Optional. A mapping of property names to values. Used to set Presto session properties (https://prestodb.io/docs/current/sql/set-session.html) Equivalent to using the --session flag in the Presto CLI
- Query
File Uri This property is required. string - The HCFS URI of the script that contains SQL queries.
- Query
List This property is required. Pulumi.Google Native. Dataproc. V1Beta2. Inputs. Query List Response - A list of queries.
This property is required. []string- Optional. Presto client tags to attach to this query
- Continue
On Failure This property is required. bool - Optional. Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries.
- Logging
Config This property is required. LoggingConfig Response - Optional. The runtime log config for job execution.
- Output
Format This property is required. string - Optional. The format in which query output will be displayed. See the Presto documentation for supported output formats
- Properties
This property is required. map[string]string - Optional. A mapping of property names to values. Used to set Presto session properties (https://prestodb.io/docs/current/sql/set-session.html) Equivalent to using the --session flag in the Presto CLI
- Query
File Uri This property is required. string - The HCFS URI of the script that contains SQL queries.
- Query
List This property is required. QueryList Response - A list of queries.
This property is required. List<String>- Optional. Presto client tags to attach to this query
- continue
On Failure This property is required. Boolean - Optional. Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries.
- logging
Config This property is required. LoggingConfig Response - Optional. The runtime log config for job execution.
- output
Format This property is required. String - Optional. The format in which query output will be displayed. See the Presto documentation for supported output formats
- properties
This property is required. Map<String,String> - Optional. A mapping of property names to values. Used to set Presto session properties (https://prestodb.io/docs/current/sql/set-session.html) Equivalent to using the --session flag in the Presto CLI
- query
File Uri This property is required. String - The HCFS URI of the script that contains SQL queries.
- query
List This property is required. QueryList Response - A list of queries.
This property is required. string[]- Optional. Presto client tags to attach to this query
- continue
On Failure This property is required. boolean - Optional. Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries.
- logging
Config This property is required. LoggingConfig Response - Optional. The runtime log config for job execution.
- output
Format This property is required. string - Optional. The format in which query output will be displayed. See the Presto documentation for supported output formats
- properties
This property is required. {[key: string]: string} - Optional. A mapping of property names to values. Used to set Presto session properties (https://prestodb.io/docs/current/sql/set-session.html) Equivalent to using the --session flag in the Presto CLI
- query
File Uri This property is required. string - The HCFS URI of the script that contains SQL queries.
- query
List This property is required. QueryList Response - A list of queries.
This property is required. Sequence[str]- Optional. Presto client tags to attach to this query
- continue_
on_ failure This property is required. bool - Optional. Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries.
- logging_
config This property is required. LoggingConfig Response - Optional. The runtime log config for job execution.
- output_
format This property is required. str - Optional. The format in which query output will be displayed. See the Presto documentation for supported output formats
- properties
This property is required. Mapping[str, str] - Optional. A mapping of property names to values. Used to set Presto session properties (https://prestodb.io/docs/current/sql/set-session.html) Equivalent to using the --session flag in the Presto CLI
- query_
file_ uri This property is required. str - The HCFS URI of the script that contains SQL queries.
- query_
list This property is required. QueryList Response - A list of queries.
This property is required. List<String>- Optional. Presto client tags to attach to this query
- continue
On Failure This property is required. Boolean - Optional. Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries.
- logging
Config This property is required. Property Map - Optional. The runtime log config for job execution.
- output
Format This property is required. String - Optional. The format in which query output will be displayed. See the Presto documentation for supported output formats
- properties
This property is required. Map<String> - Optional. A mapping of property names to values. Used to set Presto session properties (https://prestodb.io/docs/current/sql/set-session.html) Equivalent to using the --session flag in the Presto CLI
- query
File Uri This property is required. String - The HCFS URI of the script that contains SQL queries.
- query
List This property is required. Property Map - A list of queries.
PySparkJobResponse
- Archive
Uris This property is required. List<string> - Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.
- Args
This property is required. List<string> - Optional. The arguments to pass to the driver. Do not include arguments, such as --conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
- File
Uris This property is required. List<string> - Optional. HCFS URIs of files to be placed in the working directory of each executor. Useful for naively parallel tasks.
- Jar
File Uris This property is required. List<string> - Optional. HCFS URIs of jar files to add to the CLASSPATHs of the Python driver and tasks.
- Logging
Config This property is required. Pulumi.Google Native. Dataproc. V1Beta2. Inputs. Logging Config Response - Optional. The runtime log config for job execution.
- Main
Python File Uri This property is required. string - The HCFS URI of the main Python file to use as the driver. Must be a .py file.
- Properties
This property is required. Dictionary<string, string> - Optional. A mapping of property names to values, used to configure PySpark. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.
- Python
File Uris This property is required. List<string> - Optional. HCFS file URIs of Python files to pass to the PySpark framework. Supported file types: .py, .egg, and .zip.
- Archive
Uris This property is required. []string - Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.
- Args
This property is required. []string - Optional. The arguments to pass to the driver. Do not include arguments, such as --conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
- File
Uris This property is required. []string - Optional. HCFS URIs of files to be placed in the working directory of each executor. Useful for naively parallel tasks.
- Jar
File Uris This property is required. []string - Optional. HCFS URIs of jar files to add to the CLASSPATHs of the Python driver and tasks.
- Logging
Config This property is required. LoggingConfig Response - Optional. The runtime log config for job execution.
- Main
Python File Uri This property is required. string - The HCFS URI of the main Python file to use as the driver. Must be a .py file.
- Properties
This property is required. map[string]string - Optional. A mapping of property names to values, used to configure PySpark. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.
- Python
File Uris This property is required. []string - Optional. HCFS file URIs of Python files to pass to the PySpark framework. Supported file types: .py, .egg, and .zip.
- archive
Uris This property is required. List<String> - Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.
- args
This property is required. List<String> - Optional. The arguments to pass to the driver. Do not include arguments, such as --conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
- file
Uris This property is required. List<String> - Optional. HCFS URIs of files to be placed in the working directory of each executor. Useful for naively parallel tasks.
- jar
File Uris This property is required. List<String> - Optional. HCFS URIs of jar files to add to the CLASSPATHs of the Python driver and tasks.
- logging
Config This property is required. LoggingConfig Response - Optional. The runtime log config for job execution.
- main
Python File Uri This property is required. String - The HCFS URI of the main Python file to use as the driver. Must be a .py file.
- properties
This property is required. Map<String,String> - Optional. A mapping of property names to values, used to configure PySpark. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.
- python
File Uris This property is required. List<String> - Optional. HCFS file URIs of Python files to pass to the PySpark framework. Supported file types: .py, .egg, and .zip.
- archive
Uris This property is required. string[] - Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.
- args
This property is required. string[] - Optional. The arguments to pass to the driver. Do not include arguments, such as --conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
- file
Uris This property is required. string[] - Optional. HCFS URIs of files to be placed in the working directory of each executor. Useful for naively parallel tasks.
- jar
File Uris This property is required. string[] - Optional. HCFS URIs of jar files to add to the CLASSPATHs of the Python driver and tasks.
- logging
Config This property is required. LoggingConfig Response - Optional. The runtime log config for job execution.
- main
Python File Uri This property is required. string - The HCFS URI of the main Python file to use as the driver. Must be a .py file.
- properties
This property is required. {[key: string]: string} - Optional. A mapping of property names to values, used to configure PySpark. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.
- python
File Uris This property is required. string[] - Optional. HCFS file URIs of Python files to pass to the PySpark framework. Supported file types: .py, .egg, and .zip.
- archive_
uris This property is required. Sequence[str] - Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.
- args
This property is required. Sequence[str] - Optional. The arguments to pass to the driver. Do not include arguments, such as --conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
- file_
uris This property is required. Sequence[str] - Optional. HCFS URIs of files to be placed in the working directory of each executor. Useful for naively parallel tasks.
- jar_
file_ uris This property is required. Sequence[str] - Optional. HCFS URIs of jar files to add to the CLASSPATHs of the Python driver and tasks.
- logging_
config This property is required. LoggingConfig Response - Optional. The runtime log config for job execution.
- main_
python_ file_ uri This property is required. str - The HCFS URI of the main Python file to use as the driver. Must be a .py file.
- properties
This property is required. Mapping[str, str] - Optional. A mapping of property names to values, used to configure PySpark. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.
- python_
file_ uris This property is required. Sequence[str] - Optional. HCFS file URIs of Python files to pass to the PySpark framework. Supported file types: .py, .egg, and .zip.
- archive
Uris This property is required. List<String> - Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.
- args
This property is required. List<String> - Optional. The arguments to pass to the driver. Do not include arguments, such as --conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
- file
Uris This property is required. List<String> - Optional. HCFS URIs of files to be placed in the working directory of each executor. Useful for naively parallel tasks.
- jar
File Uris This property is required. List<String> - Optional. HCFS URIs of jar files to add to the CLASSPATHs of the Python driver and tasks.
- logging
Config This property is required. Property Map - Optional. The runtime log config for job execution.
- main
Python File Uri This property is required. String - The HCFS URI of the main Python file to use as the driver. Must be a .py file.
- properties
This property is required. Map<String> - Optional. A mapping of property names to values, used to configure PySpark. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.
- python
File Uris This property is required. List<String> - Optional. HCFS file URIs of Python files to pass to the PySpark framework. Supported file types: .py, .egg, and .zip.
QueryListResponse
- Queries
This property is required. List<string> - The queries to execute. You do not need to end a query expression with a semicolon. Multiple queries can be specified in one string by separating each with a semicolon. Here is an example of a Dataproc API snippet that uses a QueryList to specify a HiveJob: "hiveJob": { "queryList": { "queries": [ "query1", "query2", "query3;query4", ] } }
- Queries
This property is required. []string - The queries to execute. You do not need to end a query expression with a semicolon. Multiple queries can be specified in one string by separating each with a semicolon. Here is an example of a Dataproc API snippet that uses a QueryList to specify a HiveJob: "hiveJob": { "queryList": { "queries": [ "query1", "query2", "query3;query4", ] } }
- queries
This property is required. List<String> - The queries to execute. You do not need to end a query expression with a semicolon. Multiple queries can be specified in one string by separating each with a semicolon. Here is an example of a Dataproc API snippet that uses a QueryList to specify a HiveJob: "hiveJob": { "queryList": { "queries": [ "query1", "query2", "query3;query4", ] } }
- queries
This property is required. string[] - The queries to execute. You do not need to end a query expression with a semicolon. Multiple queries can be specified in one string by separating each with a semicolon. Here is an example of a Dataproc API snippet that uses a QueryList to specify a HiveJob: "hiveJob": { "queryList": { "queries": [ "query1", "query2", "query3;query4", ] } }
- queries
This property is required. Sequence[str] - The queries to execute. You do not need to end a query expression with a semicolon. Multiple queries can be specified in one string by separating each with a semicolon. Here is an example of a Dataproc API snippet that uses a QueryList to specify a HiveJob: "hiveJob": { "queryList": { "queries": [ "query1", "query2", "query3;query4", ] } }
- queries
This property is required. List<String> - The queries to execute. You do not need to end a query expression with a semicolon. Multiple queries can be specified in one string by separating each with a semicolon. Here is an example of a Dataproc API snippet that uses a QueryList to specify a HiveJob: "hiveJob": { "queryList": { "queries": [ "query1", "query2", "query3;query4", ] } }
SparkJobResponse
- Archive
Uris This property is required. List<string> - Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.
- Args
This property is required. List<string> - Optional. The arguments to pass to the driver. Do not include arguments, such as --conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
- File
Uris This property is required. List<string> - Optional. HCFS URIs of files to be placed in the working directory of each executor. Useful for naively parallel tasks.
- Jar
File Uris This property is required. List<string> - Optional. HCFS URIs of jar files to add to the CLASSPATHs of the Spark driver and tasks.
- Logging
Config This property is required. Pulumi.Google Native. Dataproc. V1Beta2. Inputs. Logging Config Response - Optional. The runtime log config for job execution.
- Main
Class This property is required. string - The name of the driver's main class. The jar file that contains the class must be in the default CLASSPATH or specified in jar_file_uris.
- Main
Jar File Uri This property is required. string - The HCFS URI of the jar file that contains the main class.
- Properties
This property is required. Dictionary<string, string> - Optional. A mapping of property names to values, used to configure Spark. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.
- Archive
Uris This property is required. []string - Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.
- Args
This property is required. []string - Optional. The arguments to pass to the driver. Do not include arguments, such as --conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
- File
Uris This property is required. []string - Optional. HCFS URIs of files to be placed in the working directory of each executor. Useful for naively parallel tasks.
- Jar
File Uris This property is required. []string - Optional. HCFS URIs of jar files to add to the CLASSPATHs of the Spark driver and tasks.
- Logging
Config This property is required. LoggingConfig Response - Optional. The runtime log config for job execution.
- Main
Class This property is required. string - The name of the driver's main class. The jar file that contains the class must be in the default CLASSPATH or specified in jar_file_uris.
- Main
Jar File Uri This property is required. string - The HCFS URI of the jar file that contains the main class.
- Properties
This property is required. map[string]string - Optional. A mapping of property names to values, used to configure Spark. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.
- archive
Uris This property is required. List<String> - Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.
- args
This property is required. List<String> - Optional. The arguments to pass to the driver. Do not include arguments, such as --conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
- file
Uris This property is required. List<String> - Optional. HCFS URIs of files to be placed in the working directory of each executor. Useful for naively parallel tasks.
- jar
File Uris This property is required. List<String> - Optional. HCFS URIs of jar files to add to the CLASSPATHs of the Spark driver and tasks.
- logging
Config This property is required. LoggingConfig Response - Optional. The runtime log config for job execution.
- main
Class This property is required. String - The name of the driver's main class. The jar file that contains the class must be in the default CLASSPATH or specified in jar_file_uris.
- main
Jar File Uri This property is required. String - The HCFS URI of the jar file that contains the main class.
- properties
This property is required. Map<String,String> - Optional. A mapping of property names to values, used to configure Spark. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.
- archive
Uris This property is required. string[] - Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.
- args
This property is required. string[] - Optional. The arguments to pass to the driver. Do not include arguments, such as --conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
- file
Uris This property is required. string[] - Optional. HCFS URIs of files to be placed in the working directory of each executor. Useful for naively parallel tasks.
- jar
File Uris This property is required. string[] - Optional. HCFS URIs of jar files to add to the CLASSPATHs of the Spark driver and tasks.
- logging
Config This property is required. LoggingConfig Response - Optional. The runtime log config for job execution.
- main
Class This property is required. string - The name of the driver's main class. The jar file that contains the class must be in the default CLASSPATH or specified in jar_file_uris.
- main
Jar File Uri This property is required. string - The HCFS URI of the jar file that contains the main class.
- properties
This property is required. {[key: string]: string} - Optional. A mapping of property names to values, used to configure Spark. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.
- archive_
uris This property is required. Sequence[str] - Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.
- args
This property is required. Sequence[str] - Optional. The arguments to pass to the driver. Do not include arguments, such as --conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
- file_
uris This property is required. Sequence[str] - Optional. HCFS URIs of files to be placed in the working directory of each executor. Useful for naively parallel tasks.
- jar_
file_ uris This property is required. Sequence[str] - Optional. HCFS URIs of jar files to add to the CLASSPATHs of the Spark driver and tasks.
- logging_
config This property is required. LoggingConfig Response - Optional. The runtime log config for job execution.
- main_
class This property is required. str - The name of the driver's main class. The jar file that contains the class must be in the default CLASSPATH or specified in jar_file_uris.
- main_
jar_ file_ uri This property is required. str - The HCFS URI of the jar file that contains the main class.
- properties
This property is required. Mapping[str, str] - Optional. A mapping of property names to values, used to configure Spark. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.
- archive
Uris This property is required. List<String> - Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.
- args
This property is required. List<String> - Optional. The arguments to pass to the driver. Do not include arguments, such as --conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
- file
Uris This property is required. List<String> - Optional. HCFS URIs of files to be placed in the working directory of each executor. Useful for naively parallel tasks.
- jar
File Uris This property is required. List<String> - Optional. HCFS URIs of jar files to add to the CLASSPATHs of the Spark driver and tasks.
- logging
Config This property is required. Property Map - Optional. The runtime log config for job execution.
- main
Class This property is required. String - The name of the driver's main class. The jar file that contains the class must be in the default CLASSPATH or specified in jar_file_uris.
- main
Jar File Uri This property is required. String - The HCFS URI of the jar file that contains the main class.
- properties
This property is required. Map<String> - Optional. A mapping of property names to values, used to configure Spark. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.
SparkRJobResponse
- Archive
Uris This property is required. List<string> - Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.
- Args
This property is required. List<string> - Optional. The arguments to pass to the driver. Do not include arguments, such as --conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
- File
Uris This property is required. List<string> - Optional. HCFS URIs of files to be placed in the working directory of each executor. Useful for naively parallel tasks.
- Logging
Config This property is required. Pulumi.Google Native. Dataproc. V1Beta2. Inputs. Logging Config Response - Optional. The runtime log config for job execution.
- Main
RFile Uri This property is required. string - The HCFS URI of the main R file to use as the driver. Must be a .R file.
- Properties
This property is required. Dictionary<string, string> - Optional. A mapping of property names to values, used to configure SparkR. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.
- Archive
Uris This property is required. []string - Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.
- Args
This property is required. []string - Optional. The arguments to pass to the driver. Do not include arguments, such as --conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
- File
Uris This property is required. []string - Optional. HCFS URIs of files to be placed in the working directory of each executor. Useful for naively parallel tasks.
- Logging
Config This property is required. LoggingConfig Response - Optional. The runtime log config for job execution.
- Main
RFile Uri This property is required. string - The HCFS URI of the main R file to use as the driver. Must be a .R file.
- Properties
This property is required. map[string]string - Optional. A mapping of property names to values, used to configure SparkR. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.
- archive
Uris This property is required. List<String> - Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.
- args
This property is required. List<String> - Optional. The arguments to pass to the driver. Do not include arguments, such as --conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
- file
Uris This property is required. List<String> - Optional. HCFS URIs of files to be placed in the working directory of each executor. Useful for naively parallel tasks.
- logging
Config This property is required. LoggingConfig Response - Optional. The runtime log config for job execution.
- main
RFile Uri This property is required. String - The HCFS URI of the main R file to use as the driver. Must be a .R file.
- properties
This property is required. Map<String,String> - Optional. A mapping of property names to values, used to configure SparkR. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.
- archive
Uris This property is required. string[] - Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.
- args
This property is required. string[] - Optional. The arguments to pass to the driver. Do not include arguments, such as --conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
- file
Uris This property is required. string[] - Optional. HCFS URIs of files to be placed in the working directory of each executor. Useful for naively parallel tasks.
- logging
Config This property is required. LoggingConfig Response - Optional. The runtime log config for job execution.
- main
RFile Uri This property is required. string - The HCFS URI of the main R file to use as the driver. Must be a .R file.
- properties
This property is required. {[key: string]: string} - Optional. A mapping of property names to values, used to configure SparkR. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.
- archive_
uris This property is required. Sequence[str] - Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.
- args
This property is required. Sequence[str] - Optional. The arguments to pass to the driver. Do not include arguments, such as --conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
- file_
uris This property is required. Sequence[str] - Optional. HCFS URIs of files to be placed in the working directory of each executor. Useful for naively parallel tasks.
- logging_
config This property is required. LoggingConfig Response - Optional. The runtime log config for job execution.
- main_
r_ file_ uri This property is required. str - The HCFS URI of the main R file to use as the driver. Must be a .R file.
- properties
This property is required. Mapping[str, str] - Optional. A mapping of property names to values, used to configure SparkR. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.
- archive
Uris This property is required. List<String> - Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.
- args
This property is required. List<String> - Optional. The arguments to pass to the driver. Do not include arguments, such as --conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
- file
Uris This property is required. List<String> - Optional. HCFS URIs of files to be placed in the working directory of each executor. Useful for naively parallel tasks.
- logging
Config This property is required. Property Map - Optional. The runtime log config for job execution.
- main
RFile Uri This property is required. String - The HCFS URI of the main R file to use as the driver. Must be a .R file.
- properties
This property is required. Map<String> - Optional. A mapping of property names to values, used to configure SparkR. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.
SparkSqlJobResponse
- Jar
File Uris This property is required. List<string> - Optional. HCFS URIs of jar files to be added to the Spark CLASSPATH.
- Logging
Config This property is required. Pulumi.Google Native. Dataproc. V1Beta2. Inputs. Logging Config Response - Optional. The runtime log config for job execution.
- Properties
This property is required. Dictionary<string, string> - Optional. A mapping of property names to values, used to configure Spark SQL's SparkConf. Properties that conflict with values set by the Dataproc API may be overwritten.
- Query
File Uri This property is required. string - The HCFS URI of the script that contains SQL queries.
- Query
List This property is required. Pulumi.Google Native. Dataproc. V1Beta2. Inputs. Query List Response - A list of queries.
- Script
Variables This property is required. Dictionary<string, string> - Optional. Mapping of query variable names to values (equivalent to the Spark SQL command: SET name="value";).
- Jar
File Uris This property is required. []string - Optional. HCFS URIs of jar files to be added to the Spark CLASSPATH.
- Logging
Config This property is required. LoggingConfig Response - Optional. The runtime log config for job execution.
- Properties
This property is required. map[string]string - Optional. A mapping of property names to values, used to configure Spark SQL's SparkConf. Properties that conflict with values set by the Dataproc API may be overwritten.
- Query
File Uri This property is required. string - The HCFS URI of the script that contains SQL queries.
- Query
List This property is required. QueryList Response - A list of queries.
- Script
Variables This property is required. map[string]string - Optional. Mapping of query variable names to values (equivalent to the Spark SQL command: SET name="value";).
- jar
File Uris This property is required. List<String> - Optional. HCFS URIs of jar files to be added to the Spark CLASSPATH.
- logging
Config This property is required. LoggingConfig Response - Optional. The runtime log config for job execution.
- properties
This property is required. Map<String,String> - Optional. A mapping of property names to values, used to configure Spark SQL's SparkConf. Properties that conflict with values set by the Dataproc API may be overwritten.
- query
File Uri This property is required. String - The HCFS URI of the script that contains SQL queries.
- query
List This property is required. QueryList Response - A list of queries.
- script
Variables This property is required. Map<String,String> - Optional. Mapping of query variable names to values (equivalent to the Spark SQL command: SET name="value";).
- jar
File Uris This property is required. string[] - Optional. HCFS URIs of jar files to be added to the Spark CLASSPATH.
- logging
Config This property is required. LoggingConfig Response - Optional. The runtime log config for job execution.
- properties
This property is required. {[key: string]: string} - Optional. A mapping of property names to values, used to configure Spark SQL's SparkConf. Properties that conflict with values set by the Dataproc API may be overwritten.
- query
File Uri This property is required. string - The HCFS URI of the script that contains SQL queries.
- query
List This property is required. QueryList Response - A list of queries.
- script
Variables This property is required. {[key: string]: string} - Optional. Mapping of query variable names to values (equivalent to the Spark SQL command: SET name="value";).
- jar_
file_ uris This property is required. Sequence[str] - Optional. HCFS URIs of jar files to be added to the Spark CLASSPATH.
- logging_
config This property is required. LoggingConfig Response - Optional. The runtime log config for job execution.
- properties
This property is required. Mapping[str, str] - Optional. A mapping of property names to values, used to configure Spark SQL's SparkConf. Properties that conflict with values set by the Dataproc API may be overwritten.
- query_
file_ uri This property is required. str - The HCFS URI of the script that contains SQL queries.
- query_
list This property is required. QueryList Response - A list of queries.
- script_
variables This property is required. Mapping[str, str] - Optional. Mapping of query variable names to values (equivalent to the Spark SQL command: SET name="value";).
- jar
File Uris This property is required. List<String> - Optional. HCFS URIs of jar files to be added to the Spark CLASSPATH.
- logging
Config This property is required. Property Map - Optional. The runtime log config for job execution.
- properties
This property is required. Map<String> - Optional. A mapping of property names to values, used to configure Spark SQL's SparkConf. Properties that conflict with values set by the Dataproc API may be overwritten.
- query
File Uri This property is required. String - The HCFS URI of the script that contains SQL queries.
- query
List This property is required. Property Map - A list of queries.
- script
Variables This property is required. Map<String> - Optional. Mapping of query variable names to values (equivalent to the Spark SQL command: SET name="value";).
YarnApplicationResponse
- Name
This property is required. string - The application name.
- Progress
This property is required. double - The numerical progress of the application, from 1 to 100.
- State
This property is required. string - The application state.
- Tracking
Url This property is required. string - The HTTP URL of the ApplicationMaster, HistoryServer, or TimelineServer that provides application-specific information. The URL uses the internal hostname, and requires a proxy server for resolution and, possibly, access.
- Name
This property is required. string - The application name.
- Progress
This property is required. float64 - The numerical progress of the application, from 1 to 100.
- State
This property is required. string - The application state.
- Tracking
Url This property is required. string - The HTTP URL of the ApplicationMaster, HistoryServer, or TimelineServer that provides application-specific information. The URL uses the internal hostname, and requires a proxy server for resolution and, possibly, access.
- name
This property is required. String - The application name.
- progress
This property is required. Double - The numerical progress of the application, from 1 to 100.
- state
This property is required. String - The application state.
- tracking
Url This property is required. String - The HTTP URL of the ApplicationMaster, HistoryServer, or TimelineServer that provides application-specific information. The URL uses the internal hostname, and requires a proxy server for resolution and, possibly, access.
- name
This property is required. string - The application name.
- progress
This property is required. number - The numerical progress of the application, from 1 to 100.
- state
This property is required. string - The application state.
- tracking
Url This property is required. string - The HTTP URL of the ApplicationMaster, HistoryServer, or TimelineServer that provides application-specific information. The URL uses the internal hostname, and requires a proxy server for resolution and, possibly, access.
- name
This property is required. str - The application name.
- progress
This property is required. float - The numerical progress of the application, from 1 to 100.
- state
This property is required. str - The application state.
- tracking_
url This property is required. str - The HTTP URL of the ApplicationMaster, HistoryServer, or TimelineServer that provides application-specific information. The URL uses the internal hostname, and requires a proxy server for resolution and, possibly, access.
- name
This property is required. String - The application name.
- progress
This property is required. Number - The numerical progress of the application, from 1 to 100.
- state
This property is required. String - The application state.
- tracking
Url This property is required. String - The HTTP URL of the ApplicationMaster, HistoryServer, or TimelineServer that provides application-specific information. The URL uses the internal hostname, and requires a proxy server for resolution and, possibly, access.
Package Details
- Repository
- Google Cloud Native pulumi/pulumi-google-native
- License
- Apache-2.0
Google Cloud Native is in preview. Google Cloud Classic is fully supported.
Google Cloud Native v0.32.0 published on Wednesday, Nov 29, 2023 by Pulumi