Class/Object

com.krux.hyperion.activity

HiveActivity

Related Docs: object HiveActivity | package activity

Permalink

case class HiveActivity[A <: EmrCluster] extends EmrTaskActivity[A] with Product with Serializable

Runs a Hive query on an Amazon EMR cluster. HiveActivity makes it easier to set up an Amzon EMR activity and automatically creates Hive tables based on input data coming in from either Amazon S3 or Amazon RDS. All you need to specify is the HiveQL to run on the source data. AWS Data Pipeline automatically creates Hive tables with ${input1}, ${input2}, etc. based on the input fields in the Hive Activity object. For S3 inputs, the dataFormat field is used to create the Hive column names. For MySQL (RDS) inputs, the column names for the SQL query are used to create the Hive column names.

Source
HiveActivity.scala
Linear Supertypes
Serializable, Serializable, Product, Equals, EmrTaskActivity[A], EmrActivity[A], PipelineActivity[A], NamedPipelineObject, PipelineObject, Ordered[PipelineObject], Comparable[PipelineObject], AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. HiveActivity
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. EmrTaskActivity
  7. EmrActivity
  8. PipelineActivity
  9. NamedPipelineObject
  10. PipelineObject
  11. Ordered
  12. Comparable
  13. AnyRef
  14. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. type Self = HiveActivity[A]

    Permalink

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. def <(that: PipelineObject): Boolean

    Permalink
    Definition Classes
    Ordered
  4. def <=(that: PipelineObject): Boolean

    Permalink
    Definition Classes
    Ordered
  5. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  6. def >(that: PipelineObject): Boolean

    Permalink
    Definition Classes
    Ordered
  7. def >=(that: PipelineObject): Boolean

    Permalink
    Definition Classes
    Ordered
  8. val activityFields: ActivityFields[A]

    Permalink
    Definition Classes
    HiveActivityPipelineActivity
  9. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  10. def attemptTimeout: Option[HDuration]

    Permalink
    Definition Classes
    PipelineActivity
  11. val baseFields: BaseFields

    Permalink
    Definition Classes
    HiveActivityNamedPipelineObject
  12. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  13. def compare(that: PipelineObject): Int

    Permalink
    Definition Classes
    PipelineObject → Ordered
  14. def compareTo(that: PipelineObject): Int

    Permalink
    Definition Classes
    Ordered → Comparable
  15. def dependsOn: Seq[PipelineActivity[_]]

    Permalink
    Definition Classes
    PipelineActivity
  16. val emrTaskActivityFields: EmrTaskActivityFields

    Permalink
    Definition Classes
    HiveActivityEmrTaskActivity
  17. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  18. def failureAndRerunMode: Option[FailureAndRerunMode]

    Permalink
    Definition Classes
    PipelineActivity
  19. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  20. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  21. def groupedBy(group: String): Self

    Permalink

    Postfix the name field

    Postfix the name field

    Definition Classes
    NamedPipelineObject
  22. val hadoopQueue: Option[HString]

    Permalink
  23. val hiveScript: Script

    Permalink
  24. def id: PipelineObjectId

    Permalink
    Definition Classes
    NamedPipelineObjectPipelineObject
  25. def idGroupedBy(group: String): Self

    Permalink

    Have a grouping postfix in the id field

    Have a grouping postfix in the id field

    Definition Classes
    NamedPipelineObject
    Note

    Id naming is more restrictive, it is recommended to not changing the id unleass you have a good reason

  26. def idNamed(namePrefix: String): Self

    Permalink

    Id field will be prefixed with name

    Id field will be prefixed with name

    Definition Classes
    NamedPipelineObject
    Note

    Id naming is more restrictive, it is recommended to not changing the id unless you have a good reason

  27. val input: Seq[DataNode]

    Permalink
  28. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  29. def lateAfterTimeout: Option[HDuration]

    Permalink
    Definition Classes
    PipelineActivity
  30. def maxActiveInstances: Option[HInt]

    Permalink
    Definition Classes
    PipelineActivity
  31. def maximumRetries: Option[HInt]

    Permalink
    Definition Classes
    PipelineActivity
  32. def name: Option[String]

    Permalink

    Name of the pipeline object, if not set, it will defaults to

    Name of the pipeline object, if not set, it will defaults to

    Option(id)
    Definition Classes
    NamedPipelineObject
  33. def named(namePrefix: String): Self

    Permalink

    Give the object a name prefix

    Give the object a name prefix

    Definition Classes
    NamedPipelineObject
  34. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  35. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  36. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  37. def objects: Iterable[PipelineObject]

    Permalink
  38. def onFail(alarms: SnsAlarm*): Self

    Permalink
    Definition Classes
    PipelineActivity
  39. def onFailAlarms: Seq[SnsAlarm]

    Permalink
    Definition Classes
    PipelineActivity
  40. def onLateAction(alarms: SnsAlarm*): Self

    Permalink
    Definition Classes
    PipelineActivity
  41. def onLateActionAlarms: Seq[SnsAlarm]

    Permalink
    Definition Classes
    PipelineActivity
  42. def onSuccess(alarms: SnsAlarm*): Self

    Permalink
    Definition Classes
    PipelineActivity
  43. def onSuccessAlarms: Seq[SnsAlarm]

    Permalink
    Definition Classes
    PipelineActivity
  44. val output: Seq[DataNode]

    Permalink
  45. def postActivityTaskConfig: Option[ShellScriptConfig]

    Permalink
    Definition Classes
    EmrTaskActivity
  46. def preActivityTaskConfig: Option[ShellScriptConfig]

    Permalink
    Definition Classes
    EmrTaskActivity
  47. def preconditions: Seq[Precondition]

    Permalink
    Definition Classes
    PipelineActivity
  48. def ref: AdpRef[AdpActivity]

    Permalink
    Definition Classes
    PipelineActivityPipelineObject
  49. def retryDelay: Option[HDuration]

    Permalink
    Definition Classes
    PipelineActivity
  50. def runsOn: Resource[A]

    Permalink
    Definition Classes
    PipelineActivity
  51. val scriptVariables: Seq[HString]

    Permalink
  52. implicit def seq2Option[A](anySeq: Seq[A]): Option[Seq[A]]

    Permalink
    Definition Classes
    PipelineObject
  53. def seqToOption[A, B](anySeq: Seq[A])(transform: (A) ⇒ B): Option[Seq[B]]

    Permalink
    Definition Classes
    PipelineObject
  54. lazy val serialize: AdpHiveActivity

    Permalink
    Definition Classes
    HiveActivityPipelineActivityPipelineObject
  55. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  56. implicit def uniquePipelineId2String(id: PipelineObjectId): String

    Permalink
    Definition Classes
    PipelineObject
  57. def updateActivityFields(fields: ActivityFields[A]): HiveActivity[A]

    Permalink
    Definition Classes
    HiveActivityPipelineActivity
  58. def updateBaseFields(fields: BaseFields): HiveActivity[A]

    Permalink
    Definition Classes
    HiveActivityNamedPipelineObject
  59. def updateEmrTaskActivityFields(fields: EmrTaskActivityFields): HiveActivity[A]

    Permalink
    Definition Classes
    HiveActivityEmrTaskActivity
  60. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  61. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  62. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  63. def whenMet(conditions: Precondition*): Self

    Permalink
    Definition Classes
    PipelineActivity
  64. def withAttemptTimeout(duration: HDuration): Self

    Permalink
    Definition Classes
    PipelineActivity
  65. def withFailureAndRerunMode(mode: FailureAndRerunMode): Self

    Permalink
    Definition Classes
    PipelineActivity
  66. def withHadoopQueue(queue: HString): HiveActivity[A]

    Permalink
  67. def withLateAfterTimeout(duration: HDuration): Self

    Permalink
    Definition Classes
    PipelineActivity
  68. def withMaxActiveInstances(activeInstances: HInt): Self

    Permalink
    Definition Classes
    PipelineActivity
  69. def withMaximumRetries(retries: HInt): Self

    Permalink
    Definition Classes
    PipelineActivity
  70. def withPostActivityTaskConfig(config: ShellScriptConfig): Self

    Permalink
    Definition Classes
    EmrTaskActivity
  71. def withPreActivityTaskConfig(config: ShellScriptConfig): Self

    Permalink
    Definition Classes
    EmrTaskActivity
  72. def withRetryDelay(duration: HDuration): Self

    Permalink
    Definition Classes
    PipelineActivity
  73. def withScriptVariable(scriptVariable: HString*): HiveActivity[A]

    Permalink

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from EmrTaskActivity[A]

Inherited from EmrActivity[A]

Inherited from PipelineActivity[A]

Inherited from NamedPipelineObject

Inherited from PipelineObject

Inherited from Ordered[PipelineObject]

Inherited from Comparable[PipelineObject]

Inherited from AnyRef

Inherited from Any

Ungrouped