activity

Type Members

1. case class ActivityFields[A <: ResourceObject](runsOn: Resource[A], dependsOn: Seq[PipelineActivity[_]] = Seq.empty, preconditions: Seq[Precondition] = Seq.empty, onFailAlarms: Seq[SnsAlarm] = Seq.empty, onSuccessAlarms: Seq[SnsAlarm] = Seq.empty, onLateActionAlarms: Seq[SnsAlarm] = Seq.empty, maximumRetries: Option[HInt] = None, attemptTimeout: Option[HDuration] = None, lateAfterTimeout: Option[HDuration] = None, retryDelay: Option[HDuration] = None, failureAndRerunMode: Option[FailureAndRerunMode] = None, maxActiveInstances: Option[HInt] = None) extends Product with Serializable

   

2. case class AwsS3CpActivity extends BaseShellCommandActivity with WithS3Input with Product with Serializable

   

3. trait BaseShellCommandActivity extends PipelineActivity[Ec2Resource]

   

4. case class CopyActivity extends PipelineActivity[Ec2Resource] with Product with Serializable

   

   The activity that copies data from one data node to the other.

   The activity that copies data from one data node to the other.

   Note

   it seems that both input and output format needs to be in CsvDataFormat for this copy to work properly and it needs to be a specific variance of the CSV, for more information check the web page: http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-object-copyactivity.html From our experience it's really hard to export using TsvDataFormat, in both import and export especially for tasks involving RedshiftCopyActivity. A general rule of thumb is always use default CsvDataFormat for tasks involving both exporting to S3 and copy to redshift.

5. case class DeleteS3PathActivity extends BaseShellCommandActivity with Product with Serializable

   

   Activity to recursively delete files in an S3 path.

6. trait EmrActivity[A <: EmrCluster] extends PipelineActivity[A]

   

   The base trait for activities that run on an Amazon EMR cluster

7. trait EmrTaskActivity[A <: EmrCluster] extends EmrActivity[A]

   

8. case class EmrTaskActivityFields(preActivityTaskConfig: Option[ShellScriptConfig] = None, postActivityTaskConfig: Option[ShellScriptConfig] = None) extends Product with Serializable

   

9. trait FailureAndRerunMode extends AnyRef

   

10. trait GoogleStorageActivity extends BaseShellCommandActivity

    

11. case class GoogleStorageDownloadActivity extends GoogleStorageActivity with WithS3Output with Product with Serializable

    

    Google Storage Download activity

12. case class GoogleStorageUploadActivity extends GoogleStorageActivity with WithS3Input with Product with Serializable

    

    Google Storage Upload activity

13. case class HadoopActivity[A <: EmrCluster] extends EmrTaskActivity[A] with Product with Serializable

    

    Runs a MapReduce job on a cluster.

    Runs a MapReduce job on a cluster. The cluster can be an EMR cluster managed by AWS Data Pipeline or another resource if you use TaskRunner. Use HadoopActivity when you want to run work in parallel. This allows you to use the scheduling resources of the YARN framework or the MapReduce resource negotiator in Hadoop 1. If you would like to run work sequentially using the Amazon EMR Step action, you can still use EmrActivity.

14. case class HiveActivity[A <: EmrCluster] extends EmrTaskActivity[A] with Product with Serializable

    

    Runs a Hive query on an Amazon EMR cluster.

    Runs a Hive query on an Amazon EMR cluster. HiveActivity makes it easier to set up an Amzon EMR activity and automatically creates Hive tables based on input data coming in from either Amazon S3 or Amazon RDS. All you need to specify is the HiveQL to run on the source data. AWS Data Pipeline automatically creates Hive tables with ${input1}, ${input2}, etc. based on the input fields in the Hive Activity object. For S3 inputs, the dataFormat field is used to create the Hive column names. For MySQL (RDS) inputs, the column names for the SQL query are used to create the Hive column names.

15. case class HiveCopyActivity[A <: EmrCluster] extends EmrTaskActivity[A] with Product with Serializable

    

    Runs a Hive query on an Amazon EMR cluster.

    Runs a Hive query on an Amazon EMR cluster. HiveCopyActivity makes it easier to copy data between Amazon S3 and DynamoDB. HiveCopyActivity accepts a HiveQL statement to filter input data from Amazon S3 or DynomoDB at the column and row level.

16. case class JarActivity extends BaseShellCommandActivity with WithS3Input with WithS3Output with Product with Serializable

    

    Shell command activity that runs a given Jar

17. class MainClass extends AnyRef

    

18. case class MapReduceActivity[A <: EmrCluster] extends EmrActivity[A] with Product with Serializable

    

    Runs map reduce steps on an Amazon EMR cluster

19. case class MapReduceStep extends Product with Serializable

    

    A MapReduce step that runs on MapReduce Cluster

20. trait PgpActivity extends BaseShellCommandActivity with WithS3Input with WithS3Output

    

    Base trait for server-side PGP encryption activities.

21. case class PgpDecryptActivity extends PgpActivity with Product with Serializable

    

    The server-side PGP decryption activity decrypts files from the input location to the output location using the private decryption key.

22. case class PgpEncryptActivity extends PgpActivity with Product with Serializable

    

    The server-side PGP encryption activity encrypts files from the input location to the output location using the public encryption key.

23. case class PigActivity[A <: EmrCluster] extends EmrTaskActivity[A] with Product with Serializable

    

    PigActivity provides native support for Pig scripts in AWS Data Pipeline without the requirement to use ShellCommandActivity or EmrActivity.

    PigActivity provides native support for Pig scripts in AWS Data Pipeline without the requirement to use ShellCommandActivity or EmrActivity. In addition, PigActivity supports data staging. When the stage field is set to true, AWS Data Pipeline stages the input data as a schema in Pig without additional code from the user.

24. trait PipelineActivity[A <: ResourceObject] extends NamedPipelineObject

    

    The activity trait.

    The activity trait. All activities should mixin this trait.

25. case class PythonActivity extends BaseShellCommandActivity with WithS3Input with WithS3Output with Product with Serializable

    

    Shell command activity that runs a given python script

26. case class RedshiftCopyActivity extends PipelineActivity[Ec2Resource] with Product with Serializable

    

    Copies data directly from DynamoDB or Amazon S3 to Amazon Redshift.

    Copies data directly from DynamoDB or Amazon S3 to Amazon Redshift. You can load data into a new table, or easily merge data into an existing table.

27. case class RedshiftCopyOption(repr: Seq[String]) extends Product with Serializable

    

28. case class RedshiftUnloadActivity extends PipelineActivity[Ec2Resource] with Product with Serializable

    

    Unload result of the given sql script from redshift to given s3Path.

29. trait RedshiftUnloadOption extends AnyRef

    

30. case class S3DistCpActivity[A <: EmrCluster] extends EmrActivity[A] with Product with Serializable

    

31. case class S3DistCpActivityFields(source: Option[HString], dest: Option[HString], sourcePattern: Option[HString], groupBy: Option[HString], targetSize: Option[HInt], appendLastToFile: HBoolean, outputCodec: OutputCodec, s3ServerSideEncryption: HBoolean, deleteOnSuccess: HBoolean, disableMultipartUpload: HBoolean, chunkSize: Option[HInt], numberFiles: HBoolean, startingIndex: Option[HInt], outputManifest: Option[HString], previousManifest: Option[HString], requirePreviousManifest: HBoolean, copyFromManifest: HBoolean, endpoint: Option[HString], storageClass: Option[StorageClass], sourcePrefixesFile: Option[HString]) extends Product with Serializable

    

32. sealed trait Script extends AnyRef

    

33. sealed case class ScriptContent(content: Option[HString]) extends Script with Product with Serializable

    

34. sealed case class ScriptUri(uri: Option[HS3Uri]) extends Script with Product with Serializable

    

35. case class SendEmailActivity extends BaseShellCommandActivity with WithS3Input with Product with Serializable

    

36. case class SendFlowdockMessageActivity extends BaseShellCommandActivity with Product with Serializable

    

37. case class SendSlackMessageActivity extends BaseShellCommandActivity with Product with Serializable

    

38. case class SendSnsMessageActivity extends BaseShellCommandActivity with Product with Serializable

    

39. case class SendSqsMessageActivity extends BaseShellCommandActivity with Product with Serializable

    

40. case class SetS3AclActivity extends BaseShellCommandActivity with Product with Serializable

    

41. trait SftpActivity extends BaseShellCommandActivity

    

42. case class SftpActivityFields(host: HString, port: Option[HInt] = None, username: Option[HString] = None, password: Option[EncryptedParameter[String]] = None, identity: Option[HS3Uri] = None, pattern: Option[HString] = None, sinceDate: Option[HDateTime] = None, untilDate: Option[HDateTime] = None, skipEmpty: HBoolean = false, markSuccessfulJobs: HBoolean = false) extends Product with Serializable

    

43. case class SftpDownloadActivity extends SftpActivity with WithS3Output with Product with Serializable

    

    Activity that downloads from an SFTP endpoint into an S3 endpoint.

44. case class SftpUploadActivity extends SftpActivity with WithS3Input with Product with Serializable

    

    Activity that uploads from an S3 endpoint to an SFTP endpoint.

45. case class ShellCommandActivity extends BaseShellCommandActivity with WithS3Input with WithS3Output with Product with Serializable

    

    Runs a command or script

46. case class ShellCommandActivityFields(script: Script, scriptArguments: Seq[HString] = Seq.empty, stdout: Option[HString] = None, stderr: Option[HString] = None, stage: Option[HBoolean] = None, input: Seq[S3DataNode] = Seq.empty, output: Seq[S3DataNode] = Seq.empty) extends Product with Serializable

    

47. case class ShellScriptConfig(baseFields: BaseFields, scriptUri: HS3Uri, scriptArguments: Seq[HString]) extends NamedPipelineObject with Product with Serializable

    

48. case class SparkActivity extends EmrActivity[SparkCluster] with Product with Serializable

    

    Runs spark steps on given spark cluster with Amazon EMR

49. case class SparkStep extends Product with Serializable

    

    A Spark step that runs on Spark Cluster

50. case class SparkTaskActivity extends EmrTaskActivity[SparkCluster] with Product with Serializable

    

    Runs a Spark job on a cluster.

    Runs a Spark job on a cluster. The cluster can be an EMR cluster managed by AWS Data Pipeline or another resource if you use TaskRunner. Use SparkActivity when you want to run work in parallel. This allows you to use the scheduling resources of the YARN framework or the MapReduce resource negotiator in Hadoop 1. If you would like to run work sequentially using the Amazon EMR Step action, you can still use SparkActivity.

51. case class SplitMergeFilesActivity extends BaseShellCommandActivity with WithS3Input with WithS3Output with Product with Serializable

    

52. case class SqlActivity extends PipelineActivity[Ec2Resource] with Product with Serializable

    

    Runs an SQL query on a RedShift cluster.

    Runs an SQL query on a RedShift cluster. If the query writes out to a table that does not exist, a new table with that name is created.

53. trait WithS3Input extends AnyRef

    

54. trait WithS3Output extends AnyRef