Blog Details

bt_bb_section_bottom_section_coverage_image

Continuous Streaming Data Ingestion

The customer requires continuous ingestion of a social media feed into Amazon S3. For this purpose, they utilize Amazon Kinesis Data Firehose to seamlessly ingest data without the need for dedicated servers. To set up the data migration process, the customer follows these steps in the Amazon Kinesis Data Firehose console:

  1. Creation of an Amazon Kinesis Data Firehose Delivery Stream involves: a. Selection of a Delivery Stream name. b. Selection of the Amazon S3 bucket where data will be stored, along with specifying the IAM role granting Firehose access to the bucket. c. Configuration of Firehose buffer settings to control data delivery frequency. Options include setting a Buffer Size (ranging from 1 to 128 MBs) or a Buffer Interval (ranging from 60 to 900 seconds), with data being delivered to Amazon S3 when either condition is met. d. Selection of one of three compression formats (GZIP, ZIP, or SNAPPY), or choosing not to compress the data. e. Choice of data encryption options, including encryption with a key from the customer’s list of AWS Key Management Service (AWS KMS) keys.

The customer implements code using the AWS SDK to send streaming data to the Amazon Kinesis Firehose delivery stream.

Leave a Reply

Your email address will not be published. Required fields are marked *