Artikel

AWS Kinesis Firehose and Teradata Vantage

Many Teradata customers are interested in integrating Vantage with Amazon AWS First Party Services. This Getting Started Guide will help you to connect Vantage with AWS Kinesis service.

Wenjie Tehan
Wenjie Tehan
21. September 2021 3 min Lesezeit
AWS and Teradata

Many Teradata customers are interested in integrating Teradata Vantage with Amazon AWS First Party Services. This Getting Started Guide will help you to connect Teradata Vantage with AWS Kinesis service. 

Although this approach has been implemented and tested internally, it is offered on an as-is basis. Neither AWS nor Teradata provide validation of Teradata Vantage with AWS services. 

We encourage your feedback. We want to understand what you found useful and how we can improve this guide.  

Please send your feedback to shamira.joshua@teradata.com and wenjie.tehan@teradata.com

Disclaimer: This guide includes content from both AWS and Teradata product documentation. 

Overview 

AWS Kinesis is a streaming service that makes it easy to collect, process, and analyze real-time, streaming data. 

Kinesis streaming data platform offers Kinesis Data Streams, Kinesis Data Firehose, Kinesis Data Analytics, and Kinesis Video Streams. Kinesis Data Streams is manually managed and can store data in the stream for up to seven days, in which transformation can be done with the data. Kinesis Firehose is fully managed, and collects the data and stores it in Amazon S3, Redshift, Splunk and Elasticsearch. Kinesis Video streams is used to stream live video and Kinesis Data Analytics can process and analyze streaming data using standard SQL. 

Teradata Vantage Native Object Store (NOS) makes it easy for users to explore data in external object stores like Amazon S3 using standard SQL and application interfaces like ODBC, JDBC, .NET, Python and R native drivers. No special object storage-side compute infrastructure is required to use NOS. You can explore data located in Amazon S3 bucket by simply creating a NOS table definition to point to the bucket you are authorized to access.  

This guide describes the process to stream data from source to Amazon S3 via AWS Kinesis firehose,  transform it to JSON format by an AWS Glue ETL job, and then use Teradata NOS to access data from Amazon S3. Lambda functions and a CloudWatch event rule is also created to automate the whole process. 
Picture1-(1).png

Prerequisites

You are expected to be familiar with AWS Kinesis, Lambda, CloudWatch services, and Teradata Vantage.
You will need the following accounts, and systems:

•    An AWS account
•    A Teradata Vantage instance with SQLE 17.0+
•    An Amazon S3 bucket to store streaming data
•    An Amazon S3 bucket to store JSON files
•    IAM roles that allow Glue Crawler, ETL and Lambda services
•    AccessKeyId and SecretAccessKey

Getting Started

Create Amazon S3 buckets
Amazon S3 buckets can be created using instructions here. Two buckets are needed in this example: one to store streaming data (i.e., ptctstoutput), and another one to store JSON files (i.e., awspilbucket) after transformation.

Create IAM role
AWS services require you to use roles to allow the service to access resource in other services on your behalf. In this example, three roles are needed – a role for Kinesis Firehose, a role for Glue, and a role for Lambda. 
Kinesis Firehose role will be created on the fly. Instructions below create roles for Glue and Lambda services.
Screen-Shot-2021-09-23-at-9-29-08-AM.pngScreen-Shot-2021-09-23-at-9-29-46-AM.png



Screen-Shot-2021-09-23-at-9-31-48-AM.png

Screen-Shot-2021-09-23-at-9-22-59-AM.png
Screen-Shot-2021-09-23-at-9-20-43-AM.png
Screen-Shot-2021-09-23-at-9-21-45-AM.png

Create Firehose Delivery System

Screen-Shot-2021-09-23-at-9-25-03-AM.pngScreen-Shot-2021-09-23-at-9-25-29-AM.pngScreen-Shot-2021-09-23-at-9-25-51-AM.pngScreen-Shot-2021-09-23-at-9-26-17-AM.pngScreen-Shot-2021-09-23-at-9-26-45-AM.png

Screen-Shot-2021-09-23-at-9-32-47-AM.pngScreen-Shot-2021-09-23-at-9-33-12-AM.pngCreate Glue ETL Transformation Job

Screen-Shot-2021-09-23-at-9-33-58-AM.pngScreen-Shot-2021-09-23-at-9-35-51-AM.pngScreen-Shot-2021-09-23-at-9-36-28-AM.pngScreen-Shot-2021-09-23-at-9-36-52-AM.pngScreen-Shot-2021-09-23-at-9-37-16-AM.pngScreen-Shot-2021-09-23-at-9-37-39-AM.pngAccessing Streaming Data Using NOS

Screen-Shot-2021-09-23-at-9-38-32-AM.pngScreen-Shot-2021-09-23-at-9-39-05-AM.pngScreen-Shot-2021-09-23-at-9-39-42-AM.pngScreen-Shot-2021-09-23-at-9-40-05-AM.pngCreate Lambda functions, Trigger, and CloudWatch event

Screen-Shot-2021-09-23-at-9-41-02-AM.pngScreen-Shot-2021-09-23-at-9-42-17-AM.pngScreen-Shot-2021-09-23-at-9-42-41-AM.pngScreen-Shot-2021-09-23-at-9-43-02-AM.pngScreen-Shot-2021-09-23-at-9-43-33-AM.pngScreen-Shot-2021-09-23-at-9-44-00-AM.pngScreen-Shot-2021-09-23-at-9-44-25-AM.pngScreen-Shot-2021-09-23-at-9-45-05-AM.pngRun

Screen-Shot-2021-09-23-at-9-45-39-AM.pngScreen-Shot-2021-09-23-at-9-46-02-AM.png
Tags

Über Wenjie Tehan

Wenjie is a Technical Consulting Manager, currently working with the Teradata Global Alliances team. 
 
With over 20 years in the IT industry, Wenjie has worked as developer, tester, business analyst, solution designer and project manager. This breadth of roles makes her perfect for the current role, understanding how the business needs data and how this data can be managed to meet those business needs.  
 
Wenjie has a BS in computer science from University of California at San Diego, and ME in computer engineering at Cornell University. Wenjie is also certified on both Teradata and AWS. Zeige alle Beiträge von Wenjie Tehan

Bleiben Sie auf dem Laufenden

Abonnieren Sie den Blog von Teradata, um wöchentliche Einblicke zu erhalten



Ich erkläre mich damit einverstanden, dass mir die Teradata Corporation als Anbieter dieser Website gelegentlich Marketingkommunikations-E-Mails mit Informationen über Produkte, Data Analytics und Einladungen zu Events und Webinaren zusendet. Ich nehme zur Kenntnis, dass ich mein Einverständnis jederzeit widerrufen kann, indem ich auf den Link zum Abbestellen klicke, der sich am Ende jeder von mir erhaltenen E-Mail befindet.

Der Schutz Ihrer Daten ist uns wichtig. Ihre persönlichen Daten werden im Einklang mit der globalen Teradata Datenschutzrichtlinie verarbeitet.