So, I started the investigation: what are my options?.Īt the beginning, I excluded the idea of scanning the table at the lambda level. In my case, BI team asked about a daily snapshot of our table from DynamoDB but only exported partially. They divide it to:ĮTL: Perform ETL (Extract, Transform, Load) operations on the exported data in S3, and then import the transformed data back into DynamoDB.ĭata Archiving: Retain historical snapshots for audit and compliance requirementsĭata Integration: Integrate the data with other services and applicationsĭata Lake: Build a data lake in S3, allowing users to perform analytics across multiple data sources using services such as Amazon Athena, Amazon Redshift, and Amazon SageMakerĪd-hoc queries: Query data from Athena or Amazon EMR without affecting your DynamoDB capacity
📔 One side note: I explore universal options, but keep in mind that my table size is below 1 GB. Also, I'll answer: why and how do this, and compare what solutions AWS offers. Additionally, I'd like my data to be filtered by secondary index. Therefore, in this article I'll try to cover the whole process of exporting AWS DynamoDB data to S3 as a recurring task.
You went to AWS console only to discover that it limits you to a single "on click" export? Have you ever tried to schedule export of DynamoDB data to S3? I mean the automated recurring task everyday e.g.
0 kommentar(er)
