AWS DynamoDB's BatchWriteItem and BatchGetItem operations offer efficient ways to interact with multiple items in a single request, enhancing performance and reducing network overhead compared to individual item operations.
The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)
Interacting with DynamoDB Using Java
The AWS SDK for Java provides interface like DynamoDbClient and DynamoDbAsyncClient to interact with the DynamoDB database. These clients offer methods for various DynamoDB operations, including getItem, putItem, batchGetItem, and batchWriteItem
Spring Boot simplifies DynamoDB interaction by auto-configuring the DynamoDbClient based on settings in your application properties file when using the Spring Cloud AWS Starter DynamoDB
Dependencies: Gradle
// spring cloud aws BOM(Bill of materials)
implementation platform("io.awspring.cloud:spring-cloud-aws-dependencies:${springCloudAwsVersion}")
// dynamoDB
implementation 'io.awspring.cloud:spring-cloud-aws-starter-dynamodb'Setting Up DynamoDB for Your Application: application.properties
If you're new to Spring Cloud AWS and DynamoDB, I recommend reading our (getButton) #text=(Spring Boot, DynamoDB, and Spring Cloud AWS: A Step-by-Step Tutorial) #icon=(link) #color=(#35a576), which will help you build up your project and get deep into the CRUD (Create, Read, Update, and Delete) operations.
spring.application.name=spring-cloud-aws-dynamodb
# DynamoDB connection
spring.cloud.aws.credentials.access-key=
spring.cloud.aws.credentials.secret-key=
# Choose your region according to your table. Because I have a DynamoDB table in the ap-south-1 area, I've set the endpoint and region.
spring.cloud.aws.region.static=ap-south-1
spring.cloud.aws.endpoint=https://dynamodb.ap-south-1.amazonaws.comDynamoDB BatchWriteItem: Efficiently Writing Multiple Items to DynamoDB
The BatchWriteItem approach puts or deletes multiple items into one or more tables. A single call to BatchWriteItem may send up to 16MB of data across the network, which includes up to 25 item put or delete operations.
While individual items can be up to 400 KB once saved, it's crucial to remember that an item's representation may be bigger than 400 KB when submitted in DynamoDB's JSON format for the API request.
BatchWriteItemcannot update items. If you run a BatchWriteItem action on an existing item, the operation will overwrite the item's values, making it look as if it was changed. To update things, AWS recommend using theUpdateItemaction.(alert-warning)
Java Program to Perform BatchWriteItems Operations on DynamoDB
This Java program demonstrates how to efficiently insert data into multiple DynamoDB tables using the BatchWriteItem operation. We'll create sample data for Orders and products tables and insert them in a single batch for improved performance.
package in.learnjavaskills.springcloudawsdynamodb.service;
import org.springframework.stereotype.Service;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
import software.amazon.awssdk.services.dynamodb.model.*;
import java.time.LocalDate;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
@Service
public class BatchItemExample
{
private final DynamoDbClient dynamoDbClient;
public BatchItemExample(DynamoDbClient dynamoDbClient) {
this.dynamoDbClient = dynamoDbClient;
}
public void batchWriteItem() {
Map<String, AttributeValue> productsMap = new HashMap<>();
productsMap.put("product_id", AttributeValue.fromS("pd_12345"));
productsMap.put("product_name", AttributeValue.fromS("Laptop"));
productsMap.put("description", AttributeValue.fromS("14 inch screen laptop"));
productsMap.put("price", AttributeValue.fromN("49.87"));
productsMap.put("stock", AttributeValue.fromN("101"));
PutRequest productsPutRequest = PutRequest.builder().item(productsMap).build();
WriteRequest productsWriteRequest = WriteRequest.builder().putRequest(productsPutRequest).build();
Map<String, AttributeValue> ordersMap = new HashMap<>();
ordersMap.put("order_id", AttributeValue.fromS("od_12345"));
ordersMap.put("customer_id", AttributeValue.fromS("c_12345"));
ordersMap.put("product_id", AttributeValue.fromS("pd_12345"));
ordersMap.put("order_date", AttributeValue.fromS(LocalDate.now().toString()));
PutRequest orderPutRequest = PutRequest.builder().item(ordersMap).build();
WriteRequest orderWriteRequest = WriteRequest.builder().putRequest(orderPutRequest).build();
Map<String, List<WriteRequest>> writeRequestMap = new HashMap<>();
writeRequestMap.put("products", Arrays.asList(productsWriteRequest));
writeRequestMap.put("Orders", Arrays.asList(orderWriteRequest));
BatchWriteItemRequest batchWriteItemRequest = BatchWriteItemRequest.builder().requestItems(writeRequestMap).build();
BatchWriteItemResponse batchWriteItemResponse = dynamoDbClient.batchWriteItem(batchWriteItemRequest);
// Implement a retry mechanism for batchWriteItem operations that encounter errors.
while (!batchWriteItemResponse.unprocessedItems().isEmpty()) {
Map<String, List<WriteRequest>> unprocessedItems = batchWriteItemResponse.unprocessedItems();
batchWriteItemRequest
= BatchWriteItemRequest.builder().requestItems(unprocessedItems).build();
batchWriteItemResponse = dynamoDbClient.batchWriteItem(batchWriteItemRequest);
}
}
}Understanding BatchWriteItemRequest
The batchWriteItem() method in the DynamoDBClient allows you to perform multiple write operations (puts or deletes) across one or more tables in a single request. You provide a BatchWriteItemRequest object to specify the items and actions to be performed.
The requestItems() method in a BatchWriteItemRequest class defines the data to be written to DynamoDB. It takes a map where each key is a DynamoDB table name and the corresponding value is a list of WriteRequest operations (put or delete) to be performed on that table.
Handling Unprocessed Items: Retrying BatchWriteItem Operations
unprocessedItems() is a response parameter returned by the DynamoDB BatchWriteItemRequest operation. It contains a list of items that couldn't be processed due to reasons like exceeding provisioned throughput or internal processing errors.
Essentially, it's a mechanism for handling failed write operations. You can retry these unprocessed items in subsequent BatchWriteItemRequest until all items are successfully written to DynamoDB.
If the requested data exceeds these limits, or if there's insufficient table capacity, or an internal error occurs, the operation might return only a portion of the results. In such cases, the system provides information about the UnprocessedKeys, allowing you to retry the operation to retrieve the remaining data.(alert-passed)
When DynamoDB returns unprocessed items in a batch operation, it's essential to retry these items. However, to avoid overwhelming the system and increasing the likelihood of success, implement an exponential backoff strategy. This means progressively increasing the delay between retries. By doing so, you reduce the chance of encountering throttling issues and improve the overall reliability of your batch operations.(alert-error)
Understanding WriteRequest: Building Blocks for Batch Write Operations
A WriteRequest is a container for specifying either a PutItem or DeleteItem operation within a BatchWriteItem request.
You can only perform one of these operations (Put or Delete) per WriteRequest. If you need to perform both, you'll need separate WriteRequest objects. Each WriteRequest contains either a PutRequest or a DeleteRequest object.
PutRequest: Inserting or Updating Items
If you want to insert or update an item, you use a PutRequest object within your WriteRequest. A PutRequest contains an Item method, which is a map representing the item to be put into the table.
DeleteRequest: Removing Items from DynamoDB
If you want to delete an item, you use a DeleteRequest object within your WriteRequest. A DeleteRequest contains a Key method, which is a map representing the primary key of the item to be deleted.
DynamoDB BatchGetItems: Efficiently Retrieve Multiple Items
You can use the BatchGetItem operation to fetch data from one or more tables. Specify the items you need by their primary keys, and the operation will return their corresponding attributes.
A single BatchGetItem operation can fetch up to 100 items with a maximum data size of 16 MB.
Java Program to Perform BatchGetItem Operations on DynamoDB
This Java program demonstrates how to efficiently fetch data from two DynamoDB tables, Orders and products, using the BatchGetItem operation. By specifying multiple items and their primary keys within a single request, we can optimize data retrieval performance compared to individual GetItem operations.
package in.learnjavaskills.springcloudawsdynamodb.service;
import org.springframework.stereotype.Service;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
import software.amazon.awssdk.services.dynamodb.model.*;
import java.time.LocalDate;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
@Service
public class BatchItemExample
{
private final DynamoDbClient dynamoDbClient;
public BatchItemExample(DynamoDbClient dynamoDbClient) {
this.dynamoDbClient = dynamoDbClient;
}
public void batchGetItem() {
Map<String, AttributeValue> productsKeys = new HashMap<>();
productsKeys.put("product_id", AttributeValue.fromS("pd_12345"));
productsKeys.put("product_name", AttributeValue.fromS("Laptop"));
Map<String, AttributeValue> ordersKeys = new HashMap<>();
ordersKeys.put("order_id", AttributeValue.fromS("od_12345"));
ordersKeys.put("customer_id", AttributeValue.fromS("c_12345"));
Map<String, KeysAndAttributes> requestItems = new HashMap<>();
KeysAndAttributes productsKeysAndAttributes = KeysAndAttributes.builder().consistentRead(false)
.keys(productsKeys).build();
requestItems.put("products", productsKeysAndAttributes);
KeysAndAttributes ordersKeysAndAttributes = KeysAndAttributes.builder()
.keys(ordersKeys)
.build();
requestItems.put("Orders", ordersKeysAndAttributes);
BatchGetItemRequest batchGetItemRequest = BatchGetItemRequest.builder().requestItems(requestItems).build();
BatchGetItemResponse batchGetItemResponse = dynamoDbClient.batchGetItem(batchGetItemRequest);
Map<String, List<Map<String, AttributeValue>>> responses = batchGetItemResponse.responses();
responses.forEach( (table, itemsList) -> {
System.out.println("table name: " + table);
itemsList.forEach( attributesAndValues -> {
attributesAndValues.forEach( (key, value) -> {
System.out.println("attributes name : " + key + ", attributes value: " + value.s());
});
});
} );
// Implement a retry mechanism for batchGetItem operations that encounter errors.
while(!batchGetItemResponse.unprocessedKeys().isEmpty()) {
Map<String, KeysAndAttributes> unprocessedKeys = batchGetItemResponse.unprocessedKeys();
batchGetItemRequest = BatchGetItemRequest.builder().requestItems(unprocessedKeys).build();
batchGetItemResponse = dynamoDbClient.batchGetItem(batchGetItemRequest);
responses = batchGetItemResponse.responses();
responses.forEach( (table, itemsList) -> {
System.out.println("table name: " + table);
itemsList.forEach( attributesAndValues -> {
attributesAndValues.forEach( (key, value) -> {
System.out.println("attributes name : " + key + ", attributes value: " + value.s());
});
});
} );
}
}
}Understanding BatchGetItemRequest
The batchGetItem() method in the DynamoDBClient allows you to efficiently retrieve multiple items from one or more tables in a single operation. To specify the items you want to fetch, you create a BatchGetItemRequest object.
The requestItems() method within a BatchGetItemRequest specifies the tables and items to retrieve. It takes a map where the keys are table names, and the values are objects containing information about the items to fetch from that table, such as their primary keys and any projection expressions.
Conclusion
By effectively utilizing the BatchWriteItem and BatchGetItem operations, you can significantly enhance the efficiency and performance of your DynamoDB interactions. By understanding the core concepts, including WriteRequest, PutRequest, DeleteRequest, and handling unprocessed items, you're well-equipped to optimize your data manipulation strategies.
To learn more about DynamoDB transactions and how to use TransactWriteItems and TransactGetItems, check out my latest blog post on (getButton) #text=(DynamoDB Transactions with TransactWriteItems and TransactGetItems) #icon=(link) #color=(#35a576)
The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)
Keep learning and keep growing.
