In today's digital age, storing and managing large amounts of data is crucial for many applications. Amazon Simple Storage Service (S3) offers a reliable and scalable solution for object storage in the cloud.
Spring Cloud AWS provides a convenient way to integrate S3 functionality into Spring Boot applications.
This article will guide you through the process of uploading files to Amazon S3 using Spring Cloud AWS, covering essential steps and configurations. By the end, you'll have a solid understanding of how to leverage S3 storage for your Spring Boot applications.
To get started with uploading files to Amazon S3, make sure you have a bucket ready. Refer to my previous blog posts for instructions on (getButton) #text=(creating, deleting, and managing buckets) #icon=(link) #color=(#35a576)
The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)
Dependencies
To get started with this tutorial, you'll need to add the following dependencies to your Spring Boot project's. This dependency provides the necessary components for interacting with AWS services, including S3.
Gradle
ext {
springCloudAwsVersion = '3.0.0'
}
dependencies {
// spring cloud aws BOM(Bill of materials)
implementation platform("io.awspring.cloud:spring-cloud-aws-dependencies:${springCloudAwsVersion}")
implementation 'io.awspring.cloud:spring-cloud-aws-starter-s3'
// AWS launched a high level file transfer utility, called Transfer Manager and a CRT based S3 client.
// The starter automatically configures and registers a software.amazon.awssdk.transfer.s3.S3TransferManager bean if the following dependency is added to the project:
implementation 'software.amazon.awssdk:s3-transfer-manager'
//Transfer Manager works the best with CRT S3 Client. To auto-configure CRT based S3AsyncClient add following dependency to your project:
implementation 'software.amazon.awssdk.crt:aws-crt'
}Maven
<dependencies>
<dependency>
<groupId>io.awspring.cloud</groupId>
<artifactId>spring-cloud-aws-starter-s3</artifactId>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>s3-transfer-manager</artifactId>
</dependency>
<dependency>
<groupId>software.amazon.awssdk.crt</groupId>
<artifactId>aws-crt</artifactId>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>io.awspring.cloud</groupId>
<artifactId>spring-cloud-aws-dependencies</artifactId>
<version>{project-version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>Configuration and Credentials
Before we get started with uploading files, let's configure our Spring Boot application to interact with AWS S3. Spring Cloud AWS offers auto-configuration for S3Client, S3TransferManager, and S3Template, making setup a breeze.
Here's what you need to add to your application.properties file:
# s3 Configuration
spring.cloud.aws.credentials.access-key=
spring.cloud.aws.credentials.secret-key=
# Configures endpoint used by S3Client, I'm woorking in the Asia Pacific (Mumbai) hence, I've configure ap-south1 region and endpoint
spring.cloud.aws.s3.endpoint=https://s3.ap-south-1.amazonaws.com
spring.cloud.aws.region.static=ap-south-1IAM Permissions for Bucket Management
In this in-depth tutorial, we'll explore various features of AWS S3, including tagging, metadata, and file uploading. We'll delve into multiple approaches for uploading files.
To seamlessly upload files with tags and metadata, your Amazon Simple Storage Service (S3) bucket will require the following IAM permissions.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "uploadFile",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:PutObjectTagging",
"s3:GetObjectTagging"
],
"Resource": "arn:aws:s3:::*/*"
}
]
}AWS S3 File Upload with S3Client
One of the most straightforward ways to upload a file to Amazon Simple Storage Service (S3) is by using the S3Client interface. S3Client is a service client designed for interacting with Amazon S3.
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.core.sync.RequestBody;
import java.nio.file.Paths;
public void uploadFileUsingS3Client(String bucketName, String key, String filePath) {
try {
PutObjectRequest putObjectRequest = PutObjectRequest.builder()
.bucket(bucketName)
.key(key)
.build();
s3Client.putObject(putObjectRequest, RequestBody.fromFile(Paths.get(filePath)));
} catch (S3Exception s3Exception) {
s3Exception.printStackTrace();
}
}The putObject() method of S3Client is your gateway to uploading files to your Amazon S3 bucket. It takes two arguments: a PutObjectRequest object and a RequestBody, which specifies the exact file you want to upload.
• PutObjectRequest is a class (located in the com.amazonaws.services.s3.model package) that allows you to configure the upload process.
• To create a valid PutObjectRequest, specifying the target bucket name and the key (unique identifier) for the file is essential.
Running Unit Tests for File Uploading
@Test
void uploadFileUsingS3Client()
{
uploadObjects.uploadFileUsingS3Client(bucketName, "robot.gif", filePath);
}File Upload Successful to Amazon S3
Efficient File Uploads and Tagging with S3Client
Tagging in Amazon S3 is a powerful feature that allows you to attach metadata to your objects. These metadata tags are key-value pairs that provide additional information about your objects, making them easier to categorize, search, and manage.
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.model.Tag;
import software.amazon.awssdk.services.s3.model.Tagging;
import software.amazon.awssdk.core.sync.RequestBody;
import java.nio.file.Paths;
import java.util.List;
public void uploadFileWithTag(String bucketName, String key, String filePath) {
try {
Tag tag1 = Tag.builder().
key("tag 1")
.value("tag 1 value")
.build();
Tag tag2 = Tag.builder()
.key("tag 2")
.value("tag 2 value")
.build();
List<Tag> tagList = List.of(tag1, tag2);
Tagging tagging = Tagging.builder()
.tagSet(tagList)
.build();
PutObjectRequest putObjectRequest = PutObjectRequest.builder()
.bucket(bucketName)
.key(key)
.tagging(tagging)
.build();
s3Client.putObject(putObjectRequest, RequestBody.fromFile(Paths.get(filePath)));
} catch (S3Exception s3Exception) {
s3Exception.printStackTrace();
}
}To tag an object in Amazon S3, simply use the Tagging class to add your desired key-value pairs while uploading the file. The Tagging class has a tagSet() method that accepts a list of Tag objects.
To tag an object while uploading it to Amazon S3, use the tagging() method of the PutObjectRequest class to provide the desired tagging information.
Testing File Uploads with Tags
@Test
void uploadFileWithTag()
{
uploadObjects.uploadFileWithTag(bucketName, "robot-with-tag.gif", filePath);
}Verifying Tagged File Uploads
Efficiently Updating Tag Information for S3 Files
You may need to update or modify the tag information associated with an Amazon S3 object. To do this, you must first retrieve the existing tags from the S3 bucket.
You can then update or create new tags as needed. Once you've configured the desired tag information, you can apply these changes to the S3 object by using the PutObjectTaggingRequest class.
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.model.GetObjectTaggingRequest;
import software.amazon.awssdk.services.s3.model.GetObjectTaggingResponse;
import software.amazon.awssdk.services.s3.model.PutObjectTaggingRequest;
import software.amazon.awssdk.services.s3.model.Tag;
import software.amazon.awssdk.services.s3.model.Tagging;
import java.util.List;
public void updateObjectTag(String bucketName, String key) {
try {
GetObjectTaggingRequest getObjectTaggingRequest = GetObjectTaggingRequest.builder()
.bucket(bucketName)
.key(key)
.build();
GetObjectTaggingResponse objectTagging = s3Client.getObjectTagging(getObjectTaggingRequest);
// print existing tags
objectTagging.tagSet()
.stream()
.forEach(tag -> System.out.println("tag key : " + tag.key() + " tag value : " + tag.value()));
// new tags add
Tag tag3 = Tag.builder()
.key("tag 3")
.value("tag 3 value")
.build();
Tag tag4 = Tag.builder()
.key("tag 4")
.value("tag 4 value")
.build();
List<Tag> tagList = List.of(tag3, tag4);
Tagging updatedTags = Tagging.builder()
.tagSet(tagList)
.build();
PutObjectTaggingRequest putObjectTaggingRequest = PutObjectTaggingRequest.builder()
.bucket(bucketName)
.key(key)
.tagging(updatedTags)
.build();
// update tags
s3Client.putObjectTagging(putObjectTaggingRequest);
} catch (S3Exception s3Exception) {
s3Exception.printStackTrace();
}
}Testing Tag Updates in Amazon S3
@Test
void updateObjectTag()
{
uploadObjects.updateObjectTag(bucketName, "robot-with-tag.gif");
}Tag Update Successful
AWS S3 File Upload with S3Client: Adding Metadata to Uploading Files
Metadata is a set of data that describes an object stored in Amazon S3. It provides additional information about the object, such as its creation date, size, content type, and custom user-defined attributes.
This metadata can be used to organize, search, and manage your objects more effectively.
Key types of metadata in S3:
• System-defined metadata: This metadata is automatically created and managed by S3 and includes information like object size, creation date, and storage class.
• User-defined metadata: This metadata is added by the user and can contain any key-value pairs. It's a flexible way to store custom information about your objects.
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.core.sync.RequestBody;
import java.nio.file.Paths;
import java.util.Map;
public void uploadFileWithMetaData(String bucketName, String key, String filePath) {
try {
Map<String, String> metadata = new HashMap<>();
metadata.put("author", "learnjavaskills.in");
metadata.put("file-type", "txt");
metadata.put("version", "1.0");
PutObjectRequest putObjectRequest = PutObjectRequest.builder()
.bucket(bucketName)
.key(key)
.metadata(metadata)
.build();
s3Client.putObject(putObjectRequest, RequestBody.fromFile(Paths.get(filePath)));
} catch (S3Exception s3Exception) {
s3Exception.printStackTrace();
}
}Testing File Uploads with Metadata in Amazon S3
@Test
void uploadFileWithMetaData()
{
uploadObjects.uploadFileWithMetaData(bucketName, "robot-with-meta-data.gif", filePath);
}File Upload Successful with Metadata
AWS S3 File Upload with S3TransferManager
The S3TransferManager class in the AWS SDK for Java provides a high-level API for managing asynchronous data transfers to and from Amazon S3.
It handles the complexities of multi-part uploads, retries, and error handling, making it a convenient choice for large file transfers or bulk data operations.
import software.amazon.awssdk.transfer.s3.S3TransferManager;
import software.amazon.awssdk.transfer.s3.model.UploadFileRequest;
import software.amazon.awssdk.transfer.s3.model.FileUpload;
import software.amazon.awssdk.transfer.s3.model.CompletedFileUpload;
import java.nio.file.Paths;
import java.util.OptionalLong;
public void uploadFileWithTransferManager(String bucketName, String key, String filePath) {
try
{
UploadFileRequest uploadFileRequest = UploadFileRequest.builder()
.putObjectRequest(builder -> builder.bucket(bucketName)
.key(key))
.source(Paths.get(filePath))
.build();
FileUpload fileUpload = s3transferManager.uploadFile(uploadFileRequest);
// check progress of file upload
while (true)
{
OptionalLong optionalLong = fileUpload.progress()
.snapshot()
.remainingBytes();
if (optionalLong.isPresent())
{
System.out.println("remaining byte transfer : " + optionalLong.getAsLong());
if (optionalLong.getAsLong() == 0)
break;
}
}
CompletedFileUpload completedFileUpload = fileUpload.completionFuture().join();
boolean successful = completedFileUpload.response().sdkHttpResponse().isSuccessful();
System.out.println("is Successful : " + successful);
} catch (S3Exception s3Exception) {
s3Exception.printStackTrace();
}
}AWS S3 File Upload with S3Template
The S3Template class in the Spring Cloud AWS library provides a high-level abstraction for interacting with Amazon S3. It simplifies common S3 operations, making it easier to integrate S3 functionality into your Spring Boot applications.
A straightforward way to upload a file to Amazon Simple Storage Service (S3) is by using the upload() method provided by S3Template.
The upload() method requires three essential arguments: the bucket name, the object key, and the file's input stream.
import io.awspring.cloud.s3.S3Template;
import io.awspring.cloud.s3.S3Resource;
import java.io.FileInputStream;
import java.io.InputStream;
import java.net.URL;
public void uploadFileUsingS3Template(String bucketName, String key, String filePath) {
try {
InputStream inputStream = new FileInputStream(filePath);
S3Resource s3Resource = s3Template.upload(bucketName, key, inputStream);
URL url = s3Resource.getURL();
System.out.println("File uploaded successfully at " + url.toString());
} catch (S3Exception | IOException exception) {
exception.printStackTrace();
}
}AWS S3 File Upload with S3Template: Adding Metadata to Uploaded Files
S3Template also allows you to add custom metadata to an Amazon S3 object. An overloaded version of the upload() method accepts metadata as an argument, in addition to the bucket name, object key, and input stream of the file.
import io.awspring.cloud.s3.S3Template;
import io.awspring.cloud.s3.S3Resource;
import java.io.FileInputStream;
import java.io.InputStream;
import io.awspring.cloud.s3.ObjectMetadata;
import java.net.URL;
public void uploadFileUsingS3TemplateWithMetadata(String bucketName, String key, String filePath) {
try {
InputStream inputStream = new FileInputStream(filePath);
ObjectMetadata objectMetadata = ObjectMetadata.builder()
.metadata("author", "learnjavaskills.in")
.metadata("version", "1.0")
.contentType("text/plain")
.build();
S3Resource s3Resource = s3Template.upload(bucketName, key, inputStream, objectMetadata);
URL url = s3Resource.getURL();
System.out.println("File uploaded successfully at " + url.toString());
}catch (S3Exception | IOException exception) {
exception.printStackTrace();
}
}Testing File Uploads with Metadata
@Test
void uploadFileUsingS3TemplateWithMetadata()
{
uploadObjects.uploadFileUsingS3TemplateWithMetadata(bucketName, "robot-s3-template-metadata.gif", filePath);
}Verifying Amazon S3 Objects with Metadata
Conclussion
For Spring Boot projects, S3Template simplifies file uploads with its convenient upload() method.
S3TransferManager offers significant advantages over S3Client, including multi-part uploads, asynchronous transfers, robust error handling, and progress tracking.
While S3Client is suitable for small file uploads, S3TransferManager excels in handling larger files and complex transfer scenarios.
Now that you've successfully uploaded your files to Amazon S3, let's explore how to (getButton) #text=(download files from S3 bucket) #icon=(link) #color=(#35a576) when needed.
The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)
Keep learning and keep growing.
