Amazon S3 is a scalable, object-based storage service. Today's enterprise applications often leverage serverless storage solutions like S3 to store and retrieve files efficiently.
This tutorial will demonstrate how to download files or directories from an Amazon S3 bucket using Spring Cloud AWS.
Spring Cloud AWS provides a convenient way to integrate S3 functionality into Spring Boot applications.
To download files from Amazon S3, you'll first need to (getButton) #text=(upload files to your S3 bucket) #icon=(link) #color=(#35a576) Then, follow the steps below
The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)
Dependencies
To get started with this tutorial, you'll need to add the following dependencies to your Spring Boot project's. This dependency provides the necessary components for interacting with AWS services, including S3.
Gradle
ext {
springCloudAwsVersion = '3.0.0'
}
dependencies {
// spring cloud aws BOM(Bill of materials)
implementation platform("io.awspring.cloud:spring-cloud-aws-dependencies:${springCloudAwsVersion}")
implementation 'io.awspring.cloud:spring-cloud-aws-starter-s3'
// AWS launched a high level file transfer utility, called Transfer Manager and a CRT based S3 client.
// The starter automatically configures and registers a software.amazon.awssdk.transfer.s3.S3TransferManager bean if the following dependency is added to the project:
implementation 'software.amazon.awssdk:s3-transfer-manager'
//Transfer Manager works the best with CRT S3 Client. To auto-configure CRT based S3AsyncClient add following dependency to your project:
implementation 'software.amazon.awssdk.crt:aws-crt'
}Maven
<dependencies>
<dependency>
<groupId>io.awspring.cloud</groupId>
<artifactId>spring-cloud-aws-starter-s3</artifactId>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>s3-transfer-manager</artifactId>
</dependency>
<dependency>
<groupId>software.amazon.awssdk.crt</groupId>
<artifactId>aws-crt</artifactId>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>io.awspring.cloud</groupId>
<artifactId>spring-cloud-aws-dependencies</artifactId>
<version>{project-version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>Configuration and Credentials
Before initiating the object download process, let's configure our Spring Boot application to interact with AWS S3. Spring Cloud AWS offers auto-configuration for S3Client, S3TransferManager, and S3Template, making setup a breeze.
Here's what you need to add to your application.properties file:
# s3 Configuration
spring.cloud.aws.credentials.access-key=
spring.cloud.aws.credentials.secret-key=
# Configures endpoint used by S3Client, I'm woorking in the Asia Pacific (Mumbai) hence, I've configure ap-south1 region and endpoint
spring.cloud.aws.s3.endpoint=https://s3.ap-south-1.amazonaws.com
spring.cloud.aws.region.static=ap-south-1IAM Permissions: Downloading S3 Objects
Each AWS service requires specific permissions to perform operations. These permissions can be granted to IAM users through policies, AWS service roles, or other mechanisms.
For this tutorial, we'll use an IAM user to download an object from S3. To enable this, we've attached the necessary policy to the IAM user. Alternatively, you can use an IAM role
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DownloadFilePolicy",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucketVersions",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::*/*",
"arn:aws:s3:::learnjavaskills"
]
}
]
}Downloading Files from AWS S3 with S3Client
Let's explore our first approach to downloading an object from Amazon S3. We'll utilize the S3Client interface, a dedicated service client for interacting with the S3 service.
The S3Client interface is a fundamental component of the AWS SDK for Java. It provides a high-level API for interacting with Amazon Simple Storage Service (S3). This interface allows developers to perform various operations on S3 buckets and objects
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
import java.nio.file.Path;
public void downloadFileUsingS3Client(String bucketName, String key, String downloadDestinationPath) {
try {
GetObjectRequest getObjectRequest = GetObjectRequest.builder()
.bucket(bucketName)
.key(key)
.build();
s3Client.getObject(getObjectRequest, Path.of(downloadDestinationPath));
} catch (Exception exception) {
exception.printStackTrace();
}
}The getObject() method of S3Client is the primary mechanism for downloading objects from an S3 bucket. It requires two parameters: a GetObjectRequest object specifying the bucket name and object key, and a local file path where the downloaded object will be saved.
Downloading Files from AWS S3 with S3TransferManager
S3TransferManager is a powerful utility within the AWS SDK for Java that simplifies the process of transferring files to and from Amazon S3.
It offers enhanced performance and reliability by leveraging Amazon S3's multipart upload and byte-range fetch capabilities.
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
import software.amazon.awssdk.transfer.s3.S3TransferManager;
import software.amazon.awssdk.transfer.s3.model.DownloadFileRequest;
import software.amazon.awssdk.transfer.s3.model.FileDownload;
import software.amazon.awssdk.transfer.s3.model.CompletedFileDownload;
import java.nio.file.Path;
public void downloadFileUsingS3TransferManager(String bucketName, String key, String downloadDestinationPath) {
try {
GetObjectRequest getObjectRequest = GetObjectRequest.builder()
.bucket(bucketName)
.key(key)
.build();
DownloadFileRequest downloadFileRequest = DownloadFileRequest.builder()
.destination(Path.of(downloadDestinationPath))
.getObjectRequest(getObjectRequest)
.build();
FileDownload fileDownload = s3TransferManager.downloadFile(downloadFileRequest);
CompletedFileDownload completedFileDownload = fileDownload.completionFuture()
.join();
String contentType = completedFileDownload.response()
.contentType();
System.out.println("Download completed, content type is : " + contentType);
} catch (Exception exception) {
exception.printStackTrace();
}
}The downloadFile() method of S3TransferManager facilitates downloading S3 objects. It requires a DownloadFileRequest object as an argument.
This request specifies the object's details, including the local destination path and an GetObjectRequest instance containing the bucket name and key, as discussed in our previous example.
Understanding completionFuture()
The completionFuture method in the FileDownload interface of the AWS SDK for Java provides a way to asynchronously track the progress and completion of an S3 file download operation.
It returns a CompletableFuture<Void> object, which represents the future result of the download operation. Key points about completionFuture:
• Asynchronous Execution
• Progress Tracking
• Error Handling
• Completion Notification
Downloading Directories from AWS S3 with S3TransferManager
Similarly, we can efficiently download entire directories or buckets from Amazon S3 using S3TransferManager. To accomplish this, we employ the downloadDirectory() method instead of downloadFile() method.
import software.amazon.awssdk.transfer.s3.S3TransferManager;
import software.amazon.awssdk.transfer.s3.model.DownloadDirectoryRequest;
import software.amazon.awssdk.transfer.s3.model.DirectoryDownload;
import software.amazon.awssdk.transfer.s3.model.CompletedDirectoryDownload;
import java.nio.file.Path;
public void downloadDirectoryUsingS3TransferManager(String bucketName, String downloadDestinationPath) {
try {
DownloadDirectoryRequest downloadDirectoryRequest = DownloadDirectoryRequest.builder()
.bucket(bucketName)
.destination(Path.of(downloadDestinationPath))
.build();
DirectoryDownload directoryDownload = s3TransferManager.downloadDirectory(downloadDirectoryRequest);
CompletedDirectoryDownload completedDirectoryDownload = directoryDownload.completionFuture()
.join();
completedDirectoryDownload.failedTransfers()
.forEach(failedFileDownload -> System.out.println("fail to download : " + failedFileDownload.toString()));
} catch (Exception exception) {
exception.printStackTrace();
}
}Downloading Files with S3Template
The S3Template class in the Spring Cloud AWS library provides a high-level abstraction for interacting with Amazon S3. It simplifies common S3 operations, making it easier to integrate S3 functionality into your Spring Boot applications.
The S3Template provides a convenient download() method for retrieving objects from Amazon S3. This method requires two arguments: the bucket name and the object key, which uniquely identify the desired object within the S3 storage.
import io.awspring.cloud.s3.S3Resource;
import io.awspring.cloud.s3.S3Template;
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardCopyOption;
public void downloadFileUsingS3Template(String bucketName, String key, String downloadDestinationPath) {
try {
S3Resource s3Resource = s3Template.download(bucketName, key);
InputStream inputStream = s3Resource.getInputStream();
Files.copy(inputStream, Path.of(downloadDestinationPath), StandardCopyOption.REPLACE_EXISTING);
} catch (Exception exception) {
exception.printStackTrace();
}
}Conclussion
For Spring Boot applications requiring single file downloads, S3Template offers an efficient and convenient approach.
In legacy Java applications without Spring Cloud AWS integration, consider using S3Client for object downloads.
S3TransferManager provides the most efficient and feature-rich option for object downloads, offering benefits like progress tracking, error handling, and asynchronous communication, making it well-suited for multithreaded environments.
Learn how to efficiently (getButton) #text=(read files directly into a Java input stream) #icon=(link) #color=(#35a576) without the need for downloading
The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)
Keep learning and keep growing.
