Struct opendal::services::S3

pub struct S3 { /* private fields */ }

Available on crate feature services-s3 only.

Aws S3 and compatible services (including minio, digitalocean space, Tencent Cloud Object Storage(COS) and so on) support. For more information about s3-compatible services, refer to Compatible Services.

Capabilities

This service can be used to:

• stat
• read
• write
• create_dir
• delete
• copy
• rename
• list
• scan
• presign
• blocking

Configuration

• root: Set the work dir for backend.
• bucket: Set the container name for backend.
• endpoint: Set the endpoint for backend.
• region: Set the region for backend.
• access_key_id: Set the access_key_id for backend.
• secret_access_key: Set the secret_access_key for backend.
• security_token: Set the security_token for backend.
• default_storage_class: Set the default storage_class for backend.
• server_side_encryption: Set the server_side_encryption for backend.
• server_side_encryption_aws_kms_key_id: Set the server_side_encryption_aws_kms_key_id for backend.
• server_side_encryption_customer_algorithm: Set the server_side_encryption_customer_algorithm for backend.
• server_side_encryption_customer_key: Set the server_side_encryption_customer_key for backend.
• server_side_encryption_customer_key_md5: Set the server_side_encryption_customer_key_md5 for backend.
• disable_config_load: Disable aws config load from env
• enable_virtual_host_style: Enable virtual host style.

Refer to S3Builder’s public API docs for more information.

Temporary security credentials

OpenDAL now provides support for S3 temporary security credentials in IAM.

The way to take advantage of this feature is to build your S3 backend with Builder::security_token.

But OpenDAL will not refresh the temporary security credentials, please keep in mind to refresh those credentials in time.

Server Side Encryption

OpenDAL provides full support of S3 Server Side Encryption(SSE) features.

The easiest way to configure them is to use helper functions like

• SSE-KMS: server_side_encryption_with_aws_managed_kms_key
• SSE-KMS: server_side_encryption_with_customer_managed_kms_key
• SSE-S3: server_side_encryption_with_s3_key
• SSE-C: server_side_encryption_with_customer_key

If those functions don’t fulfill need, low-level options are also provided:

• Use service managed kms key
  • server_side_encryption="aws:kms"
• Use customer provided kms key
  • server_side_encryption="aws:kms"
  • server_side_encryption_aws_kms_key_id="your-kms-key"
• Use S3 managed key
  • server_side_encryption="AES256"
• Use customer key
  • server_side_encryption_customer_algorithm="AES256"
  • server_side_encryption_customer_key="base64-of-your-aes256-key"
  • server_side_encryption_customer_key_md5="base64-of-your-aes256-key-md5"

After SSE have been configured, all requests send by this backed will attach those headers.

Reference: Protecting data using server-side encryption

Example

Via Builder

Basic Setup

use std::sync::Arc;

use anyhow::Result;
use opendal::services::S3;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    // Create s3 backend builder.
    let mut builder = S3::default();
    // Set the root for s3, all operations will happen under this root.
    //
    // NOTE: the root must be absolute path.
    builder.root("/path/to/dir");
    // Set the bucket name. This is required.
    builder.bucket("test");
    // Set the region. This is required for some services, if you don't care about it, for example Minio service, just set it to "auto", it will be ignored.
    builder.region("us-east-1");
    // Set the endpoint.
    //
    // For examples:
    // - "https://s3.amazonaws.com"
    // - "http://127.0.0.1:9000"
    // - "https://oss-ap-northeast-1.aliyuncs.com"
    // - "https://cos.ap-seoul.myqcloud.com"
    //
    // Default to "https://s3.amazonaws.com"
    builder.endpoint("https://s3.amazonaws.com");
    // Set the access_key_id and secret_access_key.
    //
    // OpenDAL will try load credential from the env.
    // If credential not set and no valid credential in env, OpenDAL will
    // send request without signing like anonymous user.
    builder.access_key_id("access_key_id");
    builder.secret_access_key("secret_access_key");

    let op: Operator = Operator::new(builder)?.finish();

    Ok(())
}

S3 with SSE-C

use anyhow::Result;
use log::info;
use opendal::services::S3;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    let mut builder = S3::default();

    // Setup builders
    builder.root("/path/to/dir");
    builder.bucket("test");
    builder.region("us-east-1");
    builder.endpoint("https://s3.amazonaws.com");
    builder.access_key_id("access_key_id");
    builder.secret_access_key("secret_access_key");

    // Enable SSE-C
    builder.server_side_encryption_with_customer_key("AES256", "customer_key".as_bytes());

    let op = Operator::new(builder)?.finish();
    info!("operator: {:?}", op);

    // Writing your testing code here.

    Ok(())
}

S3 with SSE-KMS and aws managed kms key

use anyhow::Result;
use log::info;
use opendal::services::S3;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    let mut builder = S3::default();

    // Setup builders
    builder.root("/path/to/dir");
    builder.bucket("test");
    builder.region("us-east-1");
    builder.endpoint("https://s3.amazonaws.com");
    builder.access_key_id("access_key_id");
    builder.secret_access_key("secret_access_key");
    
    // Enable SSE-KMS with aws managed kms key
    builder.server_side_encryption_with_aws_managed_kms_key();

    let op = Operator::new(builder)?.finish();
    info!("operator: {:?}", op);

    // Writing your testing code here.

    Ok(())
}

S3 with SSE-KMS and customer managed kms key

use anyhow::Result;
use log::info;
use opendal::services::S3;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    let mut builder = S3::default();

    // Setup builders
    builder.root("/path/to/dir");
    builder.bucket("test");
    builder.region("us-east-1");
    builder.endpoint("https://s3.amazonaws.com");
    builder.access_key_id("access_key_id");
    builder.secret_access_key("secret_access_key");

    // Enable SSE-KMS with customer managed kms key
    builder.server_side_encryption_with_customer_managed_kms_key("aws_kms_key_id");

    let op = Operator::new(builder)?.finish();
    info!("operator: {:?}", op);

    // Writing your testing code here.

    Ok(())
}

S3 with SSE-S3

use anyhow::Result;
use log::info;
use opendal::services::S3;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    let mut builder = S3::default();

    // Setup builders
    builder.root("/path/to/dir");
    builder.bucket("test");
    builder.region("us-east-1");
    builder.endpoint("https://s3.amazonaws.com");
    builder.access_key_id("access_key_id");
    builder.secret_access_key("secret_access_key");

    // Enable SSE-S3
    builder.server_side_encryption_with_s3_key();

    let op = Operator::new(builder)?.finish();
    info!("operator: {:?}", op);

    // Writing your testing code here.

    Ok(())
}

Compatible Services

AWS S3

AWS S3 is the default implementations of s3 services. Only bucket is required.

builder.bucket("<bucket_name>");

Alibaba Object Storage Service (OSS)

OSS is a s3 compatible service provided by Alibaba Cloud.

To connect to OSS, we need to set:

• endpoint: The endpoint of oss, for example: https://oss-cn-hangzhou.aliyuncs.com
• bucket: The bucket name of oss.

OSS provide internal endpoint for used at alibabacloud internally, please visit OSS Regions and endpoints for more details.

OSS only supports the virtual host style, users could meet errors like:

<?xml version="1.0" encoding="UTF-8"?>
<Error>
 <Code>SecondLevelDomainForbidden</Code>
 <Message>The bucket you are attempting to access must be addressed using OSS third level domain.</Message>
 <RequestId>62A1C265292C0632377F021F</RequestId>
 <HostId>oss-cn-hangzhou.aliyuncs.com</HostId>
</Error>

In that case, please enable virtual host style for requesting.

builder.endpoint("https://oss-cn-hangzhou.aliyuncs.com");
builder.region("<region>");
builder.bucket("<bucket_name>");
builder.enable_virtual_host_style();

Minio

minio is an open-source s3 compatible services.

To connect to minio, we need to set:

• endpoint: The endpoint of minio, for example: http://127.0.0.1:9000
• region: The region of minio. If you don’t care about it, just set it to “auto”, it will be ignored.
• bucket: The bucket name of minio.

builder.endpoint("http://127.0.0.1:9000");
builder.region("<region>");
builder.bucket("<bucket_name>");

QingStor Object Storage

QingStor Object Storage is a S3-compatible service provided by QingCloud.

To connect to QingStor Object Storage, we need to set:

• endpoint: The endpoint of QingStor s3 compatible endpoint, for example: https://s3.pek3b.qingstor.com
• bucket: The bucket name.

Scaleway Object Storage

Scaleway Object Storage is a S3-compatible and multi-AZ redundant object storage service.

To connect to Scaleway Object Storage, we need to set:

• endpoint: The endpoint of scaleway, for example: https://s3.nl-ams.scw.cloud
• region: The region of scaleway.
• bucket: The bucket name of scaleway.

Tencent Cloud Object Storage (COS)

COS is a s3 compatible service provided by Tencent Cloud.

To connect to COS, we need to set:

• endpoint: The endpoint of cos, for example: https://cos.ap-beijing.myqcloud.com
• bucket: The bucket name of cos.

Wasabi Object Storage

Wasabi is a s3 compatible service.

Cloud storage pricing that is 80% less than Amazon S3.

To connect to wasabi, we need to set:

• endpoint: The endpoint of wasabi, for example: https://s3.us-east-2.wasabisys.com
• bucket: The bucket name of wasabi.

Refer to What are the service URLs for Wasabi’s different storage regions? for more details.

Cloudflare R2

Cloudflare R2 provides s3 compatible API.

Cloudflare R2 Storage allows developers to store large amounts of unstructured data without the costly egress bandwidth fees associated with typical cloud storage services.

To connect to r2, we need to set:

• endpoint: The endpoint of r2, for example: https://<account_id>.r2.cloudflarestorage.com
• bucket: The bucket name of r2.
• region: When you create a new bucket, the data location is set to Automatic by default. So please use auto for region.
• batch_max_operations: R2’s delete objects will return Internal Error if the batch is larger than 700. Please set this value <= 700 to make sure batch delete work as expected.
• enable_exact_buf_write: R2 requires the non-tailing parts size to be exactly the same. Please enable this option to avoid the error All non-trailing parts must have the same length.

Google Cloud Storage XML API

Google Cloud Storage XML API provides s3 compatible API.

• endpoint: The endpoint of Google Cloud Storage XML API, for example: https://storage.googleapis.com
• bucket: The bucket name.
• To access GCS via S3 API, please enable features = ["native-tls"] in your Cargo.toml to avoid connection being reset when using rustls. Tracking in https://github.com/seanmonstar/reqwest/issues/1809

Ceph Rados Gateway

Ceph supports a RESTful API that is compatible with the basic data access model of the Amazon S3 API.

For more information, refer: https://docs.ceph.com/en/latest/radosgw/s3/

Implementations

impl S3Builder

pub fn root(&mut self, root: &str) -> &mut Self

Set root of this backend.

All operations will happen under this root.

pub fn bucket(&mut self, bucket: &str) -> &mut Self

Set bucket name of this backend.

pub fn endpoint(&mut self, endpoint: &str) -> &mut Self

Set endpoint of this backend.

Endpoint must be full uri, e.g.

• AWS S3: https://s3.amazonaws.com or https://s3.{region}.amazonaws.com
• Cloudflare R2: https://<ACCOUNT_ID>.r2.cloudflarestorage.com
• Aliyun OSS: https://{region}.aliyuncs.com
• Tencent COS: https://cos.{region}.myqcloud.com
• Minio: http://127.0.0.1:9000

If user inputs endpoint without scheme like “s3.amazonaws.com”, we will prepend “https://” before it.

pub fn region(&mut self, region: &str) -> &mut Self

Region represent the signing region of this endpoint. This is required if you are using the default AWS S3 endpoint.

If using a custom endpoint,

• If region is set, we will take user’s input first.
• If not, we will try to load it from environment.

pub fn access_key_id(&mut self, v: &str) -> &mut Self

Set access_key_id of this backend.

• If access_key_id is set, we will take user’s input first.
• If not, we will try to load it from environment.

pub fn secret_access_key(&mut self, v: &str) -> &mut Self

Set secret_access_key of this backend.

• If secret_access_key is set, we will take user’s input first.
• If not, we will try to load it from environment.

pub fn role_arn(&mut self, v: &str) -> &mut Self

Set role_arn for this backend.

If role_arn is set, we will use already known config as source credential to assume role with role_arn.

pub fn external_id(&mut self, v: &str) -> &mut Self

Set external_id for this backend.

pub fn default_storage_class(&mut self, v: &str) -> &mut Self

Set default storage_class for this backend.

Available values:

• DEEP_ARCHIVE
• GLACIER
• GLACIER_IR
• INTELLIGENT_TIERING
• ONEZONE_IA
• OUTPOSTS
• REDUCED_REDUNDANCY
• STANDARD
• STANDARD_IA

pub fn server_side_encryption(&mut self, v: &str) -> &mut Self

Set server_side_encryption for this backend.

Available values: AES256, aws:kms.

Note

This function is the low-level setting for SSE related features.

SSE related options should be set carefully to make them works. Please use server_side_encryption_with_* helpers if even possible.

pub fn server_side_encryption_aws_kms_key_id(&mut self, v: &str) -> &mut Self

Set server_side_encryption_aws_kms_key_id for this backend

• If server_side_encryption set to aws:kms, and server_side_encryption_aws_kms_key_id is not set, S3 will use aws managed kms key to encrypt data.
• If server_side_encryption set to aws:kms, and server_side_encryption_aws_kms_key_id is a valid kms key id, S3 will use the provided kms key to encrypt data.
• If the server_side_encryption_aws_kms_key_id is invalid or not found, an error will be returned.
• If server_side_encryption is not aws:kms, setting server_side_encryption_aws_kms_key_id is a noop.

Note

This function is the low-level setting for SSE related features.

SSE related options should be set carefully to make them works. Please use server_side_encryption_with_* helpers if even possible.

pub fn server_side_encryption_customer_algorithm( &mut self, v: &str ) -> &mut Self

Set server_side_encryption_customer_algorithm for this backend.

Available values: AES256.

Note

This function is the low-level setting for SSE related features.

SSE related options should be set carefully to make them works. Please use server_side_encryption_with_* helpers if even possible.

pub fn server_side_encryption_customer_key(&mut self, v: &str) -> &mut Self

Set server_side_encryption_customer_key for this backend.

Args

v: base64 encoded key that matches algorithm specified in server_side_encryption_customer_algorithm.

Note

This function is the low-level setting for SSE related features.

SSE related options should be set carefully to make them works. Please use server_side_encryption_with_* helpers if even possible.

pub fn server_side_encryption_customer_key_md5(&mut self, v: &str) -> &mut Self

Set server_side_encryption_customer_key_md5 for this backend.

Args

v: MD5 digest of key specified in server_side_encryption_customer_key.

Note

This function is the low-level setting for SSE related features.

SSE related options should be set carefully to make them works. Please use server_side_encryption_with_* helpers if even possible.

pub fn server_side_encryption_with_aws_managed_kms_key(&mut self) -> &mut Self

Enable server side encryption with aws managed kms key

As known as: SSE-KMS

NOTE: This function should not be used along with other server_side_encryption_with_ functions.

pub fn server_side_encryption_with_customer_managed_kms_key( &mut self, aws_kms_key_id: &str ) -> &mut Self

Enable server side encryption with customer managed kms key

As known as: SSE-KMS

NOTE: This function should not be used along with other server_side_encryption_with_ functions.

pub fn server_side_encryption_with_s3_key(&mut self) -> &mut Self

Enable server side encryption with s3 managed key

As known as: SSE-S3

NOTE: This function should not be used along with other server_side_encryption_with_ functions.

pub fn server_side_encryption_with_customer_key( &mut self, algorithm: &str, key: &[u8] ) -> &mut Self

Enable server side encryption with customer key.

As known as: SSE-C

NOTE: This function should not be used along with other server_side_encryption_with_ functions.

pub fn security_token(&mut self, token: &str) -> &mut Self

Set temporary credential used in AWS S3 connections

Warning

security token’s lifetime is short and requires users to refresh in time.

pub fn disable_config_load(&mut self) -> &mut Self

Disable config load so that opendal will not load config from environment.

For examples:

• envs like AWS_ACCESS_KEY_ID
• files like ~/.aws/config

pub fn disable_ec2_metadata(&mut self) -> &mut Self

Disable load credential from ec2 metadata.

This option is used to disable the default behavior of opendal to load credential from ec2 metadata, a.k.a, IMDSv2

pub fn allow_anonymous(&mut self) -> &mut Self

Allow anonymous will allow opendal to send request without signing when credential is not loaded.

pub fn enable_virtual_host_style(&mut self) -> &mut Self

Enable virtual host style so that opendal will send API requests in virtual host style instead of path style.

• By default, opendal will send API to https://s3.us-east-1.amazonaws.com/bucket_name
• Enabled, opendal will send API to https://bucket_name.s3.us-east-1.amazonaws.com

pub fn disable_stat_with_override(&mut self) -> &mut Self

Disable stat with override so that opendal will not send stat request with override queries.

For example, R2 doesn’t support stat with response_content_type query.

pub fn customed_credential_load( &mut self, cred: Box<dyn AwsCredentialLoad> ) -> &mut Self

Adding a customed credential load for service.

If customed_credential_load has been set, we will ignore all other credential load methods.

pub fn http_client(&mut self, client: HttpClient) -> &mut Self

Specify the http client that used by this service.

Notes

This API is part of OpenDAL’s Raw API. HttpClient could be changed during minor updates.

pub fn batch_max_operations(&mut self, batch_max_operations: usize) -> &mut Self

Set maximum batch operations of this backend.

pub fn checksum_algorithm(&mut self, checksum_algorithm: &str) -> &mut Self

Set checksum algorithm of this backend. This is necessary when writing to AWS S3 Buckets with Object Lock enabled for example.

Available options:

• “crc32c”

pub async fn detect_region(endpoint: &str, bucket: &str) -> Option<String>

Detect region of S3 bucket.

Args

• endpoint: the endpoint of S3 service
• bucket: the bucket of S3 service

Return

• Some(region) means we detect the region successfully
• None means we can’t detect the region or meeting errors.

Notes

We will try to detect region by the following methods.

• Match endpoint with given rules to get region
  • Cloudflare R2
  • AWS S3
  • Aliyun OSS
• Send a HEAD request to endpoint with bucket name to get x-amz-bucket-region.

Examples

use opendal::services::S3;

let region: Option<String> = S3::detect_region("https://s3.amazonaws.com", "example").await;

Reference

• Amazon S3 HeadBucket API

Trait Implementations

impl Builder for S3Builder

const SCHEME: Scheme = Scheme::S3

Associated scheme for this builder. It indicates what underlying service is.

type Accessor = S3Backend

The accessor that built by this builder.

fn from_map(map: HashMap<String, String>) -> Self

Construct a builder from given map which contains several parameters needed by underlying service.

fn build(&mut self) -> Result<Self::Accessor>

Consume the accessor builder to build a service.

impl Debug for S3Builder

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for S3Builder

fn default() -> S3Builder

Returns the “default value” for a type.