What is Reqsign?​

Reqsign stands for Request Signing. Its goal is to make signing API requests simple. Many APIs look straightforward, but they quickly become complicated once hidden behind complex abstractions.

One major challenge is authentication. For example, AWS V4 requires users to build a signature from the request and sign it using HMAC with a secret key. Another challenge is credential loading. Cloud providers like AWS, Azure, and GCP support various types of credentials and loading methods to balance security and ease of use.

All these features together make simple APIs feel complex. To send a single GetObject call, you often need to rely on many AWS crates and go through deep abstraction layers.

Reqsign's goal is to bring simplicity back: build, sign, send.

In general, Reqsign provides three main parts for different services:

• ProvideCredential: Lets users provide credentials in different ways such as environment variables, profiles, IMDS, OIDC, and more. Reqsign also provides a default chain for every service so it works out of the box without extra configuration.
• SignRequest: Implements the signing logic so that requests are correctly signed with the given provider credentials.
• Context: Offers a pluggable context mechanism that supports HTTP send, command execution, and environment access. This allows users to configure the runtime context as they wish and makes Reqsign available in WASM.

For example:

use anyhow::Result;
use reqsign::aws;

#[tokio::main]
async fn main() -> Result<()> {
    // Create a default signer for S3 in us-east-1
    // 
    // This will automatically:
    // - Load credentials from environment variables, config files, or IAM roles
    // - Set up the default HTTP client and file reader
    let signer = aws::default_signer("s3", "us-east-1");
    
    // Build your request
    let mut req = http::Request::builder()
        .method("GET")
        .uri("https://s3.amazonaws.com/testbucket")
        .body(())
        .unwrap()
        .into_parts()
        .0;
    
    // Sign the request
    signer.sign(&mut req, None).await?;
    
    // Send the request with your preferred HTTP client
    println!("Request has been signed!");
    Ok(())
}

Currently, Reqsign supports Aliyun OSS, AWS V4, Azure Storage, Google, Huawei Cloud OBS, Oracle, and Tencent COS. The community is working on adding more services so that users do not need to reimplement them.

What has changed and what has not?​

After this donation, Reqsign has become a subproject of Apache OpenDAL. The Apache OpenDAL PMC will govern the project under the Apache Way. The repository has been transferred from Xuanwo/reqsign to apache/opendal-reqsign. All Reqsign committers are now OpenDAL committers.

Reqsign is licensed under Apache 2.0. Joining the ASF strengthens this license and makes it permanent. Everyone can now use Reqsign in a safe and trusted way. The community will dedicate more effort to fixing bugs and adding new features to help Reqsign grow into a mature project.

What’s next?​

The OpenDAL community is working to make Reqsign ready as a mature ASF project and integrate it with Reqsign v1.0. We also plan to move some existing service implementations into Reqsign for better maintainability.

Welcome to join us!

I hope this post serves as a guide for our development, encourages more people to contribute, and ultimately helps achieve the vision of OpenDAL.

What's OpenDAL?​

Apache OpenDAL (/ˈoʊ.pən.dæl/, pronounced "OH-puhn-dal") is an Open Data Access Layer that enables seamless interaction with diverse storage services. Our VISION is One Layer, All Storage, and our core principles are Open Community, Solid Foundation, Fast Access, Object Storage First, and Extensible Architecture.

We are building:

• A core library built in Rust that supports all services at zero cost and offers unified retry, concurrency, logging, tracing, metrics, timeout and more layers.
• Language bindings for Python, Java, Node.js, C/C++, and more.
• Integrations with various frameworks, including Parquet, FUSE, DAV server, and others.
• Binaries for different use cases, such as CLI and FUSE.

In short, through OpenDAL, users can access ALL storage services within ONE layer.

Where's OpenDAL?​

OpenDAL's rust core has already released 143 versions, has 67 reverse dependencies listed on crates.io, and is used by 612 projects as recorded on GitHub.

OpenDAL's production users include databases such as Databend, GreptimeDB, and RisingWave, as well as tools like Loco, sccache, and Vector.

Apart from OpenDAL's Rust core, its various language bindings have also seen significant growth over the past year. Take python binding as an example. Dify, an LLM app development platform, is using OpenDAL to access different storage services.

What's next for OpenDAL?​

The following is the famous technology adoption lifecycle curve. If I were to indicate the position of OpenDAL, I would say it is at the end of the Innovators stage and moving toward the Early Adopters stage.

(picture from TechTarget technology adoption lifecycle)

Innovators are adopting OpenDAL. Projects like Databend, RisingWave, GreptimeDB, and sccache have been using OpenDAL in production for years. However, early adopters are still hesitant to use OpenDAL—and they have valid reasons.

For examples:

• OpenDAL hasn't reached version 1.0 yet and still introduces breaking changes from time to time, sometimes even requiring code modifications for an OpenDAL upgrade. This also creates a burden for libraries that depend on OpenDAL, as every breaking change affects them as well.
• OpenDAL lacks comprehensive documentation, particularly for its bindings in Python and Java. Users need to invest significant effort and love to integrate OpenDAL bindings into their projects.
• OpenDAL lacks some important features, such as checksum support, caching, metrics of underlying http requests and initialization from a URI.

I believe we should perfect production adoption in 2025 to get OpenDAL ready for early adopters. Only in this way can we implement our vision: One Layer, All Storage.

What to do in 2025?​

Our plans for 2025 include the following:

Features Needed in Production​

In 2025, we plan to implement the following features that are importmant for production adoption:

• Context: Introduce context in OpenDAL so that services and layers can share the same context. This enables users to gain deeper insights into OpenDAL's internal operations by incorporating metrics, logging, and tracing into the underlying HTTP client we use.
• Versioning: Introduce full file versioning support in OpenDAL, enabling users to read, write, delete, list, and restore versioned files. This functionality will allow users to recover mistakenly deleted files and facilitate disaster recovery.
• Checksum: Introduce end-to-end checksum support in OpenDAL, enabling users to perform checksums during reading and writing without worrying about bit flips in memory or over the network.
• Caching: Provide high-quality built-in cache support in OpenDAL while ensuring users have the flexibility to implement their own caching logic.
• Initialization From URI: Allow users to initialize OpenDAL from a URI string, making it easier to configure and use OpenDAL in various environments.

Improvements Needed for Production​

In 2025, we plan to improve the following aspects of OpenDAL to ensure users can confidently use it in production.

• Documentation: Improve the documentation for OpenDAL, particularly for bindings like Python, Node.js, and Java. The first step is to generate well-structured documentation for the configuration values of each service.
• Communitation: Bring Back Our Community Meetings. OpenDAL used to hold regular tri-weekly meetings, but we have been unable to maintain them in the last year. We now plan to revive these meetings and encourage more face-to-face discussions in the coming years.

Conclusion​

2025 marks the third year of the OpenDAL community. A huge thanks to all OpenDAL contributors and users for helping us reach this milestone. There's still a long way to go to achieve our vision, and we invite you all to join us on this incredible journey!

Discuss about this post here

On behalf of the OpenDAL community, I'm pleased to announce that the first Apache OpenDAL™ Beijing Meetup will be held on September 14th.

You are welcome to register for this event at: https://docs.qq.com/form/page/DYnpJWFFsSWVFeGRj

The event will take place at 北京市朝阳区北辰西路 8 号北辰世纪中心A座12层1215房间（国家会议中心西面的写字楼） on the afternoon of September 14th.

Agenda​

• 12:00 - 13:00: Lunch / Networking (food provided)
• 13:00 - 14:00: Welcome / Introductions
• 14:00 - 15:00: Talks
• 15:00 - 15:30: Coffee / Email Break
• 15:30 - 16:30: Breakouts / Discussions
• 16:30 - 17:00: Close / Conclusion

Welcome​

Brief introductory remarks, thank to the sponsor, outline of the day, participant introductions (name, github handle, optional affiliation, optional reason you came to the meetup)

Talks​

Five 10-15 minute talks about opendal.

• @tisonkun about OpenDAL Community Growth
• @WenyXu for using OpenDAL in GreptimeDB
• @wcy-fdu for using OpenDAL in RisingWave
• @drmingdrmer for using OpenDAL in OpenRaft
• @Xuanwo about OpenDAL Core 1.0

Breakouts / Discussions​

The idea is that we will break into small groups for 3 x 20 minute sessions. I will ring a gong or something similar every 20 minutes and you should find a new group to discuss.

Potential seed topics (but obviously feel free to discuss anything):

• How do we make it easier for people to build systems on OpenDAL? What are the current barriers to community growth as you see them?
• How do we make it easier for people to contribute back to OpenDAL? What are the current barriers to contribution
• What features would you like to see / not see

Ideally, in each session, one or more people would volunteer to write up any insights they got from the discussions, and post them to the overall ticket.

Looking forward to meet you in this meetup!

We're writing this blog post to announce that the Apache OpenDAL™ Project will be participating in Open Source Promotion Plan (OSPP) 2024. If you're not eligible or interested in participating in OSPP, then most of this post likely isn't relevant to you; if you are, this should contain some useful information and links.

Open Source Promotion Plan is a summer program organized by the Institute of Software Chinese Academy of Sciences and long-term supported by the Open Source Software Supply Chain Promotion Plan. It aims to encourage college students to actively participate in the maintenance and development of open source software, promote the vigorous development of open source software communities, and build the open source software supply chain together.

The OSPP applicants now have several weeks to send project proposals to organizations that appeal to them. If their project proposal is accepted, they will embark on a 12-week journey during which they will try to complete their proposed project under the guidance of an assigned mentor.

We have prepared a list of project ideas that can serve as inspiration for potential OSPP contributors that would like to send a project proposal to the OpenDAL project. You can try to find mentors on the maillist or Discord. We have also prepared a proposal guide that should help you with preparing your project proposals.

You can start discussing the project ideas with OpenDAL Project maintainers immediately. The project proposal application period starts on April 30, 2024, and ends on June 4, 2024. Take note of that deadline, as there will be no extensions!

If you are interested in contributing to the OpenDAL Project, we encourage you to check out our project idea list and send us a OSPP project proposal! Of course, you are also free to discuss these projects and/or try to move them forward even if you do not intend to (or cannot) participate in OSPP. We welcome all contributors to OpenDAL, as there is always enough work to do.

We are excited about this event. Hoping you all feel the same way!

This announcement is inspired a lot by the Rust participates in Google Summer of Code 2024.

We're writing this blog post to announce that the Apache OpenDAL™ Project will be participating in Google Summer of Code (GSoC) 2024. If you're not eligible or interested in participating in GSoC, then most of this post likely isn't relevant to you; if you are, this should contain some useful information and links.

Google Summer of Code (GSoC) is an annual global program organized by Google that aims to bring new contributors to the world of open-source. The program pairs organizations (such as the OpenDAL Project) with contributors (usually students), with the goal of helping the participants make meaningful open-source contributions under the guidance of experienced mentors.

Google is sponsoring the 2024 Summer of Code and The Apache Software Foundation (ASF) registered as a mentoring organization. The GSoC applicants now have several weeks to send project proposals to organizations that appeal to them. If their project proposal is accepted, they will embark on a 12-week journey during which they will try to complete their proposed project under the guidance of an assigned mentor.

We have prepared a list of project ideas that can serve as inspiration for potential GSoC contributors that would like to send a project proposal to the OpenDAL project. However, applicants can also come up with their own project ideas. You can discuss project ideas or try to find mentors on the maillist or Discord. We have also prepared a proposal guide that should help you with preparing your project proposals.

You can start discussing the project ideas with OpenDAL Project maintainers immediately. The project proposal application period starts on March 18, 2024, and ends on April 2, 2024 at 18:00 UTC. Take note of that deadline, as there will be no extensions!

If you are interested in contributing to the OpenDAL Project, we encourage you to check out our project idea list and send us a GSoC project proposal! Of course, you are also free to discuss these projects and/or try to move them forward even if you do not intend to (or cannot) participate in GSoC. We welcome all contributors to OpenDAL, as there is always enough work to do.

We are excited about this event. Hoping you all feel the same way!

This announcement is inspired a lot by the Rust participates in Google Summer of Code 2024.

What's Apache OpenDAL?​

Apache OpenDAL is a data access layer that allows users to easily and efficiently retrieve data from various storage services in a unified way. Our VISION is access data freely.

OpenDAL could be used as a better SDK for your storage services: A SDK with native integration of retry, logging, metrics, tracing, timeout, throttle, and more.

OpenDAL could be used as a super connector for your storage services: A connector that supports all kinds of storage services from Object Storage (s3, gcs, azblob), File Storage (fs, azdls, hdfs), Consumer Cloud Storage (gdrive, onedrive), Key-Value Storage (rocksdb, sled) to Cache Storage (memcached, moka).

OpenDAL could be used as an elegant client for your storage services: A client with well designed API and many language bindings: Rust, C, Cpp, Dotnet, Go, Haskell, Java, Lua, Node.js, OCaml, PHP, Python, Ruby, Swift and Zig.

Need to access data? Give OpenDAL a try!

async fn main() -> Result<()> {
    // Init s3 service.
    let mut builder = services::S3::default();
    builder.bucket("test");

    // Init an operator
    let op = Operator::via_map(builder)?
        // Add logging
        .layer(LoggingLayer::default())
        .finish();

    // Write data
    op.write("hello.txt", "Hello, World!").await?;

    // Read data
    let bs = op.read("hello.txt").await?;

    // Fetch metadata
    let meta = op.stat("hello.txt").await?;
    let mode = meta.mode();
    let length = meta.content_length();

    // Delete
    op.delete("hello.txt").await?;

    Ok(())
}

What's the ASF?​

The Apache Software Foundation (ASF) is a nonprofit corporation to support a number of open-source software projects. The Apache Software Foundation exists to provide software for the public good. We believe in the power of community over code, known as The Apache Way. Thousands of people around the world contribute to ASF open source projects every day.

The OpenDAL Community believes the apache way that:

• Earned Authority: all individuals are given the opportunity to participate, but their influence is based on publicly earned merit – what they contribute to the community.
• Community of Peers: individuals participate at the ASF, not organizations.
• Open Communications: as a virtual organization, the ASF requires all communications related to code and decision-making to be publicly accessible to ensure asynchronous collaboration, as necessitated by a globally-distributed community.
• Consensus Decision Making: Apache Projects are overseen by a self-selected team of active volunteers who are contributing to their respective projects.
• Responsible Oversight: The ASF governance model is based on trust and delegated oversight.

The original creators Databend chosen to contribute OpenDAL to the ASF, embracing the Apache way through joining the incubator program.

What's graduation?​

In the Apache Incubator, the OpenDAL community is learning the Apache Way through daily development activities, growing its community and producing Apache releases.

During the incubation, we:

• Consist of 19 committers, including mentors, with 12 serving as PPMC members.
• Boast 164 contributors.
• Made 9 releases—averaging at least one per month.
• Had 7 different release managers to date.
• Used by 10 known entities and is a dependency for 263 GitHub projects and 18 crates.io packages.
• Opened 1,200+ issues with 1,100+ successfully resolved.
• Submitted a total of 2,400+ PRs, most of them have been merged or closed.

The graduation signifies that the OpenDAL Community is recognized as a mature community, which entails:

• CODE: OpenDAL is an Apache 2.0 licensed open-source project with accessible, buildable code on GitHub, featuring a traceable history and authenticated code provenance.
• LICENSE: OpenDAL maintains open-source compliance for all code and dependencies, requires contributor agreements, and clearly documents copyright ownership.
• Releases: OpenDAL offers standardized, committee-approved source code releases with secure signatures, provides convenience binaries, and has a well-documented, repeatable release process.
• Quality: OpenDAL is committed to code quality transparency, prioritizes security with quick issue responses, ensures backward compatibility with clear documentation, and actively addresses bug reports in a timely manner.
• Community: OpenDAL offers a comprehensive homepage, welcomes diverse contributions, promotes a meritocratic approach for active contributors, operates on community consensus, and ensures timely responses to user queries through various channels.
• Consensus: OpenDAL has a public list of key decision-makers and uses a consensus approach for decisions, documented on its main communication channel. It follows standard voting rules and records all important discussions in writing.
• Independence: OpenDAL is independent, with contributors from various companies acting on their own, not as representatives of any organization.

What's next?​

After graduation, OpenDAL Community will continue to focus on the following aspects under the VISION: access data freely.

More Stable Services​

OpenDAL now supports 59 services, although only some of them are stable.

stable for OpenDAL means that

• Have integration tests covered.
• Have at least one production user.

The stable service established a feedback loop between the OpenDAL community and its users. Users can submit bug reports or feature requests to the OpenDAL community, which in turn can enhance the service using this feedback while ensuring existing features remain intact.

After graduation, the OpenDAL community will focus on improving the stability of current services instead of just expanding our offerings.

We plan to:

• Add features users wanted to services like file version, concurrently list and glob pattern.
• Add integration tests for newly added services.

More Useful Documents​

OpenDAL have good docs for its rust core, but not for other language bindings.

The lack of comprehensive documentation makes OpenDAL challenging for users to operate in Java or Python. Without user feedback, the community is unable to enhance this documentation, leading to a detrimental cycle that must be broken.

After graduation, the OpenDAL community will improve the documentation of other language bindings.

We plan to:

• Introduce code generation to automatically create documentation for the service builder due to its numerous configurations.
• Add more API Docs and examples for other language bindings.

OpenDAL have good docs for its public API, but not for its internal design.

OpenDAL is proud of its elegant design, but it is not well documented. This makes it difficult for new contributors to understand the codebase and make contributions.

After graduation, the OpenDAL community will improve the documentation of its internal design.

We plan to:

• Optimize the codebase to make it easier to understand.
• Add more blog posts to explain the design of OpenDAL.

More Production Users​

OpenDAL requires more production users, as they are vital to the success of our project. Increased user production leads to more valuable feedback, a more engaged contributor base, and a stronger community. We've started the initial loop; let's expand it!

After graduation, the OpenDAL community will focus on attracting more production users.

We plan to:

• Optimize the feature set for adoption like uri initiation and config.
• Expand more ways to use OpenDAL via fuse, cli, S3/WebDAV API, object_store binding.

Conclusion​

The OpenDAL Community aims to create a world where users can freely access data across any storage service in any manner they choose. Graduation is just the beginning—let's work together to make our VISION a reality!

OwO (Outcome, Working, Outlook) is an Apache OpenDAL™ release blog series, where we share the current work status and future plans.

Hello! It's been a while since our last update. We've been hard at work determining the optimal way to implement new features and improvements. We're thrilled to announce that we'll soon be releasing v0.40.

This post is structured into three main sections:

• Outcome (1st O in OwO): Summarizes the key accomplishments in the v0.40 release.
• Working (the w in OwO): Provides an update on our current work.
• Outlook (2nd O in OwO): Discusses what lies ahead for OpenDAL.

Outcome​

OpenDAL now comprises four primary components:

• Core: The core library written in Rust.
• Bindings: Language bindings powered by the OpenDAL Rust core.
• Applications: Applications built using the OpenDAL Rust core.
• Integrations: Collaborations with other projects.

Core​

Unifying Append and Write Functions​

OpenDAL has supported append operations since v0.36. We've found, however, that this led to significant duplication between append and write. As a result, we've streamlined the two functionalities into a single write function. Our users can now:

let mut w = op.writer_with("test.txt").append(true).await?;
w.write(content_a).await?;
w.write(content_b).await?;
w.close().await?;

This way, users can reuse the Writer in their own logic without handling append separately.

New Lister API​

To improve API consistency, we've made some adjustments to our listing functions. We've added list and list_with methods that perform single operations and renamed the original list to lister and lister_with.

// Old API
let lister: Lister = op.list("dir").await?;

// New API
let entries: Vec<Entry> = op.list("dir").await?;
let lister: Lister = op.lister("dir").await?;

This brings uniformity to our API offerings.

List With Metakey​

To speed up list operations, OpenDAL can now fetch and store metadata during the listing process. This eliminates the need for separate metadata calls:

let entries: Vec<Entry> = op
  .list_with("dir/")
  .metakey(Metakey::ContentLength | Metakey::ContentType).await?;

// Use the metadata directly!
let meta: &Metadata = entries[0].metadata();

This makes metadata retrieval more intuitive.

Buffered Writer​

We've added general buffer support to optimize writing operations.

let w = op.writer_with("path/to/file").buffer(8 * 1024 * 1024).await?

Others​

Other improvements in the core library can be found in our CHANGELOG.

Bindings​

C++​

opendal-cpp is ready for its first release! Welcome to check it out and give us some feedback.

Haskell​

opendal-hs is ready for its first release! Welcome to check it out and give us some feedback.

Java​

opendal-java enabled more available services in this release, allowing user to visit services like redis that not enabled by default in rust core. And opendal-java enabled blocking layer to allow users visit services like s3 in blocking way.

Welcome to integrate opendal-java into your project and give us some feedback.

New bindings!​

• opendal-dotnet
• opendal-php

Applications​

oay​

oay is OpenDAL Gateway that allows users to access OpenDAL services via existing protocols like s3 and webdav. It works like a proxy that forwarding requests to OpenDAL services.

In this release, we implement basic webdav support. Users can convert any storage services to a webdav server!

oli​

oli is OpenDAL CLI that allows users to access storage services via CLI like s3cmd and gcloud does.

We fixed some experience issues in this release and improved some docs. Welcome to try it out and give us some feedback.

Integrations​

object_store​

object_store instead to implement object_store's trait over OpenDAL Operator so that users can use OpenDAL as a backend for object_store.

object_store is mostly functional, but there are some edge use cases that OpenDAL has yet to support.

So far, this release hasn't seen progress in this area; we are awaiting the resolution of the issue Allow list paths that do not end with /.

Working​

We are working on the following things:

• object_store support: Make object_store integration works and find a user for it.
• Remove the / limitation for path, so we can list a path without ending with /.
• Expand the start-after support to more services (Address #2786).

Outlook​

We are exploring some innovative ideas:

• OpenDAL REST/gRPC API: A REST/gRPC Server for OpenDAL.
• OpenDAL Cache: OpenDAL native cache libs that allowing users to access data more efficiently.
• OpenDAL File System: A read-only file system that built upon OpenDAL in rust!
• kio-opendal: A kio plugin powered by OpenDAL that allows users to visit different storage services in KDE Dolphin.
• gvfs-opendal: A gvfs plugin powered by OpenDAL that allows users to visit different storage services in GNOME Files

Feel free to join in the discussion!

Summary​

This marks our first OpenDAL OwO post. We welcome your feedback.

The first article will discuss OpenDAL's most commonly used data reading function. I will start from the outermost interface and then gradually unfold according to the calling sequence of OpenDAL. Let's get started!

Overall Framework​

Before starting to introduce the specific OpenDAL interface, let's first get familiar with the OpenDAL project.

OpenDAL is an Apache Incubator project aimed at helping users access data from various storage services in a unified, convenient, and efficient way. Its project vision is "free access to data":

• Free from services: Any service can be accessed freely through native interfaces
• Free from implementations: No matter how the underlying implementation is, it can be called in a unified way
• Free to integrate: Able to freely integrate with various services and languages
• Free to zero cost: Users don't have to pay for features they don't use

On this philosophical foundation, OpenDAL Rust Core can be mainly divided into the following components:

• Operator: The outer interface exposed to users
• Layers: Specific implementation of different middleware
• Services: Specific implementation of different services

From a macroscopic perspective, OpenDAL's data reading call stack would look like this:

All Layers and Services have implemented a unified Accessor interface, erasing all type information when building the Operator. For the Operator, regardless of what services are used or how many middleware are added, all call logic is consistent. This design splits OpenDAL's API into Public API and Raw API, where the Public API is directly exposed to users, providing convenient top-level interfaces, and Raw API is provided to OpenDAL internal developers, maintaining a unified internal interface and providing some convenient implementation.

Operator​

OpenDAL's Operator API will adhere to a consistent calling paradigm as much as possible, reducing users' learning and usage costs. For example, OpenDAL offers the following APIs for read:

• op.read(path): Reads the entire content of the specified file
• op.reader(path): Creates a Reader for streaming reading
• op.read_with(path).range(1..1024): Reads file content using specified parameters, such as range
• op.reader_with(path).range(1..1024): Creates a Reader for streaming reading with specified parameters

It's not hard to see that read is more like syntactic sugar, allowing users to quickly read files without considering various traits like AsyncRead. The reader provides more flexibility, implementing widely-used community traits like AsyncSeek, AsyncRead, allowing more flexible data reading. read_with and reader_with assist users in specifying various parameters in a more natural way through Future Builder functions.

The internal logic of the Operator would look like this:

Its main job is to encapsulate the interface for the user:

• Completing the construction of OpRead: the args for read operation.
• Calling the read function provided by Accessor
• Wrapping the returned value as Reader and implementing interfaces like AsyncSeek, AsyncRead, etc., based on Reader

Layers​

A little secret here is that OpenDAL will automatically apply some Layers to the Service to implement some internal logic. As of the completion of this article, OpenDAL's automatically added Layers include:

• ErrorContextLayer: Injects context information, such as scheme, path, etc., into all returned errors of Operation
• CompleteLayer: Adds necessary capabilities to services, such as adding seek support to s3
• TypeEraseLayer: Implements type erasure, uniformly erasing associated types in Accessor, so users don't need to carry generic parameters when using it

Here, ErrorContextLayer and TypeEraseLayer are relatively simple and won't be elaborated on. The focus is on CompleteLayer, aimed at adding seek or next support to OpenDAL's returned Reader in a zero-cost way, so users don't have to re-implement it. OpenDAL initially returned Reader and SeekableReader through different function calls in early versions, but the actual user feedback was not very good; almost all users were using SeekableReader. Therefore, OpenDAL subsequently added seek support as the first priority to the internal Read trait during the refactor:

pub trait Read: Unpin + Send + Sync {
    /// Read bytes asynchronously.
    fn poll_read(&mut self, cx: &mut Context<'_>, buf: &mut [u8]) -> Poll<Result<usize>>;

    /// Seek asynchronously.
    ///
    /// Returns `Unsupported` error if underlying reader doesn't support seek.
    fn poll_seek(&mut self, cx: &mut Context<'_>, pos: io::SeekFrom) -> Poll<Result<u64>>;

    /// Stream [`Bytes`] from underlying reader.
    ///
    /// Returns `Unsupported` error if underlying reader doesn't support stream.
    ///
    /// This API exists for avoiding bytes copying inside async runtime.
    /// Users can poll bytes from underlying reader and decide when to
    /// read/consume them.
    fn poll_next(&mut self, cx: &mut Context<'_>) -> Poll<Option<Result<Bytes>>>;
}

To implement a service's reading capability in OpenDAL, one needs to implement this trait, which is an internal interface and will not be directly exposed to users. Among them:

• poll_read is the most basic requirement; all services must implement this interface.
• When the service natively supports seek, poll_seek can be implemented, and OpenDAL will correctly dispatch, such as local fs;
• When the service natively supports next, meaning it returns streaming Bytes, poll_next can be implemented, like HTTP-based services, where the underlying layer is a TCP Stream, and hyper will encapsulate it as a bytes stream.

Through the Read trait, OpenDAL ensures that all services can expose their native support capabilities as much as possible, thereby achieving efficient reading for different services.

Based on this trait, OpenDAL will complete according to the capabilities supported by each service:

• Both seek/next are supported: Direct return
• No support for next: Encapsulate using StreamableReader to simulate next support
• No support for seek: Encapsulate using ByRangeSeekableReader to simulate seek support
• Neither seek/next supported: Encapsulate using both methods

ByRangeSeekableReader mainly utilizes the service's ability to support range read, dropping the current reader when the user seeks and initiating a new request at the specified location.

OpenDAL exposes a unified Reader implementation through CompleteLayer, so users don't have to worry about whether the underlying service supports seek; OpenDAL will always choose the optimal way to initiate the request.

Services​

After the completion of the Layers, it's time to call the specific implementation of the Service. Here, the most common services fs and s3 are used as examples to explain how data is read.

Service fs​

tokio::fs::File implements tokio::AsyncRead and tokio::AsyncSeek. Using async_compat::Compat, we have transformed it into futures::AsyncRead and futures::AsyncSeek. Based on this, we provide a built-in function oio::into_read_from_file to transform it into a type that implements oio::Read.

There's nothing particularly complex in the implementation of oio::into_read_from_file; read and seek are mostly calling the functions provided by the incoming File type. The tricky part is about the correct handling of seek and range: seeking to the right side of the range is allowed, and this will not cause an error, and reading will only return empty, but seeking to the left side of the range is illegal, and the Reader must return InvalidInput for proper upper-level handling.

Interesting history: there was an issue in the initial implementation of this part, discovered during fuzz testing.

Services s3​

S3 is an HTTP-based service, and opendal provides a lot of HTTP-based wrappers to help developers reuse logic; they only need to build a request and return a well-constructed Body. OpenDAL Raw API encapsulates a set of reqwest-based interfaces, and the HTTP GET interface returns a Response<IncomingAsyncBody>:

/// IncomingAsyncBody carries the content returned by remote servers.
pub struct IncomingAsyncBody {
    /// # TODO
    ///
    /// hyper returns `impl Stream<Item = crate::Result<Bytes>>` but we can't
    /// write the types in stable. So we will box here.
    ///
    /// After [TAIT](https://rust-lang.github.io/rfcs/2515-type_alias_impl_trait.html)
    /// has been stable, we can change `IncomingAsyncBody` into `IncomingAsyncBody<S>`.
    inner: oio::Streamer,
    size: Option<u64>,
    consumed: u64,
    chunk: Option<Bytes>,
}

The stream contained in this body is the bytes stream returned by reqwest, and opendal implements content length checks and read support on this basis.

Here's an extra note about a small pitfall with reqwest/hyper: reqwest and hyper do not check the returned content length, so an illegal server may return a data volume that does not match the expected content length instead of an error, leading to unexpected data behavior. OpenDAL specifically added checks here, returning ContentIncomplete when data is insufficient and ContentTruncated when data exceeds expectations, avoiding users receiving illegal data.

Conclusion​

This article introduces from top to bottom how OpenDAL implements data reading:

• Operator is responsible for exposing user-friendly interfaces
• Layers are responsible for completing the capabilities of the services
• Services are responsible for the specific implementation of different services

Throughout the entire chain, OpenDAL adheres as much as possible to the principle of zero cost, prioritizing the use of native service capabilities, then considering simulation through other methods, and finally returning unsupported errors. Through this three-tier design, users don't need to understand the details of the underlying service, nor do they need to integrate different service SDKs to easily call op.read(path) to access data in any storage service.

This is: How OpenDAL read data freely!

What is OpenDAL?​

OpenDAL is a data access layer that allows users to easily and efficiently retrieve data from various storage services in a unified way.

Data Access Layer means: OpenDAL is in a critical position in the data read-write process. We shield the implementation details of different storage backends and provide a set of unified interface abstractions externally.

Next, let's try to answer "What OpenDAL is not" and deconstruct OpenDAL from another perspective:

Opendal Is Not a Proxy Service​

OpenDAL is provided in the form of a library, not as a service or application that proxies various storage backends.

If you want to integrate OpenDAL into an existing project, you need to call OpenDAL's interface directly through the bindings supported by OpenDAL to access the storage services.

Opendal Is Not an SDK Aggregator​

Although OpenDAL replaces various SDKs in the application architecture, it is not implemented as an SDK aggregator.

In other words, OpenDAL does not simply call various storage service SDKs. We have developed our own docking with various storage services based on a unified Rust core to ensure that the differences between services are smoothed out.

For example, for S3, OpenDAL manually constructs HTTP requests and parses HTTP responses to ensure that all behaviors comply with API specifications and are fully under the control of OpenDAL. Due to OpenDAL's native takeover of the data access process, we can easily implement unified retry and logging mechanisms for various storage backends and ensure behavioral consistency.

For compatible services with S3, due to the limitations of native storage services and differences in API implementation, compatibility and behavioral details may differ from S3. For example, OSS needs to set an independent header to ensure consistent behavior for Range. In addition to docking with native storage services, OpenDAL will also perform targeted processing for compatible services to ensure users' data access experience.

Advantages of OpenDAL​

OpenDAL is not the only project dedicated to providing data access abstraction, but compared to other similar projects, OpenDAL has the following advantages:

Rich Service Support​

OpenDAL supports dozens of storage services, covering a wide range of scenarios and supporting on-demand selection:

• Standard Storage Protocols: FTP, HTTP, SFTP, WebDAV, etc.
• Object Storage Services: azblob, gcs, obs, oss, s3, etc.
• File Storage Services: fs, azdls, hdfs, webhdfs, ipfs, etc.
• Consumer Cloud Storage Service: Google Drive, OneDrive, Dropbox, etc.
• Key-Value Storage Service: Memory, Redis, Rocksdb, etc.
• Cache Storage Service: Ghac, Memcached, etc.

Complete Cross-Language Bindings​

With Rust as the core, OpenDAL now provides binding support for multiple languages such as Python/Node.js/Java/C and is also actively developing bindings for other languages.

Cross-language bindings not only provide unified storage access abstractions for other languages but also follow naming conventions and development habits that are common in various languages as much as possible to pave the way for quick use.

Powerful Middleware Support​

OpenDAL offers native layer support, enabling users to implement middleware or intercept for all operations.

• Error Retry: OpenDAL supports fine-grained error retry capabilities. In addition to common request retries, it supports breakpoint resumable transmission without having to re-read the entire file.
• Observability: OpenDAL implements logging,tracing,and metrics support for all operations. Turning on middleware can directly obtain observability capabilities for storage.
• Concurrency control, flow control, fuzz testing, and more.

Easy to Use​

OpenDAL's API has been well designed and polished in actual use. The documentation covers everything and is easy to get started with. Here's an example of using Python bindings to access HDFS:

import opendal
    
op = opendal.Operator("hdfs", name_node="hdfs://192.16.8.10.103")
op.read("path/to/file")

Use Cases of OpenDAL​

Currently, OpenDAL is widely used in various scenarios that require cloud-native capabilities, including but not limited to databases, data pipelines, and caches. The main user cases include:

• Databend: A modern Elasticity and Performance cloud data warehouse. Using OpenDAL to read and write persistent data (s3, azblob, gcs, hdfs, etc.) and cache data (fs, redis, rocksdb, moka, etc.).
• GreptimeDB: An open-source, cloud-native, distributed time-series database. Using OpenDAL to read and write persistent data (s3, azblob, etc.).
• mozilla/sccache: sccache is ccache with cloud storage. Using OpenDAL to read and write cache data (s3 and ghac, etc.).
• RisingWave: A Distributed SQL Database for Stream Processing. Using OpenDAL to read and write persistent data (s3, azblob, hdfs, etc.).
• Vector: A high-performance observability data pipeline. Using OpenDAL to write persistent data (currently mainly using hdfs).

Future Plans of OpenDAL​

In addition to further meeting the needs of cloud-native data access, OpenDAL will continue to expand user scenarios and actively explore its use in data science and mobile applications. At the same time, OpenDAL will continue to polish its existing implementations and bindings to provide users with a better integration experience.

OpenDAL will also explore how to improve users' workflows in data management and service integration:

• Polish the oli command-line tool to help users manage data painlessly.
• Implement the oay proxy service to provide users with high-quality compatible APIs.

In addition, since OpenDAL is currently a cross-language project, we also plan to write a series of introductory tutorials to help everyone learn OpenDAL from scratch while learning the language.

On February 27th, 2023, the OpenDAL project achieved a milestone by winning the approval to join the incubator of the Apache Software Foundation (ASF), the world's leading open source software organization. On March 15th, the OpenDAL project was officially transferred to the Apache Software Foundation.

This is a significant moment for Databend, as it means that OpenDAL's technology and vision have received wider recognition and support from the open source community.

The Apache Incubator was established in October 2002 to provide a path for projects and codebases that aspire to become part of the Apache Software Foundation. Incubating projects need to follow ASF's governance and operational practices, and use ASF's infrastructure and resources. Incubating projects need to go through a series of stages and evaluations before they can graduate to become top-level projects (TLPs) of ASF.

https://incubator.apache.org/projects/opendal.html

What is OpenDAL?​

Data is one of the most important assets in the future, and data access is the key for realizing data value.

There are various kinds of storage services in the market, each with its own unique interfaces and features, which bring a lot of complexity and inconvenience to data access.

OpenDAL provides a unified, simple, efficient, reliable, and observable data access layer that allows developers to seamlessly use different storage services and enjoy the best user experience.

OpenDAL simplifies the process of interfacing different storage services, and provides features such as automatic retry, request optimization, and observability. With OpenDAL, developers can directly access a bunch of storage services, without having to understand and master the details of specific SDKs.

OpenDAL's features include but are not limited to:

• Support for dozens of storage services, including local file system, HDFS, S3, OSS, etc.
• Provide a unified data access interface, without worrying about the underlying storage details.
• Support for various common data operations, including read, write, list, etc.
• Support for automatic retry, request optimization, and observability mechanisms.
• Zero cost, directly mapped to API calls.
• Cross-language bindings: Python, Node.js, C (being worked on), etc.

The Story about OpenDAL​

Born for Universal Data Access​

OpenDAL originated from the vision of creating a universal, unified and user-friendly data access layer. It came into being in late 2021, initially as a component of the Databend project.

• On December 21, 2021, Xuanwo embarked on the design and re-implementation of Databend's storage access layer, dal2: Add basic operations of read, write, stat, delete.
• On December 27, 2021, the proposal: Vision of Databend DAL was put forward and discussed. On December 29th, dal2's implementation was integrated into Databend.
• On February 14th 2022 , dal2 officially separated from Databend's code repository and became a standalone top-level project. It was formally renamed OpenDAL.

From One to Multiple​

Thanks to Xuanwo, ClSlaid and many other contributors, OpenDAL quickly became a data access layer that supports mainstream storage services such as AWS S3 / Azure Blob / GCS / HDFS, and provided cross-cloud native storage and access support for Databend's COPY INTO, Stage, Storage.

GreptimeDB was the first large-scale Rust database project to actively use OpenDAL after Databend. Later, with Xuanwo's efforts, sccache under Mozilla also tried to use OpenDAL to take over the storage layer. In order to provide more comprehensive support, OpenDAL soon added support for GitHub Action Cache.

Then, RisingWave and Vector were supported as well. The number of OpenDAL users started to grow. More and more users choose OpenDAL as their storage access layer.

Sky's the Limit​

OpenDAL has established a small community and formed a product matrix. In addition to the Rust - opendal, it also provides Python - opendal and Node.js - opendal bindings.

OpenDAL has released 99 versions since its open source, with 700+ GitHub stars, 349K downloads, and 48 developers. The project has been actively updated. We sincerely thank every contributor for their efforts and dedication.

Being a part of Apache incubator is an important milestone in OpenDAL's development history. We hope to leverage ASF's platform and resources to let OpenDAL focus on providing a neutral, vendor-free, painless, and efficient storage access layer, and better serve the developers.

We expect OpenDAL to be widely used in the following application scenarios:

• Data analysis: OpenDAL can help data analysts quickly read or write data from different storage services, and perform various format conversions and operations.
• Data science: OpenDAL can help data scientists easily get or save data from different storage services, and perform various preprocessing and postprocessing.
• Data engineering: OpenDAL can help data engineers efficiently build and manage data pipelines between different storage services, and perform various monitoring and tuning.

Acknowledgements​

From Xuanwo

Hello everyone, I'm Xuanwo, the Committer of Apache OpenDAL.

The OpenDAL project embodies my personal dream. Now it has entered the Apache incubator with the collaboration of the community. I feel very happy at this moment. Thank you all contributors for your contributions, thank Databend Labs for your support, thank Champion tison for your guidance, thank Mentors ningjiang, wusheng, tedliu and hexiaoqiao for your advice.

Let us follow the guidance of Apache Way to build a community together and create value for users by providing free, painless and efficient data access experience!

Join Apache OpenDAL Community​

We welcome developers and users who are interested in participating in OpenDAL project to join OpenDAL community and follow OpenDAL's latest news. You can get more information through the following ways:

• Visit OpenDAL official website: https://opendal.apache.org
• Explore OpenDAL GitHub repository: https://github.com/apache/opendal
• Join OpenDAL Discord channel: https://opendal.apache.org/discord
• Subscribe to OpenDAL mailing list: dev@opendal.apache.org