Apache OpenDAL
Open Data Access Layer: Access data freely
Languages
Apache OpenDAL provides Rust Core and builds different language bindings like Node.js Binding and Python Binding.
- Rust
- Node.js
- Python
use opendal::Operator;
let op = Operator::via_map(Scheme::Fs, HashMap::new()?;
op.read("path/to/file").await?;
import { Operator } from "opendal";
const op = new Operator("fs", {});
await op.read("path/to/file")
import opendal
op = opendal.Operator("fs")
op.read("path/to/file")
Services
Apache OpenDAL provides native support for all kinds for storage systems.
Standard Storage Protocols
Object Storage Services
- azblob: Azure Storage Blob services
- gcs: Google Cloud Storage Service
- obs: Huawei Cloud Object Storage Service (OBS)
- oss: Aliyun Object Storage Service (OSS)
- s3: AWS S3 alike services
- supabase: Supabase Storage Service being worked on
- wasabi: Wasabi Cloud Storage
File Storage Services
- azdfs: Azure Data Lake Storage Gen2 services (As known as abfs)
- hdfs: Hadoop Distributed File System(HDFS)
- ipfs: InterPlanetary File System HTTP Gateway
- ipmfs: InterPlanetary File System MFS API being worked on
- webhdfs: WebHDFS Service
Consumer Cloud Storage Service
- gdrive: Google Drive being worked on
- onedrive: OneDrive being worked on
Key-Value Storage Service
Cache Storage Service
- ghac: Github Action Cache Service
- memcached: Memcached service
- moka: moka backend
- vercel_artifacts: Vercel Remote Caching Service being worked on
Layers
Apache OpenDAL offers native layer support, enabling users to implement middleware or intercept for all operations.
By using layers, we can:
RetryLayer
: Automatically retry failed requests and resume from the point of failure.ChaosLayer
: Generate simulated chaos for storage services.ConcurrentLimitLayer
: Set concurrent limit while accessing storage services.{Logging|Metrics|Tracing}Layer
: Provide native observability for storage services.