Hugging Face
Hugging Face services support.
This service will visit the Huggingface API to access the Huggingface File System.
Currently, we only support the model
and dataset
types of repositories, and operations are limited to reading and listing/stating.
Huggingface doesn't host official HTTP API docs. Detailed HTTP request API information can be found on the huggingface_hub
Source Code.
Capabilities
This service can be used to:
- stat
- read
- write
- create_dir
- delete
- copy
- rename
- list
-
scan -
presign - blocking
Configurations
repo_type
: The type of the repository.repo_id
: The id of the repository.revision
: The revision of the repository.root
: Set the work directory for backend.token
: The token for accessing the repository.
Refer to [HuggingfaceBuilder
]'s public API docs for more information.
Examples
Via Builder
use std::sync::Arc;
use anyhow::Result;
use opendal::services::Huggingface;
use opendal::Operator;
#[tokio::main]
async fn main() -> Result<()> {
// Create Huggingface backend builder
let mut builder = Huggingface::default();
// set the type of Huggingface repository
builder.repo_type("dataset");
// set the id of Huggingface repository
builder.repo_id("databricks/databricks-dolly-15k");
// set the revision of Huggingface repository
builder.revision("main");
// set the root for Huggingface, all operations will happen under this root
builder.root("/path/to/dir");
// set the token for accessing the repository
builder.token("access_token");
let op: Operator = Operator::new(builder)?.finish();
Ok(())
}
Via Config
- Rust
- Node.js
- Python
use anyhow::Result;
use opendal::Operator;
use opendal::Scheme;
use std::collections::HashMap;
#[tokio::main]
async fn main() -> Result<()> {
let mut map = HashMap::new();
map.insert("repo_type".to_string(), "dataset".to_string());
map.insert("repo_id".to_string(), "databricks/databricks-dolly-15k".to_string());
map.insert("revision".to_string(), "main".to_string());
map.insert("root".to_string(), "/path/to/dir".to_string());
map.insert("token".to_string(), "access_token".to_string());
let op: Operator = Operator::via_map(Scheme::Huggingface, map)?;
Ok(())
}
import { Operator } from "opendal";
async function main() {
// Use `huggingface` or `hf` for scheme name
const op = new Operator("huggingface", {
repo_type: "dataset",
repo_id: "databricks/databricks-dolly-15k",
revision: "main",
root: "/path/to/dir",
token: "access_token",
});
}
import opendal
# Use `huggingface` or `hf` for scheme name
op = opendal.Operator("huggingface",
repo_type="dataset",
repo_id="databricks/databricks-dolly-15k",
revision="main",
root: "/path/to/dir",
token: "access_token",
)