Merge pull request #31 from liulanzheng/main

update doc

Author: yuchen0cc

docs/README.md

@@ -4,7 +4,7 @@ OSS Connector for AI/ML contains some high-performance Python libraries specific
 
 Currently, the OSS connector is composed of two libraries: OSS Model Connector and OSS Torch Connector.
 
-- [OSS Torch Connector](https://aliyun.github.io/oss-connector-for-ai-ml/#/torchconnector/introduction) is dedicated to AI training scenarios, including loading [datasets](https://pytorch.org/docs/stable/data.html#dataset-types) from OSS and loading/saving checkpoints from/to OSS.
+- [OSS Torch Connector](https://aliyun.github.io/oss-connector-for-ai-ml/#/torchconnector/introduction) is dedicated to AI training scenarios, including loading [datasets](https://pytorch.org/docs/stable/data.html#dataset-types) from OSS and loading/saving checkpoints or [Distributed Checkpoints(DCP)](https://docs.pytorch.org/docs/stable/distributed.checkpoint.html) from/to OSS.
 
 - [OSS Model Connector](https://aliyun.github.io/oss-connector-for-ai-ml/#/modelconnector/introduction) focuses on AI inference scenarios, loading large model files from OSS into local AI inference frameworks.
 

docs/torchconnector/examples.md

@@ -273,4 +273,44 @@ with checkpoint.writer(CHECKPOINT_WRITE_URI) as writer:
    torch.save(state_dict, writer)
 ```
 
-OssCheckpoint can be used for checkpoints, and also for high-speed uploading and downloading of arbitrary objects. In our testing environment, the download speed can exceed 15GB/s.
+OssCheckpoint can be used for checkpoints, and also for high-speed uploading and downloading of arbitrary objects. In our testing environment, the download speed can exceed 15GB/s.
+
+## Distributed checkpoints
+
+OSS connector for AI/ML supports [PyTorch distributed checkpoints(DCP)](https://docs.pytorch.org/docs/stable/distributed.checkpoint.html) since v1.2.0rc2.
+
+```py
+import torchvision
+import torch.distributed.checkpoint as DCP
+from osstorchconnector import OssFileSystem
+import torch
+
+ENDPOINT = "http://oss-cn-beijing-internal.aliyuncs.com"
+CONFIG_PATH = "/etc/oss-connector/config.json"
+CRED_PATH = "/root/.alibabacloud/credentials"
+OSS_URI = "oss://ossconnectorbucket/dcp-checkpoint-resnet18"
+
+model = torchvision.models.resnet18()
+
+# write to OSS
+fs = OssFileSystem(endpoint=ENDPOINT, cred_path=CRED_PATH, config_path=CONFIG_PATH)
+oss_storage_writer = fs.writer(OSS_URI)
+# DCP.save or DCP.async_save
+checkpoint_future = DCP.async_save(
+    state_dict=model.state_dict(),
+    storage_writer=oss_storage_writer,
+)
+checkpoint_future.result()
+
+
+# load from OSS
+loaded_state_dict = {
+    key: torch.zeros_like(value) for key, value in model.state_dict().items()
+}
+oss_storage_reader = fs.reader(OSS_URI)
+DCP.load(
+    loaded_state_dict,
+    storage_reader=oss_storage_reader,
+)
+
+```

docs/torchconnector/installation.md

@@ -4,8 +4,11 @@
 
 - OS: Linux x86-64
 - glibc: >= 2.17
-- Python: 3.8-3.12
-- PyTorch: >= 2.0
+- Python: 3.8 - 3.12
+- PyTorch:
+    - `>= 2.0` (v1.0.0rc1-v1.1.0)
+    - `>= 2.3` (since v1.2.0rc1)
+- Kernel module: userfaultfd (for checkpoint)
 
 ## Install
 
@@ -25,4 +28,8 @@ For example, download the `osstorchconnector/v1.1.0rc1` for Python 3.11 and inst
 wget https://github.com/aliyun/oss-connector-for-ai-ml/releases/download/osstorchconnector%2Fv1.1.0rc1/osstorchconnector-1.1.0rc1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
 
 pip install osstorchconnector-1.1.0rc1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-```
+```
+
+### Additional configuration for Docker
+
+To use checkpoint-related features within Docker, the container must be run with `--privilege`. This is due to our reliance on userfaultfd to accelerate the reading of checkpoints.

docs/torchconnector/introduction.md

@@ -4,7 +4,7 @@
 ## Overview
 
 OSS Torch Connector provides both [Map-style and Iterable-style datasets](https://pytorch.org/docs/stable/data.html#dataset-types) for loading datasets from OSS.
-And also provides a method for loading and saving checkpoints from and to OSS.
+And also provides a method for loading and saving checkpoints or [Distributed Checkpoints(DCP)](https://docs.pytorch.org/docs/stable/distributed.checkpoint.html) from and to OSS.
 
 The core part of is OSS Connector for AI/ML is implemented in C++ using [PhotonLibOS](https://github.com/alibaba/PhotonLibOS). This repository only contains the code of Python.