File
File is a special DataModel,
which is automatically generated when a DataChain is created from files,
such as in dc.read_storage:
import datachain as dc
chain = dc.read_storage("gs://datachain-demo/dogs-and-cats")
print(chain.schema)
Output:
file: File@v1
source: str
path: str
size: int
version: str
etag: str
is_latest: bool
last_modified: datetime
location: Union[dict, list[dict], NoneType]
File classes include various metadata fields describing the underlying file,
along with methods to read and manipulate file contents.
File
Bases: DataModel
DataModel for reading binary files.
Attributes:
-
source(str) βThe source of the file (e.g., 's3://bucket-name/').
-
path(str) βThe path to the file (e.g., 'path/to/file.txt').
-
size(int) βThe size of the file in bytes. Defaults to 0.
-
version(str) βThe version of the file. Defaults to an empty string.
-
etag(str) βThe ETag of the file. Defaults to an empty string.
-
is_latest(bool) βWhether the file is the latest version. Defaults to
True. -
last_modified(datetime) βThe last modified timestamp of the file. Defaults to Unix epoch (
1970-01-01T00:00:00). -
location(dict | list[dict]) βThe location of the file. Defaults to
None.
Source code in datachain/lib/file.py
name
property
name: str
parent
property
parent: str
as_audio_file
as_audio_file() -> AudioFile
Convert the file to a AudioFile object.
Source code in datachain/lib/file.py
as_image_file
as_image_file() -> ImageFile
Convert the file to a ImageFile object.
Source code in datachain/lib/file.py
as_text_file
as_text_file() -> TextFile
Convert the file to a TextFile object.
Source code in datachain/lib/file.py
as_video_file
as_video_file() -> VideoFile
Convert the file to a VideoFile object.
Source code in datachain/lib/file.py
at
classmethod
Construct a File from a full URI in one call.
Parameters:
-
uri(str | PathLike[str]) βFull URI or path to the file (e.g.
s3://bucket/path/to/file.pngor/local/path). -
session(Session | None, default:None) βOptional session instance. If None, the current session is used.
Returns:
-
File(Self) βA new File object pointing to the given URI.
Source code in datachain/lib/file.py
ensure_cached
Download the file to the local cache.
get_local_path can be used to return the path to the cached copy on disk.
This is useful when you need to pass the file to code that expects a local
filesystem path (e.g. ffmpeg, opencv, pandas, etc).
Source code in datachain/lib/file.py
export
export(
output: str | PathLike[str],
placement: ExportPlacement = "fullpath",
use_cache: bool = True,
link_type: Literal["copy", "symlink"] = "copy",
client_config: dict | None = None,
*,
content_type: str | None = None,
content_disposition: str | None = None,
cache_control: str | None = None,
content_encoding: str | None = None,
metadata: dict[str, str] | None = None,
write_options: dict[str, Any] | None = None
) -> None
Copy or link this file into an output directory.
Parameters:
-
output(str | PathLike[str]) βDestination directory. Accepts a local OS path, a cloud prefix fsspec URI (
s3://β¦,gs://β¦,az://β¦). -
placement(ExportPlacement, default:'fullpath') βHow to build the path under output:
"fullpath"(default) βoutput/bucket/dir/file.txt"filepath"βoutput/dir/file.txt"filename"βoutput/file.txt"etag"βoutput/<etag>.txt
-
use_cache(bool, default:True) βIf True, download to local cache first. Also required for symlinking remote files.
-
link_type(Literal['copy', 'symlink'], default:'copy') β"copy"(default) or"symlink". Symlink falls back to copy for virtual files and for remote files when use_cache is False. -
client_config(dict | None, default:None) βExtra kwargs forwarded to the storage client.
-
content_type(str | None, default:None) βContent-Typeto set on the exported object. -
content_disposition(str | None, default:None) βContent-Dispositionto set on the object. -
cache_control(str | None, default:None) βCache-Controlto set on the object. -
content_encoding(str | None, default:None) βContent-Encodingto set on the object. -
metadata(dict[str, str] | None, default:None) βCustom key/value metadata to attach to the object.
-
write_options(dict[str, Any] | None, default:None) βRaw, backend-native write kwargs merged after the normalized fields above. Object metadata is applied only to copied (not symlinked) files and is ignored on the local filesystem.
Example
# flat export by filename
f.export("./export", placement="filename")
# export to a cloud prefix
f.export("s3://output-bucket/results", placement="filepath")
# pass storage credentials via client_config
f.export("s3://bucket/out", client_config={"aws_access_key_id": "β¦"})
# symlink from local cache (avoids re-downloading)
f.export("./local_out", use_cache=True, link_type="symlink")
Source code in datachain/lib/file.py
867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 | |
get_fs_path
get_fs_path() -> str
Combine source and path into the full location string.
For cloud backends the result is a full URI-like (s3://β¦, gs://β¦).
For local files the result is a bare OS path (no file://
prefix).
Examples:
# Cloud (S3, GCS, Azure, β¦)
f = File(source="s3://my-bucket", path="data/image.jpg")
f.get_fs_path() # 's3://my-bucket/data/image.jpg'
# Local files
f = File(source="file:///home/user/project", path="out/result.csv")
f.get_fs_path() # '/home/user/project/out/result.csv'
# No source β returns the relative path as-is
f = File(source="", path="dir/file.txt")
f.get_fs_path() # 'dir/file.txt'
Raises:
-
FileErrorβIf
pathis empty, ends with/, or contains./..segments. For local files, also rejects absolute paths, drive letters, and empty segments.
Source code in datachain/lib/file.py
get_local_path
get_local_path() -> str | None
Return path to a file in a local cache.
Returns None if file is not cached. Raises an exception if cache is not setup.
Source code in datachain/lib/file.py
open
open(
mode: str = "rb",
*,
client_config: dict[str, Any] | None = None,
content_type: str | None = None,
content_disposition: str | None = None,
cache_control: str | None = None,
content_encoding: str | None = None,
metadata: dict[str, str] | None = None,
write_options: dict[str, Any] | None = None,
block_size: int | None = None,
**open_kwargs
) -> Iterator[Any]
Open the file and return a file-like object.
Supports both read ("rb", "r") and write modes (e.g. "wb", "w", "ab"). When opened in a write mode, metadata is refreshed after closing.
Reads always consult the cache (hit β local, miss β remote stream);
_caching_enabled only governs whether a miss is also persisted.
In write modes, content_type, content_disposition,
cache_control, content_encoding, metadata and the raw
write_options escape hatch set object metadata on the written
object (mapped per backend, ignored on the local filesystem).
block_size sets the read-ahead block for uncached remote reads (0
turns read-ahead off), and the upload part size for writes. HTTP(S)
reads ignore it.
Source code in datachain/lib/file.py
556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 | |
read
read(length: int = -1)
read_bytes
read_bytes(length: int = -1)
Returns file contents as bytes.
Source code in datachain/lib/file.py
read_text
Return file contents decoded as text.
**open_kwargs : Any
Extra keyword arguments forwarded to open(mode="r", ...)
(e.g. encoding="utf-8", errors="ignore")
Source code in datachain/lib/file.py
rebase
Rebase the file's URI from one base directory to another.
Parameters:
-
old_base(str) βBase directory to remove from the file's URI
-
new_base(str) βNew base directory to prepend
-
suffix(str, default:'') βOptional suffix to add before file extension
-
extension(str, default:'') βOptional new file extension (without dot)
Returns:
-
str(str) βRebased URI with new base directory
Raises:
-
ValueErrorβIf old_base is not found in the file's URI
Examples:
file = File(source="s3://bucket", path="data/2025-05-27/file.wav")
file.rebase("s3://bucket/data", "s3://output-bucket/processed",
extension="mp3")
# 's3://output-bucket/processed/2025-05-27/file.mp3'
file.rebase("data/audio", "/local/output", suffix="_ch1",
extension="npy")
# '/local/output/file_ch1.npy'
Source code in datachain/lib/file.py
resolve
resolve() -> Self
Resolve a File object by checking its existence and updating its metadata.
Returns:
-
File(Self) βThe resolved File object with updated metadata.
Source code in datachain/lib/file.py
save
save(
destination: str,
client_config: dict | None = None,
*,
content_type: str | None = None,
content_disposition: str | None = None,
cache_control: str | None = None,
content_encoding: str | None = None,
metadata: dict[str, str] | None = None,
write_options: dict[str, Any] | None = None
) -> File
Write file contents to destination.
Parameters:
-
destination(str) βTarget path or URI. Accepts a local OS path, a cloud URI (
s3://β¦,gs://β¦,az://β¦), or an unencodedfile://URI as produced by :func:~datachain.fs.utils.path_to_fsspec_uri. Do not passPath.as_uri()output β that produces RFC percent-encoded URIs (e.g.file:///my%20dir) which fsspec does not decode, causingFileNotFoundErrorfor paths containing spaces,#, or%. -
client_config(dict | None, default:None) βOptional extra kwargs forwarded to the storage client (e.g. credentials, endpoint URL).
-
content_type(str | None, default:None) βContent-Typeto set on the written object. -
content_disposition(str | None, default:None) βContent-Dispositionto set on the object. -
cache_control(str | None, default:None) βCache-Controlto set on the object. -
content_encoding(str | None, default:None) βContent-Encodingto set on the object. -
metadata(dict[str, str] | None, default:None) βCustom key/value metadata to attach to the object.
-
write_options(dict[str, Any] | None, default:None) βRaw, backend-native write kwargs merged after the normalized fields; forwarded on S3 only (e.g.
{"ACL": "public-read"}) and rejected on GCS/Azure.
The normalized metadata fields are mapped to each backend's native parameters and ignored by the local filesystem.
Returns:
-
FileβA
Filerepresenting the newly written destination, with -
Fileβversionpopulated on versioned backends.
Example
Source code in datachain/lib/file.py
729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 | |
upload
classmethod
upload(
data: bytes,
path: str | PathLike[str],
catalog: Catalog | None = None,
*,
content_type: str | None = None,
content_disposition: str | None = None,
cache_control: str | None = None,
content_encoding: str | None = None,
metadata: dict[str, str] | None = None,
write_options: dict[str, Any] | None = None
) -> Self
Upload bytes to a storage path and return a File pointing to it.
Parameters:
-
data(bytes) βThe raw bytes to upload.
-
path(str | PathLike[str]) βDestination path (local or remote, e.g.
s3://bucket/file.txt). -
catalog(Catalog | None, default:None) βOptional catalog instance. If None, the current session catalog is used.
-
content_type(str | None, default:None) βContent-Typeto set on the object. -
content_disposition(str | None, default:None) βContent-Dispositionto set on the object. -
cache_control(str | None, default:None) βCache-Controlto set on the object. -
content_encoding(str | None, default:None) βContent-Encodingto set on the object. -
metadata(dict[str, str] | None, default:None) βCustom key/value metadata to attach to the object.
-
write_options(dict[str, Any] | None, default:None) βRaw, backend-native write kwargs merged after the normalized fields; forwarded on S3 only (e.g.
{"ACL": "public-read"}) and rejected on GCS/Azure.
The normalized metadata fields are mapped to each backend's native parameters and ignored by the local filesystem.
Returns:
-
File(Self) βA new File object with metadata populated from the upload.
Example
Note
To write data as a stream, use File.at with open instead:
Source code in datachain/lib/file.py
410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 | |
FileError
TarVFile
Bases: VFile
Virtual file model for files extracted from tar archives.
open
classmethod
Stream file from tar archive based on location in archive.
Reads always consult the cache (hit β local, miss β remote stream);
_caching_enabled only governs whether a miss is also persisted.
Read-ahead defaults to 5 MiB, capped at the member size.
Explicit block_size values are forwarded unchanged.
HTTP(S) clients ignore this setting.