AWS_S3_ENDPOINT_URL
environment variable to point at your MinIO server.mnist/
we have our dataset, a collection of images. Let's track it with an artifact:max_objects=
in calls to add_reference
.mnist:latest
looks and acts just like a regular artifact. The only difference is that the artifact only consists of metadata about the S3/GCS object such as its ETag, size, and version ID (if object versioning is enabled on the bucket).AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN
GOOGLE_APPLICATION_CREDENTIALS
~/.aws/credentials
application_default_credentials.json
in ~/.config/gcloud/
~/.aws.config
/mount
with the following structure:mnist/
we have our dataset, a collection of images. Let's track it with an artifact:max_objects=
in calls to add_reference
.file://
prefix that denotes the use of filesystem references. The second is the path to our dataset, /mount/datasets/mnist/
.mnist:latest
looks and acts just like a regular artifact. The only difference is that the artifact only consists of metadata about the files, such as their sizes and MD5 checksums. The files themselves never leave your system.download()
operation copies the files from the referenced paths to construct the artifact directory. In the above example, the contents of /mount/datasets/mnist
will be copied into the directory artifacts/mnist:v0/
. If an artifact contains a reference to a fail that was overwritten, then download()
will throw an error as the artifact can no longer be reconstructed.