Available from: Resilio Active Everywhere 4.0
Available for: Azure Blob storage, AWS S3, S3 compatible storage, GCP
Metadata creation and updates
Metadata details
Limitations and peculiarities
Metadata Creation and Updates
Starting with 4.0, for any job that syncs data with object storage, file metadata is stored in the cloud storage by default for all new jobs (not for jobs that were created prior to the 4.0 upgrade).
When creating an object, the Agent stores metadata in a format standardized for each cloud vendor using a single PUT request. Metadata updates automatically when the associated file changes. Pre-seeded objects cannot store metadata unless initially created by the Agent.
Enabling and disabling metadata storage is done with a custom parameter fs_enable_meta:true/false in the Job profile.
Metadata Details
Below are outlined metadata examples for S3 storage with x-amz-meta- prefix.
For Azure, the metadata prefix is x-ms-.
Google Cloud Storage accepts x-amz-meta- and x-goog-meta-.
S3-compatible object stores (e.g., Wasabi) use the same prefix as S3.
Metadata prefix is often hidden in GUI apps, however it is always there and is required by the standard.
An object can store both Windows and Posix ACL fields at the same time.
x-amz-meta-mtime: file modification time, Unix timestamp. x-amz-meta-ctime: file creation time, Unix timestamp. x-amz-meta-mode: file mode (Posix man 2 stat, stat.st_mode), stores file type and Posix permissions bits x-amz-meta-rslhash: Resilio proprietary file hash x-amz-meta-usid: Windows Owner SID x-amz-meta-gsid: Windows Owner Group SID x-amz-meta-sddl: Windows permissions SDDL x-amz-meta-uid: Posix user id x-amz-meta-gid: Posix group id x-amz-meta-un: Posix user name x-amz-meta-gn: Posix group name
Example response to HEAD object synced by Resilio
Date: Sat, 07 Dec 2024 12:50:17 GMT x-amz-id-2: X2X3k2jpSPog2VvgX+a9aw2KU5y7IGdPzP4vTOKybI9Onf+AP5sXhtD9+E/YeEpPFlkDX4ZFn6A4WfN0bfVqZg== Accept-Ranges: bytes x-amz-request-id: S2HEQ0M27XVHFW6P x-amz-meta-gid: 1 Last-Modified: Sat, 07 Dec 2024 12:49:20 GMT ETag: "3289e7d91a537edd37c10b02ec7596e0" x-amz-meta-mode: 33152 x-amz-server-side-encryption: AES256 x-amz-meta-ctime: 1733575758 x-amz-meta-gn: jedi x-amz-meta-usid: O:S-1-12-1-2638222081-1256969586-465189761-1299699060 x-amz-meta-sddl: D:(A;;GA;;;SY)(A;;GRGX;;;BU)(D;;WD;;;AN) x-amz-meta-uid: 1 Content-Range: bytes 0-127/128 x-amz-meta-gsid: G:S-1-12-1-2638222081-1256969586-465189761-1299699060 x-amz-meta-mtime: 1733575758 x-amz-meta-rslhash: EBF49A1DF7BF8D4DD81C1B471466583BD8FE03E6-1 x-amz-meta-un: luke Content-Length: 128 Content-Type: application/octet-stream Server: AmazonS3
Limitations and peculiarities
Possible overhead: changing object metadata requires CopyObject request to be called and this request is suboptimal on some S3-like storages from our experience. When dealing with frequent changes in large file systems, performance testing is recommended.