What Are Egress Costs?
Egress costs refer to charges incurred when data is moved out of a cloud provider’s infrastructure, such as downloading objects from storage, transferring data between regions or to on-premise systems, accessing archived data, etc. These charges can quickly accumulate and lead to unexpected expenses if not carefully monitored.
Key Considerations
Data transfer and access costs may increase in Resilio Jobs due to the following factors.
Enumerating and downloading objects: APIs that list objects or retrieve them often result in charges, particularly in high volumes. Resilio Agent performs folder scan by listing the objects. The more frequent rescan interval is, the more listing API calls are made.
Region-to-Region data transfer: Moving data between geographically distant regions incurs additional fees.
Accessing Archived Data: Retrieval costs for cold storage (e.g., AWS Glacier) are higher due to its low-cost, high-latency nature.
Egress rate variations
Pricing models vary between AWS, Azure, Oracle and other storage providers supported by Resilio Active Everywhere. Some include free tiers for outbound data transfer, while others charge for even the smallest operations.
The Resilio Agent interacts with cloud storage systems to perform essential operations necessary for synchronization, seeding, and metadata management. These operations can generate various requests, and the number of requests depends significantly on the structure and organization of the data.
Listing Objects
The Agent uses LIST requests to enumerate objects within directories. This operation occurs during periodic rescans or when checking the state of the storage. Note: the Agent retrieves information for thousands of objects within a single subdirectory using one LIST request, minimizing the number of API calls. If a directory contains only one object, the Agent performs a single LIST request specific to that directory.
Reading Objects
The Agent issues READ requests to access objects stored in the cloud. In S3-compatible storage, this occurs during specific operations such as seeding to the cloud (when uploading existing data for synchronization) and/or hashing.
Writing Objects
The Agent writes objects to storage using PUT requests. This operation is employed for initial data uploads, updates, or synchronization of modified files.
Copying Objects
The Agent can duplicate objects within storage using COPY requests. Common scenarios include archiving (moving objects to the Resilio Archive for backup or versioning purposes) or metadata management (writing or updating metadata associated with objects). In Azure Storage, metadata updates use a Set Metadata operation, which is more cost-efficient than the equivalent operation in AWS. Unlike AWS, where updating metadata may involve rewriting the entire object, Azure’s approach avoids this overhead, reducing both cost and latency.
By understanding these operations, administrators can better estimate the resource consumption and optimize the performance of Resilio Agent within their cloud storage environments.
Below is approximate possible cost calculations of a periodic scan in a cloud storage.
1) Dataset: 1 million files total, 20 files per folder, folder depth 12, and total size 20 GB. This gave the following metrics per one rescan
- 28909690 bytes (~2.9- MB) and 112733 requests, in LIST request
- 1344527175 bytes and 6960782 requests to write metadata.
2) Dataset: 10 million files total, 10k files per folder, folder depth 2-3, total size 20 GB. This gave the following metrics per one rescan
- 1422530 bytes (~1,4 MB) and 5336 requests, in LIST request;
- 3862391 bytes and 65468588 requests to write metadata.
Refer to the calculators below to get the average total costs:
-
AWS (Amazon Web Services)
AWS Create Estimate
AWS Cost Management -
Microsoft Azure
Azure Pricing Calculator
Azure Cost Management
Best Practices
Monitor Usage Regularly. Use monitoring tools to track egress activities and associated costs. Cloud providers often provide built-in dashboards.
Minimize Region Transfers. Place resources in the same region whenever possible.
Use Compression: Reduce data size before transferring. Enable data compression in Resilio (note, with enabled compression performance is expected to decrease).
Leverage Free Tiers. Understand what data transfers are included for free.
Evaluate Storage Classes. Archived storage can save money on storage fees but may result in higher retrieval costs. Evaluate access patterns to choose the right storage class.
Set Alerts and Budgets. Use cost alerts to get notified when egress charges reach a threshold.
Minimize objects listing. Configure less frequent folder rescan in Resilio.