Resilio Agents keep an sqlite database file for each of the jobs they do. For Distribution and Consolidation jobs the database files are deleted from disk once the job run completes while Script job does not have the database at all. Database files for Synchronization jobs are kept on disk all the time while the Agent is in the job.
Agent keeps records of each file that is present in the job and, which is important, has ever been present in the job, as well as records about archived, ignored files and their attributes.
If an Agent is removed from the job, the database file is deleted. Adding the Agent back to the job means that it will have to build the new database from scratch - index the files on its storage, merge the folder tree with that on other Agents. This is a long lasting process and on millions of files may take up to several days.
That is why it's highly advisable to periodically backup Agents' database for jobs with 1M+ files, in case of unexpected necessity to re-add the Agent to the job, so as to have all the indices already built up. This will allow to get the job back in balance way faster. For example, having a backup of database will help in the following cases:
- database error occurred. There might be multiple causes of it, see here for more details.
- system crash or permanent storage device damage. Most likely this will lead to the database corruption.
- Agent is overwhelmed with load, for example, volume of synchronization tasks to perform it way above hardware capacity. Usually support team engineers may advise to re-add the Agent to the job after investigation.
- human mistake.
How to know which database files to backup
Databases are stored in the Agents' storage folder:
Windows: C:\ProgramData\Resilio\Connect Agent
Mac: /Users/username/Library/Application Support/Resilio Connect Agent
Linux:
- Installed as an app: ./sync in the current directory, from which you launch the agent.
- Installed from package as service: /var/lib/resilio-agent or the path where --log argument in process points.
- Installed from package and running with current user: /home/username/.config/resilio-agent or path where --log argument in process points.
See the linked guide for other platforms.
There are two ways to decide what database to backup:
- by file size. Each file in the database takes around 2kb (4 KB if synchronization of file permissions is enabled) per each file and subdirectory in the job. Mind that files which were present in the job previously also add to the size. Knowing the numbers of files in the job will help to reveal the necessary database file.
- by file name. Database files are named after job internal identifier. You can find internal identifier in dump debug status for any Agent participating in job.
Databases with extensions .db, sf.db and files.db are of interest. It's necessary to backup all such files belonging to the job, i.e. have the same name.
How to backup the database files
Unfortunately, the Agents don't have this functionality out of the box and it has to be done manually or be automated with scripts. Basically the procedure boils down to the steps:
- stop the Resilio Agent. At this point you will notice that database shared memory storage and journal files (.shm and .wal respectively) will be removed, no need to backup these.
- copy the database file(s) to a safe place. You can arrange the way to store them as you need. Be sure to group all database files from a single backup somehow so as to be able to restore them back if necessary.
- start the Agent.
It's advisable to perform backup during off-load hours. There's also no real need to keep more than two backups, three backups to be on a very safe side.
Backing up databases for an active Agent
If Agent restart takes too long, you can backup databases without shutting the Agent down. Follow the suggestions below:
- do it during off-load hours so DB gets as few changes at the moment as it is possible
- also copy *.shm and *.wal files
- once done, run the integrity check for every .db file backed up using sqlite3 CLI tool:sqlite3 <db_filename> 'pragma integrity_check'
- if the integrity check fails - backup the DB again
How to restore backup
The procedure is the opposite to the doing backup:
- stop the Agent, this is a compulsory step.
- copy the latest backup - all the backed up database files - to the Agent's storage.
- start the Agent.
It's normal that the Agent might be still missing some of the latest file updates, records of which got lost due to the initial reason why you needed to restore the backup.
Give the Agent some time to load and get the job in balance with other Agents.