Command line interface

Console's command-line interface is a python wrapper around Resilio Console's API. It lets your copy data from an agent or a group to other agents or groups. Commands can be executed on any machine with network access to Management Console. 

***
1) Preparing
2) Installing and updating
3) General usage info
4) Copy command
5) Masscopy command
6) Masscopy from a JSON config file
7) Managing output 
8) Important notes
***

Preparing

Preparation implies enabling Console API and installing python & pip. 

You need to have Resilio Connect license with Console API feature and API authentication token generated: 
- check in Settings -> Licensing that you have "Connect Console API" 
- go to Settings -> API and generate the token, or use an already created one. 

Now you need to have python 2.7 or newer and pip packet manager installed. Python3 is more preferable.

sudo apt install python3 python3-pip

To install python and pip on Windows use this nice guide. 

Installing and updating

python3 -m pip install --extra-index-url https://pypi.resilio.com/ connect-cli==0.1.8

python3 -m pip install setuptools
wget https://pypi.resilio.com/connect-api/connect-api-0.0.31.tar.gz
wget https://pypi.resilio.com/connect-cli/connect-cli-0.1.8.tar.gz
tar xf connect-api-0.0.31.tar.gz
tar xf connect-cli-0.1.8.tar.gz
cd connect-api-0.0.31
python3 setup.py install
cd ../connect-cli-0.1.8
python3 setup.py install
cd ..

python -m pip install --extra-index-url https://pypi.resilio.com/ connect-cli==0.1.8

python -m pip install setuptools
wget https://pypi.resilio.com/connect-api/connect-api-0.0.31.tar.gz | Out-File connect-api-0.0.31.tar.gz
wget https://pypi.resilio.com/connect-cli/connect-cli-0.1.8.tar.gz | Out-File connect-cli-0.1.8.tar.gz
tar xf connect-api-0.0.31.tar.gz
tar xf connect-cli-0.1.8.tar.gz
cd connect-api-0.0.31
python setup.py install
cd ../connect-cli-0.1.8
python setup.py install
cd ..

General usage info

The first command to connect to the Management Console is

connect configure

CLI will require entering Management Consoles' address, API token and timeout. Address is to be entered in https://IP:port format, where IP - is Console's local or public address and port - is Console's WebUI listening port. 
API token is the token created in the previous step. 
Timeout is necessary to abort the data transfer if it takes longer than configured. Default it 10, so just press Enter to preserve it or set your own timeout. The 'no timeout at all' mode is not implemented.

To view the Agents and agent groups on the MC run command

connect ls

The output shows two tables - agents and groups. Each table has columns with identifiers - these can be later used to execute copy and masscopy commands. The IP address shows the address of an Agent as seen on the Management Console.

Copy

The basic syntax for copy command is 

 connect copy {source} [destination1] [destination2] [destinationN]

Each of the arguments - source and destinations - consists of two parts separated by a colon: the identifier and path. For example, by agent: device_id:path, agent_name:path, ip:path, agent_id:path or by groups:group_name:path, group_id:path.
Identifier is the agent or group ID, name or agent IP address that can be learned from the listing command.  If the command is executed on the machine with an agent installed, the first identifier part can be omitted for the local agent*.
Path - is the full path to the selected directory. Relative paths are also supported and they'll be relative to the "folders_storage_path" from agent's config file. On the local agent, these will be relative to the current directory. Source path must exist before starting data transfer. 

Basically, a copy command run is a new job run with default profile settings.

For example, the command below copies files from local* /home/admin/logs directory to agents from g17 and g16, given directories:

connect copy /home/admin/logs g17:/home/username/docs g16:F:\Documents

Another example of a command that copies files from agent named S2016 to a33 agent and a35, given directories (mind the escaped backslash for Windows paths):

connect copy S2016:F:\\Documents a33:C:\\Important\\docs a35:data

Use this command to copy files from remote agent to agents from group15 and the local* Agent's /home/admin/updates directory:

connect copy CD3G6DFBJOG4ILIETH6VV53X6RS2AMCT:updates g15:updates /home/admin/updates

The successful copy command shows the following output 

[ INFO ] Start indexing...
[ INFO ] Indexing completed. Indexed: 13.0B
[ INFO ] Job size: %size, %files 
[ INFO ] Started transfer
[ INFO ] Transfer completed
[ INFO ] Average transfer speed: %speed. Transfer duration: %sec

Masscopy

Masscopy functionality allows to synchronize the files located all across different directories by creating and following reference symbolic links to the files.

First, it's advisable to enable debug mode to add more verbosity to the command line

export DEBUG=1

Configure CLI

connect configure

Add required files into masscopy index. Absolute paths must be used

connect masscopy-add full_path_1
connect masscopy-add full_path_2
......
connect masscopy-add full_path_N

CLI will calculate common path prefix for added files automatically. Prepare added files to be transferred

connect masscopy-prepare

To view calculated common path prefix run the command

connect masscopy-prepare --show-common-path-prefix

To view index file run the command

connect masscopy-prepare --show-index

CLI will create symlinks to added files in masscopy temporary directory that was set on configuration step. Start transfer (see above for specifying target directories) from the local agent to a destination

connect masscopy-start local {dst}:{dst_path}

In case you need to preserve and later review the masscopy run information, launch it with --no-cleanup and - -no-cleanup-masscopy parameters:

connect masscopy-start local {dst}:{dst_path} --no-cleanup --no-cleanup-masscopy

Masscopy from a JSON config file

Starting with CLI version 0.1.8 masscopy functionality allows to build the data index from a pre-configured json file. It  greatly saves performance on large dataset. Use this flow instead of masscopy-add. 

In advance create a json file with the following structure

{
    "destination": "<dst>:<dst_path>",
    "common_prefix": "/tmp/files",
    "files": [
        "/tmp/files/file1.txt",
        "/tmp/files/nested/file2.txt"
    ]
}

<dst> learn from connect ls command, can be agent or a group, by ID, device name or device ID. Only one dst is allowed. 
<dst_path> - the full target path on the destination.
common_prefix - is the source directories common path prefix.
files - an array of full paths to the files. 

Once the json file is ready, launch masscopy command pointing to the file's location. The example below also shows two possible, but not obligatory parameters

connect masscopy --json /path/to/json/config.json --check-common-path-prefix --no-create-symlinks

--check-common-path-prefix - checks that the common path prefix in json is correct. If it's not, it will be automatically replaced with the correct one. It will take time on large dataset, so it's better to be avoided.
--no-create-symlinks - add this parameter if json file already contains symlinks so that masscopy does not create them 

In case you need to preserve and later review the masscopy run information, launch it with --no-cleanup and --no-cleanup-masscopy parameters:

connect masscopy-start local {dst}:{dst_path} --no-cleanup --no-cleanup-masscopy

Managing output

By default, the script writes its output into stdout. Three levels of records are available: info, warning and error. 

It's possible to redirect the error into a separate log file. Append >out.log 2>err.log to your command (mind the escaped backslash for Windows paths):

connect copy /home/admin/logs g17:/home/username/docs g16:F:\\Documents >out.log 2>err.log

The result will be 

Important notes:

1) Only one source is allowed. If you use a group as source, only one agent shall be in that group.

2) While copy is being performed, you can see "Copy %foldername%" job and groups - source and destination - with the same name created. These will be removed once the copying's done.

2.1) Agents record and store events about a copy operation in its storage folder in events subdirectory. 

3) When setting path in copy command, the source directory shall already exist on the computer. On destinations it's not compulsory: if the folder already exists, it will be used, otherwise, a new one be created. 

4) Copying single file is not supported. Only directories with their content can be copied.

* 5) understanding local agent. The agent that runs on the same computer where CLI commands are executed is considered local. In listing it's marked with an asterisk symbol in column Local. For the local agent to be detected, it must listen to API calls: enable Agent API (port 3840) in Profile for this agent.
In copy command you can omit the identifier part for it.