Installation¶

Contents

Installing from source
Docker container

Installing from source ¶

First, get the latest NGAS sources:

git clone https://github.com/ICRAR/ngas

Installing NGAS from source is pretty straight-forward. There are two ways to perform an installation:

Manually.
Using Fabric. This is the recommended way, if possible, as it automates most of the installation steps while still being highly customisable.

Manual installation ¶

Note

Like any other python package, NGAS can be installed in a virtual environment or as a system-wide package.

To manually install NGAS go to the root directory and simply run:

./build.sh

Run ./build.sh -h to see the full set of options. Among the options users can choose whether to skip the build of the C-written clients, to install the python packages in development mode, and to build NGAS without CRC32c support (see CRC for details).

The script will (optionally) build and install the C NGAS client first, and then will build and install each of the python modules (i.e., ngamsCore, ngamsPClient, ngamsServer and ngamsPlugIns). The python modules will automatically pull and install their dependencies.

Note

If any step of the build fails the script will stop and notify the user. It is the user’s responsibility to install any missing build dependencies. To avoid these issues you can also try the Fabric installation

In addition, the build.sh script recognizes when a virtual environment is loaded, and will install the C client on it, as well as the python modules.

The C client compilation is based on autotools, meaning that it can also be manually compiled and installed easily via the usual:

$> ./bootstrap
$> ./configure
$> make all
$> make install

The Python modules are all setuptool-based packages, meaning that they can also be manually compiled and installed easily via the usual:

$> python setup.py install

Via Fabric ¶

Note

The installation via Fabric always installs NGAS in a virtualenv

Fabric is a tool that allows to perform commands in one or more hosts, local or remote (via SSH). NGAS comes with a set of fabric modules to ease the installation of NGAS in complex scenarios, and to automate most of the system-level preparation tasks. This enables not only a simple procedure for installing NGAS in any host or hosts at a given time, but also the customization of the hosts as necessary, plus any other extra step required by other scenarios.

Fabric’s command-line allows users to specify the username and hosts where tasks will take place, and a set of variables to be defined. For example:

fab -H host.company.com -u user some_task --set VAR1=a,VAR2

In the example the instructions of the task some_task will be carried out in host host.company.com with the user user, and the VAR1 variable will be set to the value a, while variable VAR2 will be marked as set.

For a complete list of tasks run fab -l. For a detailed description of a task run fab -d <task>. For a more complete manual visit Fabric’s documentation.

The two main fabric tasks NGAS provides are:

User installation: This task compiles and installs NGAS under a user-owned directory.
System installation: The recommended task if you have sudo access to the target machine. This task installs all the necessary dependencies in the system before compiling and installing NGAS.

Basic per-user installation ¶

To compile and install NGAS in a user-owned directory run:

fab hl.user_deploy

This task will:

Check that SSH is working on the target host
Copy the NGAS sources to the target host
Compile and install NGAS into a virtualenv on the target host
Create a minimal, working NGAS root directory
Finally, modify the corresponding ~/.bash_profile file to automatically load the NGAS virtualenv when the user enters a bash shell.

The user on the target host used for running these tasks is the SSH user given to fabric via the command line (fab -u <user>).

This task doesn’t take care of installing any dependencies needed by NGAS, assuming they all are met. For a more complete automatic procedure that takes care of that see the hl.operations_deploy task.

The following fabric variables (set via the --set command-line switch) are available to further customize the process:

Variable	Description	Default value
NGAS_SRC_DIR	The directory where the NGAS sources will be extracted on the target host	`~/ngas_src`
NGAS_INSTALL_DIR	The directory where the virtualenv will be created and NGAS installed	`~/ngas_rt`
NGAS_ROOT_DIR	The NGAS root directory created by default by the installation procedure	`~/NGAS`
NGAS_REV	The git revision of the sources used to compile and install NGAS (only for sources from a git repository). Keep in mind that after cloning, you might have a reference to remote branches without having local branches with the corresponding names, so using `NGAS_REV=v10` might not work, but `NGAS_REV=origin/v10` will.	`HEAD`
NGAS_OVERWRITE_INSTALLATION	If specified, an existing installation directory will be overwritten	Not specified
NGAS_OVERWRITE_ROOT	If specified, an existing NGAS root directory will be overwritten	Not specified
NGAS_USE_CUSTOM_PIP_CERT	If specified, configure pip to use curl.haxx.se/ca/cacert.pem as the root TLS certificate. In some old platforms this is needed so pip trusts PyPI downloads	Not specified
NGAS_EXTRA_PYTHON_PACKAGES	Comma-separated list of extra python packages to install	Not specified
NGAS_NO_CLIENT	If specified, skip the compilation and installation of the NGAS C client	Not specified
NGAS_NO_CRC32C	If specified, NGAS will not depend (and will avoid installing) the `crc32c` python package. This is useful if you know this feature is not planned to be used at runtime	Not specified
NGAS_DEVELOP	If specified, install the NGAS Python modules in development mode	Not specified
NGAS_NO_DOC_DEPENDENCIES	If specified, skip the installation of python packages needed to build the NGAS documentation	Not specified
NGAS_NO_BASH_PROFILE	If specified, skip the edition of the user’s `~/.bash_profile` for automatic virtualenv sourcing	Not specified
NGAS_SERVER_TYPE	The server type configured after installing NGAS (`archive`, `cache`)	`archive`

For example, to install the tip of the v8 branch as user foo in hosts bar1 and bar2, and without compiling the C client, the following command would do:

fab hl.user_deploy -u foo -H bar1,bar2 --set NGAS_NO_CLIENT,NGAS_REV=v8

Total system setup ¶

Note

sudo must be installed and configured in the target host for this task to work properly. Also, the user used with fab (fab -u <user>) needs to be properly configured on the target host to use sudo commands.

To perform a system-wide setup and NGAS install run:

fab hl.operations_deploy

This task will:

Check that SSH is working on the target host
Check that sudo is installed (sudo is used to run commands as root).
Install all necessary system packages (using the OS-specific package manager) for compiling NGAS and its dependencies
Compile and install a suitable version of python (2.7) if necessary
Create the NGAS_USER if necessary
Proceed with the normal NGAS compilation and installation as performed by hl.user_deploy
Install an /etc/init.d script for automatic startup of the server.

The user on the target host used for running the sudo commands is the SSH user given to fabric via the command line (fab -u <user>).

On top of the normal fabric variables used by hl.user_deploy the following additional variables control this script:

Variable	Description	Default value
NGAS_USER	The user under which the NGAS installation will take place	`ngas`
NGAS_EXTRA_PACKAGES	Comma-separated list of extra system-level packages to install on the target system(s)	Not specified

Currently supported OSs are Ubuntu, Debian, Fedora, CentOS, and MacOSX Darwin, but more might work or could be added in the future.

Other Fabric tasks ¶

On top of the two main Fabric tasks to install NGAS, our fabric modules define a number of other optional, high-level tasks that can be useful in other scenarios.

AWS deployment ¶

Note

The boto module is required for using this install option.

The fabric modules contain routines to create an NGAS installation on AWS machines. This is performed by running:

fab hl.aws_deploy

This procedure will: * Create and bring up the required AWS instances * Wait until they are fully operational, and * Perform a hl.operations_deploy on the instances.

On top of the normal fabric variables used by hl.user_deploy and hl.operations_deploy the following additional variables control the AWS-related aspects of the script:

Variable	Description	Default value
AWS_PROFILE	The profile to use when connecting to AWS	`NGAS`
AWS_REGION	The AWS region to connect to	`us-east-1`
AWS_KEY_NAME	The private SSH key to be used to create the instances, and later to connect to them	`icrar_ngas`
AWS_AMI_NAME	The name associated to an AMI (from a predetermined set of AMI IDs) which will be used to create the instance	`Amazon`
AWS_INSTANCES	The number of instances to create	`1`
AWS_INSTANCE_TYPE	The type of instances to create	`t1.micro`
AWS_INSTANCE_NAME	The name of instances to create	`NGAS_<rev>`
AWS_SEC_GROUP	The name of the security group to attach to the instances (will be created if it doesn’t exist)	`NGAS`
AWS_ELASTIC_IPS	A comma-separated list of public IPs to associate with the new instances, if specified.	Not specified

For example, to create 3 instances of type t3.micro on region us-east-2 one would run:

fab hl.aws_deploy --set AWS_REGION=us-east-2,AWS_INSTANCES=3,AWS_INSTANCE_TYPE=t3.micro

To assist with AWS-related procedures the following other tasks are also available:

fab aws.list_instances
fab aws.terminate_instance:instance_id=<the-instance-id>

Docker Image ¶

Note

These instructions are to build a docker image with NGAS. If you simply want to use the pre-built images see Docker container.

Note

The docker python package is required to use of this install option. Also, a local docker daemon must be running and the current user must have access to perform docker operations.

To build a Docker container containing an NGAS installation simply run:

fab hl.docker_image

This will generate an image called icrar/ngas:latest based on CentOS 7. When started, the container by default will run the NGAS server. The NGAS server will look for a configuration file under /home/ngas/NGAS/cfg/ngamsServer.conf, which by default needs to be provided via volume mapping.

This task will:

Create an initial Docker image with SSH on it
Start a container using that image
Perform a hl.operations_deploy on the container
Perform some cleanups on the container, including removing the NGAS_ROOT directory
Commit the container and create the final Docker image

On top of the normal fabric variables used by hl.user_deploy and hl.operations_deploy the following additional variables control the Docker-related aspects of the task:

Variable	Description	Default value
DOCKER_KEEP_NGAS_ROOT	If specified, the NGAS root directory will still be present in the final image	Not specified
DOCKER_KEEP_NGAS_SRC	If specified, the NGAS source directory will still be present in the final image	Not specified
DOCKER_IMAGE_REPOSITORY	The repository for the final image produced by this task	`icrar/ngas`

Docker container ¶

Alternatively, if you don’t need to build NGAS from sources you can use the pre-built docker image we distribute for NGAS. To do so you can do:

docker pull icrar/ngas:latest

Then follow the instructions in the DockerHub page to get yourself started.