Database access

Core database access

These are the core classes implementing the core database access logic, including dealing with connections, cursors and transactions. For the higher-level database object offered by the ngamsServer.ngamsServer.ngamsServer class please see Higher-level abstractions.

class ngamsLib.ngamsDbCore.ngamsDbCore(interface, parameters={}, createSnapshot=1, maxpoolcons=6, use_file_ignore=True, session_sql=None)

Core class for the NG/AMS DB interface.

close()

Close the DB pool.

Returns: Void.

dbCursor(sqlQuery, args=())

Create a cursor on the given query and return the cursor object.

query2(sqlQuery, args=())

Takes an SQL query and a tuple of arguments to bind to the query

transaction()

Creates a new transaction object and return it

class ngamsLib.ngamsDbCore.cursor2(pool, query, args)

A cursor that yields values and acts as a context manager

close()

Closes the underlying cursor and connection

fetch(howmany)

Fetches at most howmany results from the database at a given time, yielding them instead of returning them as a sequence. This makes client code simpler to write.

class ngamsLib.ngamsDbCore.transaction(db_core, pool)

A context manager that allows multiple SQL queries to be executed within a single transaction

execute(sql, args=())

Executes sql using args

Higher-level abstractions

ngamsLib.ngamsDb.from_config(cfg, maxpool=None)

Create a database object from a configuration object. If maxpool is not None, it overrides the value loaded from the configuration for the number of connections held by the pool.

class ngamsLib.ngamsDb.ngamsDb(interface, parameters={}, createSnapshot=1, maxpoolcons=6, use_file_ignore=True, session_sql=None)

Front-end class for the DB access module.

This class inherits from all the DB sub-classes (which in turn inherit from ngamsDbCore), thus exposing to the rest of the software a single class that implements all the database logic.

addDbChangeEvt(evtObj)

Add an Event Object (threading.Event) that will be triggered to indicate other threads that DB changes where introdiced.

evtObj: Event object (threading.Event).

Returns: Reference to object itself.

addDiskHistEntry(hostId, diskId, synopsis, descrMimeType=None, descr=None, origin=None, date=None)

Add an entry in the NGAS Disks History Table (ngas_disks_hist) indicating a major action or event occurring in the context of a disk.

dbConObj: Instance of NG/AMS DB class (ngamsDbBase).

diskId: Disk ID for the disk concerned (string).

synopsis: A short description of the action
(string/max. 255 char.s).
descrMimeType: The mime-type of the contents of the description
field. Must be specified when a description is given (string).
descr: An arbitrary long description of the action or event
in the life-time of the disk (string).
origin: Origin of the history log entry. Can either be the
name of an application or the name of an operator. If not specified (= None) it will be set to ‘NG/AMS - <host name>’ (string).
date: Date for adding the log entry. If not specified (set
to None), the function takes the current date and writes this in the new entry (string/ISO 8601).

Returns: Void.

addFileToContainer(containerId, fileId, force)

Adds the file pointed by fileId to the container pointed by containerId. If the file doesn’t exist an error will be raised. If the file is currently associated with a container and the force flag is not True an error will be raised also.

This method returns the uncompressed size of the file just added to the container. This can then be used to update the total size of the container

Parameters:
  • containerId (str) – the id of the container where the file will be added
  • fileId (str) – the id of the file to add to the container
  • force (bool) – force the operation
Returns:

the uncompressed file size of the file denoted by fileId
addSrvList(srvList)

Add a server list in the NGAS Server List Table and allocate a unique server list ID for it.

srvList: List of servers to add (string).

Returns: New server list ID (integer).

addSubscrBackLogEntry(hostId, portNo, subscrId, subscrUrl, fileId, fileName, fileVersion, ingestionDate, format)

Adds a Back-Log Entry in the DB. If there is already an entry for that file/Subscriber, a new entry is not created.

hostId: Host ID for NGAS host where Data Provider concerned
is running (string).

portNo: Port number used by Data Provider concerned (integer).

subscrUrl: Susbcriber URl to where the files are delivered
(string).

subscrId: Subscriber ID (string).

fileId: File ID (string).

fileName: Filename, i.e., name of file as stored in the
Subscription Back-Log Area (string).

fileVersion: File Version (integer).

ingestionDate: File Ingestion Date (string/ISO 8601).

format: Mime-type of file (string).

Returns: Void.

addToContainerSize(containerId, amount)

Updates the size of the indicated container by the given amount

asTimestamp(t)

Returns None if timestamp is None, otherwise calls convertTimeStamp

buildFileSummary1Query(columns, hostId=None, diskIds=[], fileIds=[], ignore=None, fileStatus=['00000000'], lowLimIngestDate=None, order=1)

Builds the SQL query for a File Summary1 query. The fields to be selected are left open (specified as %s).

For a description of the input parameters, check the man-page of ngamsDbBase.getFileSummary1().

Returns: SQL query for a File Summary 1 Query (string).

close()

Close the DB pool.

Returns: Void.

closeContainer(containerId)

Marks the container as “closed”; that is, it sets an ingestion date on it equals to the current time

containerExists(containerId)

Returns whether the container with ID containerId exists (True) or not (False).

Parameters:containerId – string
convertTimeStamp(t)

Convert a timestamp given in one of the following formats:

createContainer(containerName, containerSize=0, ingestionDate=None, parentContainerId=None, parentKnownToExist=False)

Creates a single container with name containerName. The ingestionDate parameter given to this method is a floating point number representing the number of seconds since the UNIX epoch, as returned by time.time()

If parentContainerId is given the new container will point to it as its parent. The parent container ID is checked for existence, unless parentKnownToExist indicates that the check is not necessary

Parameters:
  • containerName – string
  • containerSize – integer
  • ingestionDate – float
  • parentContainerId – string
  • parentKnownToExist – bool
Returns:

uuid.uuid4
createDbFileChangeStatusDoc(hostId, operation, fileInfoObjList, diskInfoObjList=[])

The function creates a pickle document in the ‘<Disk Mt Pt>/.db/cache’ directory from the information in the ‘fileInfoObj’ object.

operation: Has to be either ngams.NGAMS_DB_CH_FILE_INSERT or
ngams.NGAMS_DB_CH_FILE_DELETE (string).
fileInfoObj: List of instances of NG/AMS File Info Object
containing the information about the file (list/ngamsFileInfo).
diskInfoObjList: It is possible to give the information about the
disk(s) in question via a list of ngamsDiskInfo objects (list/ngamsDiskInfo).

Returns: Void.

createDbRemFileChangeStatusDoc(diskInfoObj, fileInfoObj)

The function creates a File Removal Status Document with the information about a file, which has been removed from the DB and which should be removed from the DB Snapshot for the disk concerned.

diskInfoObj: Disk Info Object with info for disk concerned
(ngamsDiskInfo).
fileInfoObj: Instance of NG/AMS File Info Object
containing the information about the file (ngamsFileInfo).

Returns: Void.

dbCursor(sqlQuery, args=())

Create a cursor on the given query and return the cursor object.

delSubscrBackLogEntries(hostId, portNo, subscrId)

Delete all entries to be delivered to a subscriber with subscrId

Parameters:
  • hostId (str) – Host ID for NGAS host where Data Provider concerned is running.
  • portNo (int) – Port number used by Data Provider concerned
  • subscrId (str) – Subscriber ID
delSubscrBackLogEntry(hostId, portNo, subscrId, fileId, fileVersion)

Delete an entry in the Subscription Back-Log Table.

hostId: Host ID for NGAS host where Data Provider concerned
is running (string).

portNo: Port number used by Data Provider concerned (integer).

subscrId: Subscriber ID (string).

fileId: File ID (string).

fileVersion: File Version (string).

fileName: Filename, i.e., name of file as stored in the
Subscription Back-Log Area (string).

Returns: Void.

deleteCacheEntry(diskId, fileId, fileVersion)

Delete an entry from the NGAS Cache Table.

diskId: Disk ID for the cached data object (string).

fileId: File ID for the cached data object (string).

fileVersion: Version of the cached data object (integer).

Returns: Reference to object itself.

deleteDiskInfo(diskId, delFileInfo=1)

Delete a record for a certain disk in the NGAS DB.

CAUTION: IF THE DB USER WITH WHICH THERE IS LOGGED IN HAS PERMISSION
TO EXECUTE DELETE STATEMENTS, THE INFORMATION ABOUT THE DISK IN THE NGAS DB WILL BE DELETED! THIS INFORMATION CANNOT BE RECOVERED!!

diskId: ID of disk for which to delete the entry (string).

delFileInfo: If set to 1, the file information for the files
stored on the disk is deleted as well (integer/0|1).

Returns: Reference to object itself.

deleteFileInfo(hostId, diskId, fileId, fileVersion, genSnapshot=1)

Delete one record for a certain file in the NGAS DB.

CAUTION: IF THE DB USER WITH WHICH THERE IS LOGGED IN HAS PERMISSION
TO EXECUTE DELETE STATEMENTS, THE INFORMATION ABOUT THE FILE(S) IN THE NGAS DB WILL BE DELETED! THIS INFORMATION CANNOT BE RECOVERED!!

diskId: ID of disk hosting the file (string).

fileId: ID of file to be deleted. No wildcards accepted
(string).

fileVersion: Version of file to delete (integer)

genSnapshot: Generate Db Snapshot (integer/0|1).

Returns: Reference to object itself.

deleteSubscriber(subscrId)

Delete the information for one Subscriber from the NGAS DB.

subscrId: Subscriber ID (string).

Returns: Reference to object itself.

destroySingleContainer(containerId, checkForChildren)

Destroys a single container with id containerId.

If the container contains subcontainers an error is issued.

Before destroying the container, all files associated to the container are removed from it first. If the container has subcontainers, it is not removed though

Parameters:
  • containerId – string
  • checkForChildren – bool
diskInDb(diskId)

Check if disk with the given Disk ID is available in the DB.

diskId: Disk ID (string).

Returns: 1 = Disk ID was found, 0 = Disk ID not found (integer).

dumpFileInfoCluster(clusterName, fileInfoDbmName=None, useFileKey=False, count=False)

Dump the info for the files registered in the name space of the referenced cluster.

Note, all files in the cluster are taken, also the ones marked as bad or to be ignored.

clusterName: Name of cluster to consider (string).

fileInfoDbmName: Base name of the DBM in which the file info will be
stored. If not given, a name will be generated automatically (string).
useFileKey: Use a file key (<File ID>_<Version>) as key as
opposed to just an integer key. NOTE: Multiple occurrences of a given File ID/Version will only appear once (boolean).
count: When useFileKey == True, if count is True, the

number of ocurrences of each File ID + Version is counted and an entry added in the DBM:

<File Key>__COUNTER

pointing to a counter indicating the number of occurrences. Note, the usage of ‘__’ in the name of the counter for each file, means it will be skipped when doing a ngamsDbm.getNext(), scanning through the contents of the DBM (boolean).

Returns: Final name of the DBM DB containing the info about
the files (string).
dumpMirroringQueue(instanceId)

Dump the entire contents of the DB Mirroring Queue into a DBM in raw format.

Returns: Name of DBM hosting the contents of the DB Mirroring
Queue (string).
fileInDb(diskId, fileId, fileVersion=-1)

Check if file with the given File ID is registered in NGAS DB in connection with the given Disk ID.

diskId: Disk ID (string)

fileId: File ID (string).

fileVersion: Version of the file. If -1 version is not taken
into account (integer).

Returns: 1 = file found, 0 = file no found (integer).

files_in_host(hostId, from_date=None)

Dump the info of the files defined by the parameters. The file info is dumped into a ngamsDbm DB.

For the parameters check man-page for: ngamsDbBase.getFileSummary1().

Returns: Name of the DBM DB containing the info about the files
(string).
fromTimestamp(timestamp)

Converts a database timestamp into a number of seconds from the epoch. This is the reverse of asTimestamp.

getAvailableVolumes(hostId)

Returns a list of rows for all disks that are not marked as completed on hostId.

getBestTargetDisk(diskIds, mtRootDir)

Find the best suitable target disk among a set of disks referred to by their Disk IDs. The condition is:

Get ID for disk that is most full, which is Main Disk, mounted, has a Host ID corresponding to this host ID, which is not completed, which has the lowest Slot ID and which has the Disk ID among the set of disks defined for the given mime-type.

diskIds: List with Disk IDs to probe for (list).

mtRootDir: Base diretory for the NG/AMS Server (string).

Returns: Disk ID or None if no matches were found (string)

getCacheContents(hostId)

Execute query by means of a cursor, with which the entire contents of the cache can be downloaded.

hostId: Name of host to consider (string).

Returns: Cursor object with which the contents can be retrieved
Cursor object (<NG/AMS DB Cursor Object API>).
getCfgPars(name)

Return the list of configuration parameters from the DB associated to the given name.

name: Name of the configuration association (string).

Returns: List with sub-lists with the information. The format is:

[[<Group ID>, <Parameter>, <Value>, <Comment>], …]

(list).

getClusterNameFromHostId(hostId)

Get the Cluster Name to which a node belongs from its Host ID.

hostId: Host ID (string).

Returns: Cluster Name (string).

getClusterReadyArchivingUnits(clusterName)

Return list of NAUs in the local cluster with archiving capability (archiving enabled + have capacity).

The resulting list of nodes will be formatted as:

[<Node>:<Port>, …] (list).

clusterName: Name of cluster to consider (string).

Returns: List with ready NAU nodes in the cluster (list/string).

getContainerIdForUniqueName(containerName)

Returns the ID of the container that can be uniquely identified by containerName. If no container with such a name exists, or if more than one exists, an error is raised

Parameters:containerName – string
Returns:string
getContainerName(containerId)

Returns the name of the container pointed by containerId. If no container with such ID exists, an error is raised

Parameters:containerId – string
getCreateDbSnapshot()

Return the flag indicating if a DB Snapshot should be created or not.

Returns: Value of the DB Snapshot Creation Flag (boolean).

getDbTime()

Return the time spent for the last DB access.

Returns: Last DB access time in seconds (float).

getDiskCompleted(diskId)

Check if a disk is marked in the NGAS DB as completed.

diskId: ID of the disk (string).

Returns: 1 = completed, 0 = not completed. If the disk
is not registered None is returned (integer).
getDiskIdFromSlotId(host, slotId)

Get a Disk ID for corresponding Slot ID and host name.

host: Host name (string).

slotId: ID of slot (string).

Returns: Disk ID or None if no match found (string).

getDiskIds()

Query the Disk IDs contained in the NGAS DB and return these in a list.

Returns: List with Disk IDs (list).

getDiskIdsMountedDisks(host, mtRootDir)

Get the Disk IDs for the disks mounted on the given host. A list is returned, which contains the Disk IDs of the disks mounted.

host: Name of host where the disk must be mounted (string).

mtRootDir: Base directory for NG/AMS (string).

Returns: List with Disk IDs (list).

getDiskIdsMtPtsMountedDisks(host)

Get the mount points for the disks mounted on the given host. A list is returned, which contains the Logical Names of the disks mounted.

host: Name of host where the disk must be mounted (string).

Returns: List with tuples containing Disk IDs and Mount
Points (list/tuple).
getDiskInfoForSlotsAndHost(host, slotIdList)

From a given host and a given list of Slot IDs, the method returns a list with the disk info for the disks matching these.

host: Host name where the disks considered must be
mounted (string).

slotIdList: List of Slot IDs for the disk considered (list).

Returns: List with Disk Info objects or [] if no matches
were found (list/ngamsDiskInfo).
getDiskInfoFromDiskId(diskId)

Query the information for one disk (referred to by its ID), and return this in raw format.

The information about the disk is stored in a list with the the lay-out following the sequence as listed in the overall man-page for the ngamsDbBase class.

diskId: ID of the disk (string).

Returns: Disk information for the disk or [] if disk was
not found (list).
getDiskInfoFromDiskIdList(diskIdList)

Get disk information from a list of Disk IDs given. The result is returned in a list containing again lists with the field as described in documentation for ngamsDbBase.getDiskInfoFromDiskId().

diskIdList: List with Disk IDs (list/string).

Returns: List with disk information (list/list).

getFileChecksum(diskId, fileId, fileVersion)

Get the checksum for the file.

diskId: ID of disk hosting the file (string).

fileId: ID of file to be deleted. No wildcards accepted
(string).

fileVersion: Version of file to delete (integer)

Returns: checksum (string | None).

getFileChecksumValueAndVariant(diskId, fileId, fileVersion)

Get the checksum value and variant for the file.

diskId: ID of disk hosting the file (string).

fileId: ID of file to be deleted. No wildcards accepted
(string).

fileVersion: Version of file to delete (integer)

Returns: checksum (string | None).

getFileInfoFromDiskIdFilename(diskId, filename)

The method queries the file information for a file referred to by the Disk ID for the disk hosting it and the filename as stored in the NGAS DB.

diskId: ID for disk hosting the file (string).

filename: NGAS (relative) filename (string).

Returns: Return ngamsFileInfo object with the information for the
file if found or None if the file was not found (ngamsFileInfo|None).
getFileInfoFromFileId(fileId, fileVersion=-1, diskId=None, ignore=None, dbCursor=1, order=1, order2=0)

The method queries the file information for the files with the given File ID and returns the information found in a list containing sub-lists each with a list with the information for the file from the ngas_files table, host ID and mount point. The following rules are applied when determining which files to return:

o All files are considered, also files which are Offline.

o Files marked to be ignored are ignored.

o Latest version - first priority.

It is possible to indicate if files marked as being ‘bad’ in the DB should be taken into account with the ‘ignoreBadFiles’ flag.

If a specific File Version is specified only that will be taken into account.

The data can be retrieved via the DB Cursor returned by this object. The format of each sub-result is:

[<see getFileInfoFromFileIdHostId()>, <host ID>, <mnt pt>]

fileId: File ID for file to be retrieved (string).

fileVersion: If a File Version is specified only information
for files with that version number and File ID are taken into account. The version must be a number in the range [1; oo[ (integer).
diskId: ID of disk where file is residing. If specified
to None (or empty string) the Disk ID is not taken into account (string).
ignore: If set to 0 or 1, this value of ignore will be
queried for. If set to None, ignore is not considered (None|0|1).
dbCursor: If set to 1, a DB cursor is returned from which
the files can be retrieved. Otherwise the result is queried and returned in a list (0|1/integer).
order: If set to 0, the list of matching file information
will not be order according to the file version (integer/0|1).
Returns: Cursor object or list with results
(<NG/AMS DB Cursor Object API>|list).
getFileInfoFromFileIdHostId(hostId, fileId, fileVersion=1, diskId=None, ignore=None)

Return list with information about a certain file referenced by its File ID. A list is returned with the following elements:

[<Disk ID>, <Filename>, <File ID>, <File Version>, <Format>,
<File Size>, <Uncompressed File Size>, <Compression>, <Ingestion Date>, <Ignore>, <Checksum>, <Checksum Plug-In>, <File Status>, <Creation Date>]
hostId: Name of host where the disk is mounted on
which the file is stored (string).

fileId: ID for file to acquire information for (string).

fileVersion: Version of the file to query information
for (integer).

diskId: Used to refer to a specific disk (string).

ignore: If set to 0 or 1, this value of ignore will be
queried for. If set to None, ignore is not considered (None|0|1).
Returns List with information about file, or [] if
no file(s) was found (list).
getFileInfoList(diskId, fileId='', fileVersion=-1, ignore=None, fetch_size=1000)

The function queries a set of files matching the conditions specified in the input parameters.

diskId: Disk ID of disk hosting the file(s) (string).

fileId: File ID of files to consider. Wildcards can be
used (string).
fileVersion: Version of file(s) to consider. If set to -1 this
is not taken into account (integer).
ignore: If set to 0 or 1, this value of ignore will be
queried for. If set to None, ignore is not considered (None|0|1).

Returns: Cursor object (<NG/AMS DB Cursor Object API>).

getFileSize(fileId, fileVersion)

Get the ingestion date for the file.

diskId: ID of disk hosting the file (string).

fileId: ID of file to be deleted. No wildcards accepted
(string).

fileVersion: Version of file to delete (integer)

Returns: Ingestion date for file or None if file not found
(string/ISO 8601 | None).
getFileStatus(fileId, fileVersion, diskId)

Get the file_status string (bit) value in the ngas_files table.

fileId: ID of file (string).

fileVersion: Version of file (integer).

diskId: Disk ID for disk where file is stored (string).

Returns: File Status (8 bits) (string).

getFileSummary1(hostId=None, diskIds=[], fileIds=[], ignore=None, fileStatus=['00000000'], lowLimIngestDate=None, order=1)

Return summary information about files. The information is returned in a list containing again sub-lists with contents as defined by ngamsDbCore.getNgasSummary1Cols() (see general documentation of the ngamsDbBase Class).

hostId: Name of NGAS host on which the files reside
(string).
diskIds: Used to limit the query to certain disks
(list/string).
fileIds: List of file IDs for which to query information.
If not specified, all files of the referenced host will be chosen (list/string|[]).
ignore: If set to 0 or 1, this value of ignore will be
queried for. If set to None, ignore is not considered (None|0|1).
fileStatus: With this parameter it is possible to indicate which
files to consider according to their File Status (list).
lowLimIngestDate: Lower limit in time for which files are taken into
account. Only files with an Ingestion Date after this date, are taken into account (string/ISO 8601).
order: Used to trigger ordering by Slot ID + Ingestion Date
(integer/0|1).

Returns: Cursor object (<NG/AMS DB Cursor Object API>).

getFileSummary1SingleFile(diskId, fileId, fileVersion)

Same as getFileSummary1() but for a single (specific) file.

Returns: List with information from query (list).

getFileSummary2(hostId=None, fileIds=[], diskId=None, ignore=None, ing_date=None, max_num_records=None, upto_ing_date=None, fetch_size=1000)

Return summary information about files. An NG/AMS DB Cursor Object is created, which can be used to query the information sequentially.

The information is returned in a list containing again sub-lists with contents as defined by ngamsDbBase._sum2Cols (see general documentation of the ngamsDbBase Class.

This method returns all the files stored on an NGAS system also the ones with a File Status indicating that it is bad.

hostId: Name of NGAS host on which the files reside. If
None is specified, the host is not taken into account (string or a list of string).
fileIds: List of file IDs for which to query information.
If not specified, all files of the referenced host will be chosen (list/string|[]).
diskId: Used to refer to all files on a given disk
(string|None).
ignore: If set to 0 or 1, this value of ignore will be
queried for. If set to None, ignore is not considered (None|0|1).

max_num_records: The maximum number of returned records (if presented) (int)

Returns: Cursor object (<NG/AMS DB Cursor Object API>).

getFileSummary3(fileId, hostId=None, domain=None, diskId=None, fileVersion=-1, cursor=True, include_compression=False)

Return information about files matching the conditions which are not in ignore and which are not marked as bad.

Files are ordered by the File Version (descending).

The resulting file information will be:

<Host ID>, <Ip Address>, <Port>, <Mountpoint>, <Filename>, <File Version>, <format>

fileId: ID of file to retrieve (string).

hostId: Host ID of node hosting file (string|None).

domain: Domain in which the node is residing (string|None).

diskId: Disk ID of disk hosting file (string|None).

fileVersion: Version of file to retrieve (integer).

cursor: Return DB cursor rather than the results (boolean).

Returns: Cursor object (<NG/AMS DB Cursor Object API>).

getFileSummarySpuriousFiles1(hostId=None, diskId=None, fileId=None, fileVersion=None, fetch_size=1000)

Return summary information about spurious files, i.e. files registered in the DB as to be ignored and/or having a status indicating that they’re not OK. The information is returned in a list containing again sub-lists with contents as defined by ngamsDbBase.getNgasSummary1Cols() (see general documentation of the ngamsDbBase Class.

hostId: Name of NGAS host on which the files reside
(string).

diskId: Disk ID of disk to take into account (string|None).

fileId: File ID of file(s) to take into account
(string|None).
fileVersion: Version of file(s) to take into account
(integer|None).

Returns: Cursor object (<NG/AMS DB Cursor Object API>).

getHostIdsFromClusterName(clusterName)

Return the list of host IDs within the context of a given cluster.

clusterName: Name of cluster to consider (string).

Returns: List with nodes in the cluster (list/string).

getHostInfoFromHostIds(hostList)

Return a dictionary with the information in connection with each host. If for a host ID given, no information is found in the NGAS Hosts Table, the value for this wil be None.

hostList: List of host IDs (list/string).

Returns: List with sub-lists containing the information about the
hosts from the NGAS Hosts Table (list).
getIngDate(diskId, fileId, fileVersion)

Get the ingestion date for the file.

diskId: ID of disk hosting the file (string).

fileId: ID of file to be deleted. No wildcards accepted
(string).

fileVersion: Version of file to delete (integer)

Returns: Ingestion date for file or None if file not found
(string/ISO 8601 | None).
getIpFromHostId(hostId)

Get the IP Address of a host from its Host ID.

hostId: Host ID (string).

Returns: IP Address (string).

getLastDiskCheck(hostId='')

Queries all the Last Check Flags for all disks or for all disks currently mounted in a specific host. A Dictionary is returned containining the Disk IDs as keys, and the time for the last check in seconds. If the value is NULL in the DB, it is set to 0.

hostId: If specified, only disks mounted in this system are
taken into account (string).
Returns: Dictionary with entry for each disk. Key is Disk
ID (dictionary).
getLatestFileVersion(fileId)

The method queries the latest File Version for the file with the given File ID. If a file with the given ID does not exist, -1 is returned.

fileId: File ID (string).

Returns: Latest File Version (integer).

getLogicalNameFromDiskId(diskId)

Query the Logical Name of a disk from the DB, based on the Disk ID of the disk.

diskId: Disk ID (string).

Returns: Logical Name or None if not found (string | None).

getLogicalNamesMountedDisks(host)

Get the Logical Names for the disks mounted on the given host. A list is returned, which contains the Logical Names of the disks mounted.

host: Name of host where the disk must be mounted (string).

Returns: List with Logical Names (list).

getMaxDiskNumber(cat=None)

Get the maximum disk index (number) in connection with the Logical Disk Names in the DB.

cat: ‘M’ for Main, ‘R’ for Replication (string).

Returns: The maximum disk number or None if this could not
be generated (integer).
getMinLastDiskCheck(hostId)

Get the timestamp for the disk that was checked longest time ago for all disks mounted in a specific NGAS Host.

hostId: Host ID of the host to consider (string).

Returns: Time since the ‘oldest’, last check (seconds since epoch)
(integer).
getMtPtFromDiskId(diskId)

Get the mount point for the disk referred to.

diskId: ID of the disk (string).

Returns: Mount point of disk or None if not mounted or
not found (string|None)
getNgasFilesMap()

Return the reference to the map (dictionary) containing the mapping between the column name and index of the ngas_files table.

Returns: Reference to NGAS Files Table name map (dictionary).

getNumberOfFiles(diskId='', fileId='', fileVersion=-1, ignore=None, onlyOnlineFiles=0)

Get the number of files stored on a disk.

diskId: Disk ID of disk to get the number of files for
(string).

fileId: File ID for file to be retrieved (string).

fileVersion: If a File Version is specified only information
for files with that version number and File ID are taken into account. The version must be a number in the range [1; oo[ (integer).
ignore: If set to 0 or 1, this value of ignore will be
queried for. If set to None, ignore is not considered (None|0|1).
onlyOnlineFiles: If specified, only files which are Online or on
suspended nodes are considered (integer/0|1).

Return: Number of files stored on the disk (integer).

getPortNoFromHostId(hostId)

Return the port number corresponding to the host ID.

hostId: Host ID (string).

Return: Port number (integer).

getSlotIdFromDiskId(diskId)

Get the Slot ID for a disk, given by the Disk ID for the disk.

diskId: ID of the disk (string).

Returns: Slot ID of disk. If disk is not found None is
returned (string | None).
getSlotIdsMountedDisks(host)

Get the Slot IDs for the disks mounted on the given host. A list is returned, which contains the Slot IDs of the disks mounted.

host: Name of host where the disk must be mounted (string).

Returns: List with Slot IDs (list).

getSpaceAvailForHost(hostId)

Return the amount of free disk space for the given host. This is calculated as the total sum of free space for all non-completed volumes mounted in this node, according to the DB.

hostId: Name of host (string).

Returns: Amount of free disk space in MB bytes (float)

getSrvDataChecking(hostId)

Return flag indicating if server is executing a Data Consistency Check.

hostId: Host ID (string).

Returns: Server suspension flag (integer/0|1).

getSrvListFromId(srvListId)

Get a server list from its ID. If no list with that ID is found, None is returned.

srvListId: Server list ID (integer).

Returns: Server list associated with the given ID (string|None).

getSrvListIdFromSrvList(srvList)

Get the server list ID associated with the server list. If not defined, a new can be allocated in the NGAS Servers Table automatically.

srvList: Server list (‘<host>:<port>,…’) (string).

Returns: Server list ID (integer).

getSrvSuspended(contactAddr, ngasHostId=None)

Return flag indicating if the server is suspended.

contactAddr: Host ID or IP address (string).

ngasHostId: NGAS Host ID, e.g. myhost:8888 (string).

Returns: Server suspension flag (integer/0|1).

getSubscrBackLog(hostId, portNo, selectDiskId=False)

Read all entries in the Subscriber Back-Log Table ‘belonging’ to a specific Data Provider, and return these in a list with sub-lists.

hostId: Host ID of Data Provider (string).

portNo: Port number used by Data Provider (integer).

Returns: List containing sub-list with the following information:

[[<Subscr. ID>, <Subscr. URL>, <File ID>, <Filename>,
<File Version>, <Ingestion Date>, <Format <Mime-Type>], …]

Note that the part of the list after the Subscriber URL is the same as generated by ngamsDbBase.getFileSummary2() (list/list).

getSubscrBackLogBySubscrId(subscrId)

Get all entries in the Susbscriber Back-log Table to be delivered to a specific subscriber

subscrId Subscriber Id

Returns List containing sublist with the following information:
[[<file_id>, <file_version>], …]
getSubscrBackLogCount(hostId, portNo)

Read the number of entries in the Subscriber Back-Log Table ‘belonging’ to a specific Data Provider/Mover

hostId: Host ID of Data Provider (string).

portNo: Port number used by Data Provider (integer).

Returns: The number of records (integer)

getSubscrQueue(subscrId, status=None)

Read all entries in the ngas_subscr_queue table ‘belonging’ to a specific subscriber, and where the status meets the “status” condition

subscrId: subscriber Id (string) status: the status of current file delivery (int or None)

getSubscriberInfo(subscrId=None, hostId=None, portNo=-1)

Get the information for one or more Subcribers from the ngas_subscribers table and return the contents in a list. The format of this list is formatted as follows:

[<Host ID>, <Port No>, <Priority>, <Subscriber ID>, <Subscriber URL>,
<Subscription Start Date>, <Subscription Filter Plug-In>, <Subscription Filter Plug-In Parameters>, <Last File Ingestion Date>]

subscrId: ID of the Subcriber (string).

hostId: Limit the query to Subscribers in connection with one
host (Data Provider) (string).
portNo: Limit the query to Subscribers in connection with one
host (Data Provider) (integer).
Returns: If a Subscriber ID is specified: List with information

about the Subscriber (if found). Otherwise [] is returned (list).

If no Subscriber ID is given: List with sub-lists with information for all Subscribers. Otherwise [] is returned (list/list).

getSubscriberStatus(subscrIds, hostId='', portNo=-1)

Method to query the information about the Ingestion Date of the last file delivered to the Subscriber. A list is returned, which contains the following:

[(<Subscriber ID>, <Last File Ingestion Date (ISO 8601)>), …]

subscrIds: List of Subscriber ID to query (list/string).

hostId: Host name of Subscriber host (string).

portNo: Port number used by Subscriber host (integer).

Returns: List with Subscriber status (list/tuple/string).

getSumBytesStored(diskId)

Get the total sum of the sizes of the data files stored on a disk and return this.

diskId: Disk ID of disk to get the sum for (string).

Return: Total sum of bytes stored on the disk (integer).

getWakeUpRequests(hostId)

Generates a tuple with suspended NGAS Hosts that have requested to be woken up by this host.

Returns: Tuple containing sub-tuples with information about hosts

to be woken up:

(({host id}, {wake-up time (secs since epoch)}), …)

(list/tuple)
hasCfgPar(groupId, parName)

Return 1 if the given configuration parameter (given by its Simplified XPath name) and Configuration Group ID is defined in the configuration table in the DB.

groupId: Group ID for the parameter (string).

parName: Name of parameter (string).

Returns: 1 = parameter defined, 0 = parameter not defined
(integer/0|1).
insertCacheEntry(diskId, fileId, fileVersion, cacheTime, delete)

Insert a new cache entry into the NGAS Cache Table.

diskId: Disk ID of the cache entry (string).

fileId: File ID of the cache entry (string).

fileVersion: File Version of the cache entry (string).

cacheTime: Time the file entered in the cache
(= ngas_files.ingestion_time) (float).
delete: Flag indicating if the entry is scheduled for
deletion (boolean).

Returns: Reference to object itself.

insertSubscriberEntry(sub_obj)

Inserts the new subscription object into the NGAS subscription table. If an object with the same subscription ID exists, its contents are returned; otherwise the given object is returned.

markHostSuspended(hostId)

Mark a host as being suspended in the NGAS DB.

hostId: Name of host to mark as suspended. If not given the
local host name is used (string).

Returns: Reference to object itself.

mirReqInQueue(fileId, fileVersion, instanceId)

Probe if the a Mirroring Request with the given ID is in the associated Mirroring Queue.

fileId: File ID (string).

fileVersion: File Version (integer).

instanceId: Identification of the NGAS instance taking care of
coordinating the mirroring (string).

Returns: Indication if the request is in the queue (boolean).

query2(sqlQuery, args=())

Takes an SQL query and a tuple of arguments to bind to the query

read(containerId)

Reads a single ngamsContainer object from the database

Parameters:containerId (str) – the id of the container to read
Returns:the container object
Return type:ngamsContainer.ngamsContainer
readHierarchy(containerId, includeFiles=False)

Reads an ngamsContainer object from the database and recursively populates it with its children containers.

Parameters:containerId (str) – the id of the container whose hierarchy is to be read
Returns:The container object recursively populated
Return type:ngamsContainer.ngamsContainer
relGlobalDbSem()

Release acquired access to a critical, global DB interaction.

Returns: Reference to object itself.

removeFileFromContainer(fileId, containerId)

Removes the file pointed by fileId from the container pointed by containerId. If the file doesn’t exist an error will be raised. If the file is currently not associated with the indicated container and error will be raised also.

This method returns the uncompressed size of the file just removed from the container. This can then be used to update the total size of the container

Parameters:
  • fileId (str) – the id to remove from the container
  • containerId (str) – the container from which the file needs to be removed
Returns:

the uncompressed size of the file just removed from the container
Return type:

integer
remove_file(file_size, disk_id)

Update the disk information to reflect that a file has been removed from the disk

Parameters:
  • file_size (int) – the size of the file on disk
  • disk_id (str) – the ID of the disk
replace_file(old_file_size, old_disk_id, new_file_size, new_disk_id)

Update the disk information to reflect a file, potentially of a different size, is being replaced, potentially in a different disk

Parameters:
  • old_file_size (int) – the size of the old copy of the file on disk
  • old_disk_id (str) – the ID of the disk with the old copy of the file
  • new_file_size (int) – the size of the new copy of the file on disk, could be the same as old_file_size.
  • new_disk_id (str) – the ID of the disk with the old copy of the file, could be the same as new_disk_id.
reqWakeUpCall(localHostId, wakeUpHostId, wakeUpTime)

Request a Wake-Up Call via the DB.

wakeUpHostId: Name of host where the NG/AMS Server requested for
the Wake-Up Call is running (string).
wakeUpTime: Absolute time for being woken up (seconds since
epoch) (integer).

Returns: Reference to object itself.

resetDbTime()

Reset the Db timer.

Returns: Reference to object itself.

resetWakeUpCall(hostId, resetSrvSusp=0)

Cancel/reset the Wake-Up Call parameters.

hostId: If specified, another host ID than the one where
this NG/AMS Server is running can be indicated (string).

Returns: Reference to object itself.

setContainerSize(containerId, containerSize)

Updates the size of the indicated container

setDbTmpDir(tmpDir)

Set the DB temporary directory.

tmpDir: Temporary directory (string).

Returns: Reference to object itself.

setFileChecksum(hostId, fileId, fileVersion, diskId, checksum, checksumPlugIn)

Set the checksum value in the ngas_files table.

hostId: ID of this NGAS host

fileId: ID of file (string).

fileVersion: Version of file (integer).

diskId: ID of disk where file is stored (string).

checksum: Checksum of file (string).

checksumPlugIn: Name of plug-in used to generate the
checksum (string).

Returns: Reference to object itself.

setLastCheckDisk(diskId, timeSecs)

Update the Last Check Flag for a disk.

diskId: ID of disk for which to update record (string).

timeSecs: Time in seconds to set for the disk (integer).

Returns: Reference to object itself.

setLogicalNameForDiskId(diskId, logicalName)

Change the Logical Name of the disk with the given Disk ID.

diskId: Disk ID (string).

logicalName: New Logical Name (string).

Returns: Void.

subscrBackLogEntryInDb(hostId, portNo, subscrId, fileId, fileVersion)

Check if there is an entry in the Subscription Back-Log for that file/Subscriber.

hostId: Host ID for NGAS host where Data Provider concerned
is running (string).

portNo: Port number used by Data Provider concerned (integer).

subscrId: Subscriber ID (string).

fileId: File ID (string).

fileVersion: File Version (string).

Returns: 1 = file found, 0 = file no found (integer).

subscriberInDb(subscrId)

Check if the Subscriber with the given ID is registered in the DB.

subscrId: Subscriber ID (string).

Returns: 1 = Subscriber registered, 0 = Subscriber not
registered (integer).
takeGlobalDbSem()

Acquire access to a critical, global DB interaction.

Returns: Reference to object itself.

transaction()

Creates a new transaction object and return it

triggerEvents(eventInfo=None)

Set the Event Objects to inform other threads about DB changes.

eventInfo: Piece of information to be transferred from one
thread to another (free format).

Returns: Reference to object itself.

unpackMirReqSqlResult(sqlResult)

Unpack a SQL result for one row in the DB Mirroring Table. The columns in the result must be ordered according to the sequence given by ngamsDbCore.getNgasMirQueueCols().

sqlResult: List with elements resulting from the query for one
row (list).

Returns: Mirroring Request Object (ngamsMirroringRequest).

updateCacheEntry(diskId, fileId, fileVersion, delete)

Update the online status of this cached data object.

diskId: Disk ID for the cached data object (string).

fileId: File ID for the cached data object (string).

fileVersion: Version of the cached data object (integer).

delete: Entry scheduled for deletion (integer/0|1).

Returns: Reference to object itself.

updateDataCheckStat(hostId, start, remain, estimTime, rate, checkMb, checkedMb, checkFiles, checkedFiles)

Update the statistics for the Data Checking Thread.

hostId: ID of NGAS Host to update statistics for (string).

start: Start of checking in seconds since epoch (integer).

remain: Estimated remaining time in seconds (integer).

estimTime: Estimated total time in seconds to complete the check
cycle (integer)

rate: Rate of checking in MB/s (float).

checkMb: Amount of data to check in MB (float).

checkedMb: Amount checked in MB (float).

checkFiles: Number of files to check (integer).

checkedFiles: Number of files checked (integer).

Returns: Reference to object itself.

updateDbTime(dbAccessTime)

Update the DB access timer.

dbAccessTime: DB access time to add in seconds (float).

Returns: Reference to object itself.

updateDiskFileStatus(diskId, fileSize)

Update the NGAS Disks Table according to a new file archived.

diskId: Disk ID (string).

fileSize: Size of file as stored on disk (integer).

Returns: Reference to object itself.

updateDiskInfo(fileSize, diskId)

Update the row for the volume diskId hosting the new file of size fileSize.

updateMirReq(mirReqObj)

Update the referenced Mirroring Request in the DB. The request is defined by a the set of File ID and File Version.

mirReqObj: Instance of the Mirroring Request Object containing the
information about the Mirroring Request (ngamsMirroringRequest).

Returns: Reference to object itself.

updateSrvHostInfo(hostId, srvInfo)

Update the information in the DB, which is managed by the server itself. All columns starting with srv_ in the ngas_hosts tables are defined. The values can be taken from an instance of the ngamsHostInfo class.

srvInfo: List containing all information about the host. These are
all fields starting with srv_ from srv_version to srv_state (list).
ignoreErr: If set to 1, a possible exception thrown will be
caught, and this error ignored. Otherwise the method will throw an exception itself (integer/0|1).

Returns: Void.

updateStatusMirReq(fileId, fileVersion, newStatus)

Update the status of the Mirroring Request.

fileId: File ID (string).

fileVersion: File Version (integer).

newStatus: New status for the Mirroring Request to write to the DB
(ngamsMirroringRequest.NGAMS_MIR_REQ_STAT_SCHED, …)

Returns: Reference to object itself.

updateSubscrQueueEntry(subscrId, fileId, fileVersion, diskId, status, status_date, comment=None)

Update the status (and comment) of a file in the persistent queue given its primary key

updateSubscrQueueEntryStatus(subscrId, oldStatus, newStatus)

change the status from old to new for files belonging to a subscriber

updateSubscrStatus(subscrId, fileIngDate)

Update the Subscriber Status so that it reflects the File Ingestion Date of the last file ingested.

subscrId: Subscriber ID (string).

fileIngDate: File Ingestion Date (string/ISO 8601).

Returns: Void.

updateSubscriberEntry(sub_obj)

The method writes the information in connection with a Subscriber in the NGAS DB. If an entry already exists for that disk, it is updated with the information given as input parameters. Otherwise, a new entry is created.

hostId: … filterPlugInPars: Parameters for the Subscriber (string).

priority: Priority of Subscriber (integer).

startDate: Date the subscription should start from
(string/ISO 8601).
lastFileIngDate: Ingestion dtae of last file delivered
(string/ISO 8601).
Returns: Returns 1 if a new entry was created in the DB
and 0 if an existing entry was updated (integer/0|1).
writeCfgPar(groupId, parName, value, comment)

Write a configuration parameter to the NGAS DB. If the parameter is already defined, the value/comment are updated.

groupId: Configuration Group ID (string).

parName: Name of parameter (string).

value: Value of parameter (string).

comment: Comment for parameter (string).

Returns: Reference to object itself.

writeDiskEntry(diskId, archive, installationDate, type, manufacturer, logicalName, hostId, slotId, mounted, mountPoint, numberOfFiles, availableMb, bytesStored, completed, completionDate, checksum, totalDiskWriteTime, lastCheck, lastHostId)

The method writes the information in connection with a disk in the NGAS DB. If an entry already exists for that disk, it is updated with the information contained in the Disk Info Object. Otherwise, a new entry is created.

Returns:1 if a new entry was created in the DB, 0 if an existing entry was updated.
writeFileEntry(hostId, diskId, filename, fileId, fileVersion, format, fileSize, uncompressedFileSize, compression, ingestionDate, ignore, checksum, checksumPlugIn, fileStatus, creationDate, iotime, ingestionRate, genSnapshot=1, updateDiskInfo=0, prev_disk_id=None)

The method writes the information in connection with a file in the NGAS DB. If an entry already exists for that file, it is updated with the information contained in the File Info Object. Otherwise, a new entry is created.

diskId Values for the columns in the ngas_disks … table (use values returned from ngamsFileInfo).

genSnapshot: Generate a snapshot file (integer/0|1).

updateDiskInfo: Update automatically the disk info for the
disk hosting this file (integer/0|1).

Returns: Void.

writeHostInfo(hostInfoObj)

Create an entry in the NGAS Hosts Table

hostInfoObj: ngamsHostInfo object containing the information for the
new entry (ngamsHostInfo).

Returns: Reference to object itself.

writeMirReq(mirReqObj, check=True)

Write the referenced Mirroring Request in the DB. The request is defined by a the set of File ID and File Version.

mirReqObj: Instance of the Mirroring Request Object containing the
information about the Mirroring Request (ngamsMirroringRequest).
check: Check if the entry is already in the queue. In case yes,
just update it (boolean).

Returns: Reference to object itself.