BEAT-I IDL

The BEAT-I IDL interface consists of just a handful of IDL 'named structures' and functions. A primary design goal was to simplify, as much as possible, the number of different types and functions a user would have to master in order to effectively use the library.

This page describes all IDL 'named structure' types and functions provided by the BEAT-I IDL interface.

Contents

• Named Structures
  • BEAT_DATAHANDLE
  • BEAT_GEOLOCATION
  • BEAT_ERROR
  • BEAT_NO_DATA
• Functions and Procedures
  • BEAT_VERSION
  • BEAT_GETOPT
  • BEAT_SETOPT
  • BEAT_OPEN
  • BEAT_CLOSE
  • BEAT_PRODUCT_CLASS
  • BEAT_PRODUCT_TYPE
  • BEAT_PRODUCT_VERSION
  • BEAT_FETCH
    • The first argument
    • Specification arguments
    • Return type
      • Fetching simple (non-composite) types
      • Fetching records
      • Fetching arrays
  • BEAT_FETCH_DATAHANDLE
  • BEAT_ATTRIBUTES
  • BEAT_FIELDAVAILABLE
  • BEAT_FIELDCOUNT
  • BEAT_FIELDNAMES
  • BEAT_SIZE
  • BEAT_DESCRIPTION
  • BEAT_UNIT
  • BEAT_TIME_TO_STRING
  • BEAT_IS_NO_DATA
  • BEAT_IS_ERROR
  • BEAT_UNLOAD

Named Structures

The BEAT-I IDL interface defines five 'named structures'. This section discusses them in detail. For reference, the five types are shown in the table below.

name	fields	type	usage
BEAT_DATAHANDLE	PF_ID OPAQUE	ULONG64 not user-accessible	represents a data-item in a product-file
BEAT_GEOLOCATION	LATITUDE LONGITUDE	DOUBLE DOUBLE	geolocation specificiation using latitude/longitude coordinates
BEAT_ERROR	ERRNO MESSAGE	INT STRING	BEAT call return status (can also indicate success)
BEAT_NO_DATA	OPAQUE	BYTE	represents an empty data item; the OPAQUE field is a dummy value and should not be used

The BEAT_DATAHANDLE named structure

The BEAT_DATAHANDLE type represents a piece of data within a currently opened product-file; internally, it maintains a pointer to a certain offset within the data product, as well as information on the type of expression residing at that offset.

BEAT_DATAHANDLE variables are used extensively 'under the hood' of the BEAT-I IDL library; however, there are two closely related cases where this type will also be visible at the IDL user level.

If a call to BEAT_FETCH is made that should return an 'array of array' or 'array of record' type, the call will return an 'array of BEAT_DATAHANDLE' instead. The reason for this is that the sophisticated type system supported by the BEAT data dictionary does not map in full generality onto the IDL type system.

The notion of an 'array of array' cannot be mapped onto IDL other than by combining the dimensions of the outer and inner arrays; however, for technical reasons, this would only work for a subset of cases described in the data dictionary.

Similarly, the notion of an 'array of record' could, on first sight, be expressed by using a structure-array in IDL. Again, this is not possible in the general case supported by the data-dictionary.

For consistency, we have chosen to handle all these cases uniformly, always returning an 'array of BEAT_DATAHANDLE' when an 'array of array' or an 'array of record' is fetched. You can then manually traverse the 'array of BEAT_DATAHANDLE', and use BEAT_FETCH to obtain the individual records or arrays represented by the BEAT_DATAHANDLE elements.

Although the proper use of BEAT_DATAHANDLE-arrays is probably the most subtle and difficult aspect of using the BEAT-I IDL interface, it has proved to be quite easy to get used to. In fact, when used properly, the use of an 'array of BEAT_DATAHANDLE' often improves performance as the BEAT_DATAHANDLE elements effectively cache the information needed to access the data, thereby speeding up subsequent accesses.

The BEAT_DATAHANDLE structure has two fields:

• The PF_ID is a product-file ID as returned by the BEAT_OPEN function (see below). At the IDL user level, you are free to inspect its value. However, do not change its value.
• The OPAQUE field is used internally to maintain pointer and data-dictionary type information. This field should not be accessed from within IDL; most importantly, its value should not be changed.

The meaning of a BEAT_DATAHANDLE becomes undefined after the product to which it is associated is closed; You should not use it in subsequent BEAT-I IDL operations. Doing so will generate an error.

The BEAT_GEOLOCATION named structure

The BEAT_GEOLOCATION structure represents a (longitude, latitude) tuple that specifies a geolocation.

The BEAT_ERROR named structure

The BEAT_ERROR structure may be returned by any BEAT-I IDL function, usually indicating that the operation failed for some reason. Despite the name, you should think of a BEAT_ERROR as a 'status report' of the last BEAT-I IDL call; the status may, in fact, indicate successful completion.

For example, the BEAT_CLOSE function will always return a value of type BEAT_ERROR, even when the operation succeeds.

The BEAT_ERROR has two fields. The ERRNO field gives a numerical error code. This code is guaranteed to be either 0 (zero), indicating success, or a negative number, indicating failure.

The MESSAGE field is a STRING that provides a human-readable description corresponding to the ERRNO field. Some errors originate from the underlying C library that BEAT-I IDL uses, others are specific to IDL. For example, if one of the parameters you pass to a BEAT-I IDL function is incorrect you will get a BEAT_ERROR indicating the type of error.

The BEAT_NO_DATA named structure

The BEAT_NO_DATA is used to represent unavailable record fields or zero-element arrays in IDL. BEAT is capable of representing arrays that have zero indices in any dimension but such a data structure cannot be represented in IDL. Whenever BEAT needs to represent a zero-element array, an instance of type BEAT_NO_DATA is used instead.

The BEAT_NO_DATA has one field. The OPAQUE field is present only to assure that the structure has a field (which is required by IDL). It does not contain any useful data.

Functions and Procedures

Function BEAT_VERSION(): STRING

The BEAT_VERSION function returns a string containing the current version number of BEAT. The version number is always of the format 'x.y.z', i.e., major, minor, and revision numbers, separated by dots.

Function BEAT_GETOPT(option_name: STRING): INT

The BEAT_GETOPT function returns the numerical value of a BEAT-I IDL option. The following options are supported:

option name	possible values	default value	meaning
FilterRecordFields	0 or 1	1	if set, the BEAT-I IDL interface will skip spare fields and fields that have a fixed value (such as header labels) when retrieving records, avoiding unnecessary clutter.
PerformConversions	0 or 1	1	if set, the BEAT-I IDL interface will perform unit/value conversions for fields that have a 'conversion' defined in the data dictionary.
PerformBoundaryChecks	0 or 1	1	if set, the BEAT-I IDL interface will perform boundary checking on array accesses. This makes BEAT more robust, at the cost of some performance. It is recommended to disable boundary checks only for thoroughly debugged programs; disabling boundary checks can lead to wrong results and/or IDL crashes if arrays are accessed beyond their boundaries.
TimeUnitDays	0 or 1	0	if set, the BEAT-I IDL interface will represent all time values as days since 1-1-2000 instead of seconds since 1-1-2000. This is true both for fetched values and for values passed to BEAT_TIME_TO_STRING. Use of this functionality is discouraged, as this will make data exchange between programs more difficult.
UseSpecialTypes	0 or 1	1	By default BEAT uses a technique called 'memory mapping' to open and access data from product files. Using mmap greatly outperforms the default approach of reading data using the open()/read() combination. The downside of mapping a file into memory is that it takes away valuable address space. When you run a 32-bit Operating System your maximum addressable memory range is 4GB and if you simultaneously try to keep a few large product files open your memory space can quickly become full. Opening additional files will then produce 'out of memory' errors. Note that this 'out of memory' situation has nothing to do with the amount of RAM you have installed in your computer. It is only related to the size of a memory pointer on your system, which is limited to 4GB. If you are using BEAT in a situation where you need to have multiple large product files open at the same time you can turn of the use of memory mapping by disabling this option. If you change the memory mapping option, the new setting will only be applicable for files that will be opened after you changed the option. Any files that were already open will keep using the mechanism with which they were opened.
UseMMap	0 or 1	1	if you disable this option, the BEAT-I IDL interface will use the base type of a special type (and not the special type itself) when reading data or retrieving information about a data item. See the BEAT Data Dictionary documentation for more information about special types.
Verbose	0 or 1	1	if set, the BEAT-I IDL interface will echo 'error' return values to the IDL command line just before returning them to the IDL user level. This can be helpful in detecting problems.

Function BEAT_SETOPT(option_name: STRING, new_value: INT): INT

The BEAT_SETOPT function is used to set the the BEAT-I IDL interface option to a new value. See the BEAT_GETOPT description above for a list of allowed options and values, and their associated meanings.

The BEAT_SETOPT function will return the previous value of the option.

Function BEAT_OPEN(STRING filename): ULONG64

BEAT_OPEN will attempt to open the file (which may fail, for example, if you specify a file that does not exist).

If BEAT_OPEN succeeds, it returns a unique positive 64-bit unsigned integer that can subsequently be used to identifty the file in BEAT_FETCH and BEAT_CLOSE calls. If it fails a BEAT_ERROR structure is returned giving the reason for failure.

It is perfectly OK to have multiple files open at the same time. Still, you are advised to close files as soon as you are done with them, since each of them consumes a vital system resource: address space. Note that this is not the same as 'memory': a data product is not read in its entirety into memory. A typical opened product file should consume a couple of tens of kilobytes of memory. However, the entire product file is virtually mapped into memory (a well-known technique that provides high-performance I/O access to a file). On a modern 32-bit computer (e.g., an Intel x86-class computer running Linux or Windows), up to 3 gigabytes worth of files may be mapped into memory at any one point in time.

In the BEAT-I IDL interface, you can have up to 100 files open simultaneously. An attempt to open more files will yield an error.

Function BEAT_CLOSE(ULONG64 pf_id): BEAT_ERROR

The BEAT_CLOSE function is used to release a previously-opened product file. Its one argument should be a 64-bit unsigned integer as returned by a previously executed successful BEAT_OPEN call.

Function BEAT_PRODUCT_CLASS(ULONG64 pf_id): STRING

The BEAT_PRODUCT_CLASS function returns a string containing the product class of a product file. It will be "HDF4" for HDF4 and netCDF files and "HDF5" for HDF5 files.

Function BEAT_PRODUCT_TYPE(ULONG64 pf_id): STRING

The BEAT_PRODUCT_TYPE function returns a string containing the product type of a product file. Note that HDF files don't have a product type (within the context of BEAT) and will therefore return an empty string.

Function BEAT_PRODUCT_VERSION(ULONG64 pf_id): INT

The BEAT_PRODUCT_VERSION function returns an integer denoting the product version, maintained by BEAT to be able to distinguish between different major versions of product specifications. The first supported version will yield 0, the next version will yield 1, and so on. Note that HDF files don't have a product type (within the context of BEAT) and will therefore return a version number -1.

Function BEAT_FETCH(...)

The BEAT_FETCH function is used to fetch data from somewhere in an open product-file and turn it into an IDL variable.

BEAT_FETCH: the first argument

Several forms of BEAT_FETCH exist that are distinguished by the first argument argument. Depending on the form of the call, BEAT_FETCH will try to read data from different header-parts or datasets in the product files. This is summarized below.

call	Fetch-root
BEAT_FETCH(pf_id)	Entire product
BEAT_FETCH(pf_id, [, what, ...])	Data (from product root)
BEAT_FETCH(data_handle [, what, ...])	Data described by handle

As noted before, in some cases the BEAT_FETCH call may return an array of BEAT_DATAHANDLE structures. The last form of the BEAT_FETCH call allows you to traverse such an array and fetch the data described by each of the data-handles in turn.

BEAT_FETCH: data-item specification arguments

After the first argument you can specify more arguments that tell BEAT_FETCH to retrieve only a specific part of the data pointed to by the first argument. It is possible to have more than one such 'specification argument' in case of nested data-types.

Three forms of 'specification arguments' exist:

• If the element is a record, it is possible to fetch a specific field of the record by providing the name of the field as a string argument.
• If the element is an array, it is possible to fetch a specific element of the record by providing the array indices as an integer vector.
• If the element is a one-dimensional array, it is possible to simply specify an integer value; this is equivalent to specifying '[i]' (a 1-element array-index vector).

An example will help to understand all this. Suppose we have a product that contains a dataset called 'MEASUREMENTS', having 1000 records. Each of these records contains a field called 'samples' that is a 2-dimensional 101x101 array of doubles.

Now suppose we want to fetch the central 'sample' contained in record #10. The following BEAT_FETCH would do the trick:

result = BEAT_FETCH(pf_id, 'MEASUREMENTS', [10], 'samples', [50,50])

Alternatively, since the '[10]' specifies the index of a one-dimensional array, we could also write

result = BEAT_FETCH(pf, 'MEASUREMENTS', 10, 'samples', [50,50])

For the BEAT_FETCH and BEAT_FETCH_DATAHANDLE functions (but not for the other BEAT-I IDL functions that use data-item specification arguments) there is an additional feature. If you provide a -1 for one or more of the dimensions of an array you will fetch all elements in this dimension. For example, with

result = BEAT_FETCH(pf, 'MEASUREMENTS', -1, 'samples', [50,50])

you can fetch all central samples of all measurements into a single array. Similarly, you can use

result = BEAT_FETCH(pf, 'MEASUREMENTS', 0, 'samples', [-1,0])

to fetch the first samples column of the first measurement.

The -1 parameter may only be used for one array in a fetch statement (e.g. BEAT_FETCH(pf, 'MEASUREMENTS', 0, 'samples', [-1,-1]) is allowed, but BEAT_FETCH(pf, 'MEASUREMENTS', -1, 'samples', [-1,0]) is not).

BEAT_FETCH return type

The IDL type of the BEAT_FETCH return-value depends on the BEAT data dictionary type of the data item being fetched. Three cases are distinguished:

Return types for simple (non-composite) types

If the data item has a basic type the return type is determined according to the following table.

BEAT read type	IDL Type	remark
int8	INT
uint8	BYTE	a BYTE is unsigned in IDL
int16	INT
uint16	UINT
int32	LONG
uint32	ULONG
int64	LONG64
uint64	ULONG64
float	FLOAT
double	DOUBLE
char	STRING
string	STRING
bytes	BYTE	This will actually translate into an array of type BYTE

BEAT_FETCH return types for basic data types

Return types for Record types

Record types will be returned as anonymous (unnamed) IDL structures, where each field will correspond to a field in the original data set. If a field has the hidden property and if the FilterRecordFields option is set then the field will not be copied into the IDL structure.

For basic types, the IDL field type will be determined according to the 'basic data types' table given above. Fields that are themselves records are expanded as 'nested records'. Fields that are themselves arrays are expanded as 'array types', as discussed below.

If a record contains no fields (i.e. an empty record) then BEAT will return a BEAT_NO_DATA named structure instead of an empty anonymous struct.

Return types for Array types

Array types will be returned as multi-dimensional IDL arrays, with the IDL array base-type derived from the data dictionary base-type according to the 'basic data types' table given above. There are, however, some complications that IDL programmers need to be aware of when the data dictionary base type itself is either an array or a record.

For array and record base-types (i.e., 'array of array', 'array of record'), there is, in general, no equivalent type in IDL. As explained in the section about the BEAT_DATAHANDLE structure above, both these cases will yield an 'array of BEAT_DATAHANDLE' instead.

If an array is empty (i.e. the size of one of its dimensions is 0) then BEAT will return a BEAT_NO_DATA named structure instead of a multi-dimensional IDL array.

Function BEAT_FETCH_DATAHANDLE(...): BEAT_DATAHANDLE

This function takes the same kind of argument as BEAT_FETCH; however, instead of returning the specified data from the product, it always returns a datahandle to the specified data (even if it would be easily representable in IDL).

Function BEAT_ATTRIBUTES(...): INT-array

This function takes the same kind of argument as BEAT_FETCH; however, instead of returning the specified data from the product, it returns a record containing all attributes of the specified data item. The record can be empty if no attributes are available.

Function BEAT_FIELDAVAILABLE(...): INT

This function takes the same kind of argument as BEAT_FETCH; however, instead of returning the specified data from the product, it returns the availabillity of a record field. It returns 1 if it is available and 0 if it is not.

If the arguments to BEAT_FIELDAVAILABLE do not point to a record, a BEAT_ERROR structure is returned.

Function BEAT_FIELDCOUNT(...): INT

This function takes the same kind of argument as BEAT_FETCH; however, instead of returning the specified data from the product, it returns the number of fields in a record.

If the arguments to BEAT_FIELDCOUNT do not point to a record, a BEAT_ERROR structure is returned.

Function BEAT_FIELDNAMES(...): STRING-array

This function takes the same kind of argument as BEAT_FETCH; however, instead of returning the specified data from the product, it returns the field-names of the fields in a record.

If the arguments to BEAT_FIELDNAMES do not point to a record, a BEAT_ERROR structure is returned.

Function BEAT_SIZE(...): INT-array

This function takes the same kind of argument as BEAT_FETCH; however, instead of returning the specified data from the product, it returns the actual dimensionality of a multi-dimensional array.

If the arguments to BEAT_SIZE do not point to an array, a BEAT_ERROR structure is returned.

Function BEAT_DESCRIPTION(...): STRING

This function takes the same kind of argument as BEAT_FETCH; however, instead of returning the specified data from the product, it returns the description of a data item.

Function BEAT_UNIT(...): STRING

This function takes the same kind of argument as BEAT_FETCH; however, instead of returning the specified data from the product, it returns the description of a data item.

Function BEAT_TIME_TO_STRING(numerical argument): STRING

BEAT by default stores time values as a double denoting the number of seconds since Midnight, January 1^st, 2000 UTC.

The BEAT_TIME_TO_STRING function takes a DOUBLE-encoded time value and converts it to a human-readable string of the form "01-JAN-2000 00:00:00.000000"

Function BEAT_IS_NO_DATA(...): INT

The BEAT_IS_NO_DATA function takes one argument; it returns a value of TRUE (1) if and only if the argument is a BEAT_EMPTY_ARRAY structure.

• its ERRNO field is not equal to zero

Function BEAT_IS_ERROR(...): INT

The BEAT_IS_ERROR function takes one argument; it returns a value of TRUE (1) if and only if

• the argument is a BEAT_ERROR structure
• its ERRNO field is not equal to zero

If either of these is not true, BEAT_IS_ERROR returns FALSE (0).

Using BEAT_IS_ERROR, you can make your BEAT-I IDL programs robust (i.e., resistant to failures). If you test the result of any BEAT-I IDL call with BEAT_IS_ERROR, you should be able to intercept any error.

Procedure BEAT_UNLOAD

The BEAT_UNLOAD procedure will close all opened product files and unload the BEAT data dictionary from memory.

The BEAT data dictionary is loaded on-demand if any BEAT function that needs it is invoked.

This function may be (slightly) useful on systems with little memory. You could open a product file, fetch the important data, close the file, and then unload the data dictionary to free extra memory before starting data-processing. Other than that, this function is of little practical use.