Published Feb 09, 2017 |
Optimizing LOG file memory allocation in IGSS
News from IGSS Development
Subtitle text: How to set up your IGSS installation to optimize your LOG data file memory allocation and hard disk storage and a description of some potential consequences of sub-optimal setup.
Correct management of the data files is in many ways critical for your IGSS installation, its performance and subsequent data backup procedures and storage management. This article describes how to optimize LOG file memory allocation and describes several issues that may occur if the data file parameters are not defined optimally.
IGSS registers process data in two ways:
- Reduced or aggregated process saved data in Base Class (BCL) files
- Raw process data saved directly in data (LOG) files.
Base Class data is always calculated or processed in some fashion, either calculating an average or sum for a set period (called the Base interval) or only registering the maximum, minimum or actual values while LOG data is inserted directly into the data files without any processing.
There are several options regarding when to register process value changes in LOG files as well whether to log data or not which must defined in order to save process data in the LOG files. All data logging options are defined on the Data Management tab on the Object Properties form in the Definition module for each IGSS object in the configuration.
LOG data from process components (objects) are stored in LOG files while Base Class Data is stored in BCL files. Each LOG file contains data from all objects in the configuration in a given period. By default, LOG and BCL files are stored in the Reports folder in the configuration but can be placed in other folders, backed up and copied as other normal files in Microsoft Windows.
LOG file size settings
You must define how the IGSS configuration is to manage the data files that are created during normal operations. This is done in the Files tab on the System Configuration form where you can set whether LOG files are to be retained, how long they are to be retained as well as how much disk space is to be allocated for each LOG file and how many “old” LOG files (LOG files collecting and containing data older than 1 hour) are to be retained in the memory for later updating.
By default, LOG files are retained for 24 hours and then deleted and the starting LOG file size is set to 1 KB. These default settings must of course be changed to reflect customer requirements for actual operations. The maximum number of open LOG files in the memory is set to 26.
The data logging process
When the IGSS configuration starts, RAM memory blocks are reserved for LOG data storage during process data gathering (i.e. normal process monitoring). The initial size of the reserved memory block is determined by the value in the Initial Size field for the LOG files.
If the Initial size field value is 100 kb, then a 100kb large block of memory is reserved in the RAM memory for LOG data.
During operations, process data from the objects in IGSS is continuously stored in the reserved RAM memory blocks, respecting the data management parameters set up for the individual objects. After one hour of operations, the contents of the reserved RAM memory block is directly transferred to the hard drive, at the location defined to contain saved LOG files – typically the Reports folder. The LOG data is transferred “as is” from the RAM memory to hard drive and is not compressed.
If the reserved memory block size is larger than the actual LOG data size, the entire reserved memory block will still be transferred, potentially resulting in larger than needed LOG files in the Reports folder.
The Initial Size field value was 100kb but the actual LOG data collected only took up 46kb, the LOG data file for that hour in the Reports folder will nonetheless be 100kb of non-compressed LOG data and not 46kb of actual LOG data. This will lead to larger than necessary LOG data files on the disk.
If the reserved memory block size is smaller than the actual LOG data size, the reserved memory block will be expanded to contain the additional incoming LOG data.
In the best of cases, the reserved memory block can be freely expanded as depicted in the upper RAM memory bar below but in many cases (the lower RAM memory bar), the reserved memory block cannot be expanded sufficiently, resulting in many smaller RAM memory blocks being used to contain the incoming LOG data. This will result in RAM memory fragmentation if IGSS is run continuously over a long period of time.
Memory fragmentation will affect performance negatively as the system attempts to create, write and update values spread out over an increasingly larger span of memory. Additionally, since the contents of the reserved RAM memory block will still be transferred to a LOG file in the Reports folder every hour, transfer speed will be adversely affected as the system will attempt to read from potentially spread out areas in the RAM while transferring.
The LOG file in the Reports folder will however, accurately reflect the actual size of the collected LOG data and will not be oversized with regards to the actual data.
The Initial Size field value was 100kb but the actual LOG data collected was 250kb. The LOG data file for that hour in the Reports folder will be 250kb of non-compressed LOG data and not 100kb.
Depending on the layout of the RAM memory of the server, the RAM memory blocks used to contain the extra LOG data might be placed in extension of the initially reserved memory block or more or less dispersed throughout the entire RAM memory, potentially leading to RAM memory fragmentation, especially of IGSS is run for long periods of time without stopping the configuration.
After the LOG data is transferred from the RAM memory to the hard drive, IGSS will continue to register LOG data in the same reserved RAM memory block for the next hour of operations if possible or a reserve a new RAM memory block if the previous block no longer is fully accessible.
Multiple open LOG files
You can have multiple LOG files open in the RAM memory at the same time in order to update older LOG files with new values.
LOG files must be loaded into the RAM memory in their entirety if IGSS needs to update or write to the LOG files. This may be relevant if IGSS receives updates to earlier LOG files, maybe once every 6 hours or once every day from remote stations over a dial-up connection.
This means LOG files older than one hour may still be open in the RAM memory for later updating and will of course occupy correspondingly large portions of the RAM memory as well as potentially lead to memory fragmentation as the many open LOG files reduce the availability of free memory areas for LOG data storage.
You can reduce this memory usage by limiting the maximum number of open LOG files in the memory in the Max files in the DC memory field to reflect the actual number of open LOG files required by the system. The Max files in the DC memory field is found in the Files tab on the System Configuration form.
The default setting is 26 and the minimum number of open LOG files in the memory is 1. There is no maximum number of open LOG files, but the amount of available free RAM memory is a natural limit to the number open LOG files.
Potential LOG memory issues
Incompatibilities between the reserved RAM memory and the actual LOG data file size as well as number of open LOG files may result in several LOG memory issues – either larger than needed LOG data files or RAM memory fragmentation.
A special case of LOG memory issues occur when updating an IGSS installation to IGSS12 or more, where the LOG data parameters are not adjusted to reflect the new LOG files in IGSS12 or more.
Large LOG data files
If the LOG data files in the Reports folder are larger than necessary - the reserved memory block size is larger than the actual LOG data size, (see above) – you can help reduce the size of future LOG data files by first gaining an estimate of the size of a LOG data file during operations in the Reports folder and then adjust the reserved RAM memory for LOG files in the future.
In the System Configuration form > Files tab, click the Find Max button to obtain an estimate of the maximum size of the LOG files during operations. The size of the largest LOG data file currently on the disk is displayed in the Current max. field. This value will, if the system has been running for a few days with the initial size of 1kb, accurately reflect the maximum expected size of the LOG file during normal operations.
The value of the Current max. field can then be transferred to the Initial Size field and the configuration restarted in order to deploy the changes and resume operations.
As the Find Max button finds the size of the largest LOG file, you should double-check the reported size to ensure the size is representative of normal IGSS data operations and not the result of a data anomaly. You should also compare the sizes of other LOG files in the Reports folder to ensure the Current Max. value is representative of normal operations.
You cannot compress LOG files that are open in the RAM memory or saved in the Reports folder. They are fixed in size.
Prevent memory fragmentation
To help avoid RAM memory fragmentation when running IGSS continuously, define the initial size of the LOG file in the RAM memory as accurately as possible.
This will reserve an area in the RAM memory for LOG data which hopefully is just the right size to contain actual LOG data and not too large, causing larger LOG files on the disk and not too small, causing potential fragmentation.
In previous versions of IGSS, one LOG file was used to collect and contain process data for one hour, regardless of the number of objects in the configuration, resulting in 24 LOG files per day.
From IGSS12, 16 LOG files are used to collect and contain process data for one hour if the IGSS configuration contains more than 2,000 objects, resulting in 384 LOG files per day. For IGSS configurations with less than 2,000 objects, one LOG data file per hour is generated.
If you upgrade your existing IGSS installation (and thereby also upgrade your configuration) from an IGSS11 or less to IGSS12 or more, you must also revisit your LOG file settings in the System Configuration form > Files tab > Data size group – especially the value in the Initial Size field for LOG files.
You can divide the value in the Initial Size field by 16 to obtain a rough estimate for the new LOG data file size or you can obtain and utilize a new estimate for the reserved RAM memory for LOG data. See Large LOG data files above.
If you do not adjust the LOG data file parameters, the new updated configuration will now reserve not one large LOG file in the RAM memory, but 16 large LOG files!
If the reserved memory blocks are too large to begin with, this will not only increase the risk of RAM memory fragmentation but will also dramatically increase the LOG files disk usage in the Report folder.
The Initial Size field value was 100kb prior to an upgrade from IGSS11 to IGSS12. After the upgrade, 16 areas in the RAM memory will be reserved by IGSS instead of one, each reserved memory block will be about 100kb and 16 LOG data files will thereafter be created in the Reports folder, each at least 100kb.
If there is little free RAM memory available, RAM fragmentation will also occur. If free RAM memory is insufficient to manage the 15 additional memory reservations, the performance of the entire machine will suffer negatively as well.
Example of error text in IGSS log
If the free RAM memory turns out to be completely insufficient to manage the additional memory reservations from IGSS, a write error will occur with the following text in the IGSS error log:
|ERROR C1010323 GENERAL -1 4996 4888 .\logfile.cpp (152) Allocation of memory failed, memory size requested 67108864, rc = Not enough storage is available to process this command. (0008) |