■Error Conditions
The same error occurs when the amount of data read by DataSpider Servista exceeds the set heap memory.
■Probable cause
It may be due to one of the following causes:
- A memory shortage error is occurring due to the increased number of data being processed.
- A known issue related to the file system
Please refer to the following for workaround:
■Workaround
- A memory shortage error is occurring due to the increased number of data being processed.
*If you expand the heap area, the memory capacity used by the DataSpiderServer process will increase. For this reason, it is recommended to start with the solution (1), which has less influence on the system.
- Use of mass data processing, parallel streaming processing (PSP)
A large-volume data processing saves read data in a temporary file and reduces the amount of data held in the memory. In addition, the parallel streaming processing (PSP) divides data to be processed into blocks and processes the reading, conversion, and writing in separate threads to reduce the amount of data held in the memory.
Therefore, using one of the above methods can reduce the memory usage.
- The heap area expansion
By expanding the heap area of a memory that reads data, you can hold more memory.
A heap area can be expanded from the key related to memory settings of the DataSpider configuration file (DataSpiderServer.lax). Please refer to the help of "Reference Information" for details.
- A known issue related to the file system
Please check if one of the following known problems applies, and if so, confirm that the problem is resolved by the "Workaround". The following is described in the known issues in Version 3.1, but it also occurs in versions prior to 3.0
In a repository DB environment, when writing a large volume of data to a file on a database file system, an "OutOfMemoryError" error may occur.
This error occurs even if PSP or mass data processing is enabled.
Workaround: Please write to a file on a local file system or an extended local file system.
If the above method does not resolve the error, please contact the support center.
Reference information
- A large amount of data processing
- Parallel Streaming Processing