Sometimes, when dealing with big data, we are faced with the problem that it takes a long additional time after the measurement to export the data to the wanted format. This application note shows a solution by starting the export already in parallel to the measurement, over a network on a dedicated Export PC.
- DataManager2 Plugin
- Sequence file (SearchFiles_Export_Move.dxd)
- Batch files (WaitForDrive.bat, Move_file.bat)
The Measurement PC is permanently storing multi-files, e.g. each 100 MB into the “Data” folder. The DataManager2 plugin is needed to copy the file in parallel (while DEWESoft is already storing the next multifile without a gap) to a shared network folder. This ensures on the network folder are only finished (closed) files! – It must be a mountable network drive, e.g. both computers are in the same workgroup.
On the Export PC, a DEWESoft sequence is running. It permanently checks the network folder for new (finished!) files and exports them.
Basically, a stable network is required, but to handle network interruptions, a batch file is running (WaitForDrive.bat), which holds the sequence until a firm connection is available. Then it moves the files one after the other over the network to the Export PC (folder: 1_Downloading). Depending on the file size the copy process can take a while, DEWESoft waits until the application is finished, then files are opened and exported (e.g. to the C:\DEWESoft\Exports folder), and then moved to the 2_Exported folder. So only files to be processed stay in the 1_Downloading folder.
The user can also extend the sequence to delete the once processed files because the original data anyway stays on the Measurement PC.
Measurement PC – Settings:
In Channel setup, the Multifile option must be enabled. There are various options.
The DataManager Plugin copies in the background while DEWESoft is still storing data. You have also the option to remove files that are older than XX days, to empty your disk space in a long-term measurement task.
Export PC – Settings:
The Data_network folder must be mapped as a drive (e.g. Z:\)
Hint: One idea for optimization: It may be better to check the local folder first, then the network. So the export of the local folder can be finished if the network connection is lost.
Screenshots during operation:
Please create exactly the same folders on both computers, as shown below. Otherwise, you need to change the paths in the sequence.
Depending on the sampling rate, the number of channels, etc. the time it takes to export the data can take shorter or longer than the time to acquire new files. When the exporting goes faster, the folder will be empty and the sequence will show “Waiting for new files…”. Once there is new data detected, it will automatically continue exporting without user interaction.
When the network connection is lost, the batch file will hold the sequence until the drive is found again. This should cover most of the network interruptions. It then automatically continues copying and exporting without user interaction.
Attached below it is possible to find a zip archive where are stored the bat files and setup file with the sequence.