http://www.wiki.mohid.com/index.php?action=history&feed=atom&title=Mercator_Automatic_Data_AcquisitionMercator Automatic Data Acquisition - Revision history2026-04-04T21:18:04ZRevision history for this page on the wikiMediaWiki 1.28.0http://www.wiki.mohid.com/index.php?title=Mercator_Automatic_Data_Acquisition&diff=489&oldid=prevGuillaume: 1 revision2008-12-03T10:27:16Z<p>1 revision</p>
<table class="diff diff-contentalign-left" data-mw="interface">
<tr style='vertical-align: top;' lang='en'>
<td colspan='1' style="background-color: white; color:black; text-align: center;">← Older revision</td>
<td colspan='1' style="background-color: white; color:black; text-align: center;">Revision as of 10:27, 3 December 2008</td>
</tr><tr><td colspan='2' style='text-align: center;' lang='en'><div class="mw-diff-empty">(No difference)</div>
</td></tr></table>Guillaumehttp://www.wiki.mohid.com/index.php?title=Mercator_Automatic_Data_Acquisition&diff=488&oldid=prev192.168.20.111 at 14:28, 27 October 20062006-10-27T14:28:05Z<p></p>
<p><b>New page</b></p><div>This article describes the automatic data acquisition system built to get large-scale circulation model gridded data from Mercator-Océan ftp server. [[Automatic Data Interpolation|Another article]] is written to describe the interpolation automated task for a given model.<br />
<br />
==Objective==<br />
Every week Mercator-Océan releases a set of 63 netcdf gzipped files to their ftp site in a timestamped folder. The goal is to get these files, store them in the local network and perform data manipulation on them.<br />
<br />
The catch is that sometimes some files are missing at the ftp site, sometimes the connection breaks down for some reason. Hence the data acquisition system must be "intelligent" to "know" when he has fetched all the files and when he must try again a new ftp connection to fetch the missing files.<br />
<br />
==Technological base==<br />
Perl and shell scripting, using ftp and smtp perl modules. Mohid tools to convert from netcdf to HDF5, and to interpolate. Scheduler or crontab to agend the task weekly.<br />
<br />
==Control flow diagram==<br />
[[Image:perl_proj.jpg|center|450px|thumb|Control Flow Diagram: Yellow boxes are perl programs or shell commands, purple boxes are input/output files, blue triangles are flow control elements, dark edges show the program flow, white edges show the input/output from/to files and white losangle edges show appending output to files.]]<br />
Aside is the control flow diagram and below is<br />
the full batch file describing all the operations:<br />
REM CalcDate.bat<br />
:CALC<br />
perl CalculateCurrentDate.pl > _date.tmp<br />
IF %ERRORLEVEL% NEQ 0 GOTO CALC<br />
IF ERRORLEVEL 0 GOTO CREATE<br />
<br />
REM ftp_get.bat<br />
:CREATE<br />
more < _date.tmp | perl Create_ftp_address.pl > _ftp_address.tmp<br />
IF %ERRORLEVEL% NEQ 0 GOTO CREATE<br />
IF ERRORLEVEL 0 GOTO GET<br />
<br />
:GET<br />
more < _ftp_address.tmp | perl ftp_get.pl<br />
IF %ERRORLEVEL% NEQ 0 GOTO CREATE<br />
IF ERRORLEVEL 0 GOTO CHECKFILES<br />
<br />
:CHECKFILES<br />
more < _ftp_address.tmp > _Extract.tmp<br />
echo (ist_meteog-mercatorPsy2v2r1v_R\d{8})>> _Extract.tmp<br />
<br />
more < _Extract.tmp | perl ExtractPattern.pl > _Count.tmp<br />
echo .nc.gz>> _Count.tmp<br />
<br />
more < _Count.tmp | perl CountFiles.pl | Checkfiles.pl<br />
IF %ERRORLEVEL% NEQ 0 GOTO CREATE<br />
IF ERRORLEVEL 0 GOTO CONVERT<br />
<br />
REM convert.bat<br />
:CONVERT<br />
more < _date.tmp | perl Convert.pl<br />
IF %ERRORLEVEL% NEQ 0 GOTO CONVERT<br />
IF ERRORLEVEL 0 GOTO CHECK<br />
<br />
REM interpol.bat<br />
REM Still misses ...<br />
<br />
REM check.bat<br />
:CHECK<br />
more < _date.tmp | perl check.pl<br />
IF %ERRORLEVEL% NEQ 0 GOTO BAD<br />
IF ERRORLEVEL 0 GOTO GOOD<br />
<br />
:BAD<br />
more mail_Bad.txt | perl smtp.pl<br />
GOTO END<br />
<br />
:GOOD<br />
more mail_Ok.txt | perl smtp.pl<br />
GOTO END <br />
<br />
:END<br />
del *.tmp<br />
The command names are pretty self-explaining. The perl files are best described at their [http://www.mohid.com/hydrogroup/codecomments.xml repository] where the [[Code Comment Method|commented code]] is published. Some care was taken so most of the little perl programs could be re-used generically.<br />
<br />
==Comments==<br />
===20061025===<br />
This week two new problems rose:<br />
#3 intruder files were in the ftp download folder. This makes a number of 66 downloaded instead of 63. Thus the automatic data acquisition system ends up looping endlessly until 63 arrives.<br />
#The other problem is that when the ftp connection fails, the program re-hooks the ftp and continues to download the next non-existent file. The problem is that a zero bytes files exists and isn't downloaded again.<br />
<br />
The solutions are:<br />
#Give in the ftp address file a regexp pattern for the ftp-get program to match, so it will download only the wanted files.<br />
#Check, besides the existence of the file, its size also. If zero bytes then download.<br />
<br />
[[Category:Technology]]<br />
[[Category:Operational Modeling]]<br />
[[Category:Mercator]]</div>192.168.20.111