Back to Index
METAMOD supports extracting metadata from netCDF files. The metadata extraction is triggered in three different ways.
Which method is used for triggering uploads are decided by the EXTERNAL_REPOSITORY
configuration variables.
If EXTERNAL_REPOSTORY
is set to false
the two first methods can be used.
In this case METAMOD is also responsible for moving the data files to the data repository in addition to extracting the metadata.
If EXTERNAL_REPOSITORY
is set to true
the second method is used.
In this case METAMOD will not move the data files to the data repository.
The data repository is then some other system responsibility.
When a user uploads a new data file to the system using the web interface the following steps are executed.
UPLOAD_DIRECTORY
.
The web application also inserts an "Upload" job into the queue to tell the backend system that a new file is available.
At this point the web application sends a reply back to the client.
For the Upload job,
Metamod::Queue::Worker::Upload
and Metamod::Queue::Job::Upload
is used to perform the job.
The reason for having two modules for doing this work is that Worker module is specific the queue system used,
while the Job module is independent of the actual queue system.
This makes it easier to switch from one queue system to another at a later date.
Metamod::Queue::Job::Upload
has started to execute the job it will leave the heavy lifting to Metamod::UploadHelper
.
UploadHelper will do the following:
MetNo::DigestNc
to extract the metadata from the netCDF files.
This processing in done twice.
Once for all the uploaded at the same time to extract the level 1 metadata for the dataset.
Then DigestNc is used again for each file to extract the level 2 metadata for that file.
DigestNc use Metamod::Dataset
for reading and writing metadata so all data is stored both in the XML metadata repository and the index database.
In case of any error in the processing of the file the UploadHelper will generate a HTML file with a description of all the errors.
Upload via FTP is very similar to upload via web except for the first steps that triggers the metadata extraction.
Instead of using the queue system the FTP upload is triggered by the script upload/scripts/ftp_monitor.pl. This script supports several different modes but the most relevant here is the daemon mode where it is running like a CRON-like job.
When running as a CRON-like job the script will wake up at regular intervals and start looking for files in the FTP_UPLOAD_DIRECTORY
.
When looking for files in the ftp upload directory it will not process all files that it finds.
What files to process is specificed in the ftp_events
file in the WEBRUN_DIRECTORY
.
When a file is found in the FTP directory the metadata extraction continous in the same way for web uploads using the Metamod::UploadHelper
.
Metadata extraction when data is stored in a external repository works differently than the two other methods.
In this case the process is as follows:
upload/script/upload_index.pl
.