Submission of Data Procedure
Any submission of data to the AeroCom database requires to follow the following steps:
- format data to agreed netCDF htap standards, cf compliant
- adjust filenames
- check format with cf compliance checker, correct and check again
- obtain account on htap server to access aerocom-users server
- transfer data to the aerocom-users.met.no at the Norwegian Meteorological Institute
- notify metno that data have been submitted
- update errors on submission log on htap wiki page
1) format data
Note: these commands cover the elder phase 3 naming scheme, but should nevertheless still usefull
The AeroCom standard is described in the section “submitting to the AeroCom database”.
https://wiki.met.no/aerocom/data_submission
Below simple nco commands are listed which can be of help to reformat.
A public set of format correction tools is under development and can be accessed here:
https://github.com/metno/aerocom-data-correction-tools
How to remind me of nco commands
Just type the command without arguments on a command line
How to rename variables
ncrename -v oldvar,newvar in.nc
How to add an attribute
ncatted -a experiment,newvar,a,c,"BASE" in.nc
Split files per variable
cdo splitname in.nc
rename dimension
ncrename -O -d LATITUDE,y temp.nc
sums
ncap2 -O -s "LOAD_BCPOM[time,x,y]=LOAD_BC+LOAD_POM" ${file} ${file}
compute vertical integrals
rm dummy.nc
ncwa -O -v AERH2O3D_AER -a z -y ttl ${file} dummy.nc
ncrename -O -v AERH2O3D_AER,AERH2O_AER dummy.nc
ncks -F -A -v AERH2O_AER dummy.nc ${file}
add variable to file
ncks -F -A -v AIRE temp.nc ${file}
make the time dimension unlimited
ncecat -O $file $file
ncpdq -O -a time,record $file $file
ncwa -O -a record $file $file
ncpdq -O -a time,y,x $file $file
turn z
ncpdq -O -a -z $file $file
create yearly mean file
from 12 months in one file
orgfile=MM_${model}.an${an}.mALLYEAR_an.nc
ncwa -F -O -a time ${file} ${orgfile}
calculate yearly average
fom 12 files each containing 1 month
ncra -F -O modis_2000??.nc ${yearfile}
create record dimension
ncecat -O $file $file
delete useless record dimension
ncwa -O -a record $file $file
turn lat to have right order
with interpolated files
ncpdq -O -a "-y" $file $file
Shift left and right hemispheres
file1=aire_1.nc
ncks -h -F -O -d x,1,180 $file $file1
file2=aire_2.nc
ncks -h -F -O -d x,181,360 $file $file2
ncap -O -s "x[x]=x-360" $file2 $file2 ncrcat -O $file2 $file1 $file
\rm $file1 $file2
Similar reorder of longitude
ncecat -O $file $file
ncpdq -O -a x,record $file $file
ncwa -O -a record $file $file file1=DD_${model}.an${AN}.mALLYEAR_1.nc
ncks -F -O -d x,1,180 $file $file1file2=DD_${model}.an${AN}.mALLYEAR_2.nc
ncks -F -O -d x,181,360 $file $file2
ncap -O -s "x[x]=x-360" $file2 $file2
ncrcat -O $file2 $file1 $file
\rm $file1 $file2 ncpdq -O -a time,y,x $file $file
missing values
ncatted -O -a missing_value,$var,o,f,0 ${file}
ncatted -O -a missing_value,$var,o,f,-9999 ${file}
actually better use attribute _FillValue (see http://nco.sourceforge.net)
compute monthly mean files
for IMTH in ${MTH[*]} ; do
file=DD_${model}.an${AN}.m${IMTH}.nc
filemean=DD_${model}.an${AN}.m${IMTH}_m.nc
ncra -F -O ${file} ${filemean}
done
delete attributes
ncatted -O -a ,,d,, interdummy.nc
ncatted -h -a ,global,d,, interdummy.nc
ncatted -h -a missing_value,,d,, interdummy.nc
2) adjust filenames
The obligatory filename for using the cf checker and the submission of data is:
aerocom3_<ModelName>-<MeteoConfigSpecifier>_<ExperimentName>-<PerturbationName>_<VariableName>_<VerticalCoordinateType>_<Period>_<Frequency>.nc
Details about new file naming can be found here: AeroCom experiments diagnostics sheets 28.2.2019
Be aware of updates !
versions will have a date attached.
Examples:
- aerocom3_GOCARTv7.2_AP3-CTRL_od550aer_Column_2010_monthly.nc
- aerocom3_GISS-MATRIX-met2010_AP3-ACRI-AFRICA_od550aer_Column_2010_daily.nc
- aerocom3_GOCARTv7.2-met2010_AP3-CTRL-od550aer_Column_1850_monthly.nc
- aerocom3_GISS-MATRIX-fSST_AP3-CTRL-od550aer_Column_1850_monthly.nc
OLD (aerocom phase3)
aerocom3_<ModelName>_<ExperimentName>_<VariableName>_<VerticalCoordinateType>_<Period>_<Frequency>.nc
OLD (aerocom phase 2) aerocom naming convention is also fine for now:
aerocom_<ModelName>_<ExperimentName>_<VariableName>_<Period>_<Frequency>.nc
- <ModelName> ⇒ can be chosen such that Model Name, Model version and possibly the institution can be identified. No underscores (_) are allowed in <!ModelName>, use (-) instead. Restrict <ModelName> to max 20 characters.
- <MeteoConfigSpecifiers> ⇒ “met2010” (default, nudged to 2010 meteo) / “fSST” (fixed 2010 SST monthly fields, not nudged) / “CPL” (coupled AOGCM model, not nudged) A MeteoSpecifier has to be present
- <ExperimentName> ⇒ see Experiment Specifications
- <PerturbationName> see experiment descriptions on aerocom wiki, if required for the experiment, use exact case sensitive name from description
- <VariableName> ⇒ see output variable names in excel sheet
- <VerticalCoordinateType> ⇒ “Surface”, “Column”, “ModelLevel”, “SurfaceAtStations”, “ModelLevelAtStations”
- <Period> ⇒ eg “2008”, “2010”
- <Frequency> ⇒ “timeinvariant”, “hourly”, “3hourly”, “daily”, “monthly”, “sat1000”, “sat1330”, “sat2200”, “sat0130”
3) check format
Typical files must be checked prior to submission via a cf-compliance-checker. cfconventions.org maintains a list of working and frequently updated CF checkers. Some come as a script, and to others one can upload a file.
For the upload checker you want to check a typical but small file. To achieve this, you can cut out the first time steps with the nco command: ncks -O -d time,0,10 sourcefile.nc filetest.nc
Per default, a check shall comprise at least one file of each VerticalCoordinateType and each Frequency and each experiment, best varying the variables!
4) obtain account
An account at met.no is needed to submit data and give users access via ssh to the aerocom-users.met.no server. Please contact anna.benedictow@met.no (and in case of missing response also michaels at met.no) for password and user name. You will then be notified further about the procedure. A public ssh-key has to be sent to Anna to activate the account.
5) transfer data
After obtaining an account, data can then be uploaded to the aerocom user server aerocom-users.met.no:
-
Create a directory with an understandable, unique name under /media/scratch/incoming/
Use the model name as used for file names.
Put files in that directory. Overall size limit of ca 10 TB (limit for all users)
Inform Anna and Michael on upload, purpose and experiment by email !
Large submissions (> 500 GB) may require extra attention (use rsync, sftp). They might require prior coordination with Met.No, please send an e-mail to anna.benedictow@met.no and michael.schulz@met.no, in particular in times of simultaneous heavy upload activity.
If correct in format and with correct file names, results uploaded here will be processed over night and appear after a day as image catalogue on https://aerocom-classic.met.no/cgi-bin/aerocom/surfobs_annualrs.pl?PROJECT=AEROCOM&MODELLIST=AEROCOM-Phase-III
6) notify met.no
When files are uploaded send email to anna.benedictow@met.no and jan.griesfeller@met.no and michael.schulz@met.no (AND coordinator of project), so that the files can be transferred from aerocom-users to the database. When the data are included in the database the files appear on aerocom-users.met.no under /metno/aerocom-users-database/PROJECT(AEROCOM-PHASE-III)/<ModelName>_<exp>/.
After the data has been transferred to the database, it will be deleted form the aerocom-users' scratch area.
7) updates
In case of updates of files, the model version identifier should also contain information on the resubmission of model results. Additional files which do not change an earlier submission may contain the same version number as an older submission. Corrections and a model rerun should be accompanied by a new model version identifier. Old still valid files may be merged upon request with new data. Old versions of model submissions will be temporally moved to a separate directory by Met.No, and where possible, deleted after some time. An exception will be made for versions that have been used in publications. Modellers are responsible for communication of major updates to their models (e-mail and log-files, see below). Model analysers are responsible for frequently verifying changes in model submissions (e.g. before submitting a paper).