Email:  Password:  
  Forgot Password?  Register Now
UCAR > NCAR > CISL
Search:  
  
FAQs | Contact Us | For Staff
ds353.4  Go to Dataset: 
 


SUPPLEMENTAL INFORMATION Latest revision 2007 Jun 08

FORWARD

We thank you for choosing to view this supplementary information, which may prove to be crucial in your work. You should bookmark this page and use it to keep up with any changes that may come along later. These changes are infrequent.

Definitions of raob, pibal and other upper air station observations are provided here.

In 1997 we got an email from Dennis Keyser (NCEP) about matters regarding the transition to BUFR. Please look at this. It is a good example of what can occur with operational data.


The following list of subjects are links to certain critical information.

If you don't take the time to study all the information below, you may experience unexpected frustrations when using the data. Being an operational dataset, it may not be as clean as you may wish or assume.


INTRODUCTION

DS353.4 is maintained in NMC Office Note 29 (ON29) format. Originally NMC decoded the GTS data and reformatted it directly to the ON29 format for use in the NCEP models. Beginning sometime in the late 1990s, NCEP put it directly into BUFR format, and then, for testing, converted this back to ON29. When they stopped this conversion in October1999, DSS began doing it using NCEP's software.

NCEP retrieves the reports off of the GTS and collects them in files which have a synoptic date/time stamp, which corresponds to the analysis times of the NCEP models. This is an operational dataset, meaning that when stations fail to report over the GTS, or there are other data losses, neither NCEP nor DSS can recover the data.

The ongoing source of the data for the BUFR to ON29 conversion is NCEP's "dump" BUFR data files. NCEP has applied some interactive quality control, removed duplicate reports, and merged upper-air "parts". See Observational Data Processing at NCEP.

Be aware that the ON29 format does not provide for as much information as does the BUFR format, Layers 1&2 and Layer 3. (BUFR documents obtained from here, also see this information). The primary information is there, but some new data types carried in BUFR do not appear in ON29. Finally, BUFR also carries the original raw reports which NCEP collected from the GTS - ON29 does not.

NCEP uses their "dump" BUFR files to prepare the PREPBUFR files which are directly fed to the models. This stage involves the addition of background ("first guess") data, observation error information and automated quality control. DSS does not archive PREPBUFR data in ds353.4. DSS does collect PREPBUFR data which happens to be included in model output datasets (e.g. ds609.2 ).

The GFS forecasts (formerly AVN), which are rushed for aviation needs, use the accumulation available at the analysis time (i.e. early cutoff "dump" BUFR files). The FNL final analyses, as in ds083.2 and ds083.0 use the accumulation through a later cutoff "dump" time. The ds353.4 dataset includes much of what goes into the FNL, but this dataset is not in PREPBUFR format.

Definitions of raob, pibal and other upper air station observations are provided here.

Please refer to the July 2002 A.M.S. B.A.M.S. article (p.1003) about U.S. N.W.S. instrument changes beginning in 1995 and their effects on the radiosonde data. Abstract, Full text in PDF.

NCEP is now using "superobbing" in some of its upper air data processing. See this CIMSS document.

For the curious, a history of U.S. weather data technology is here.

Beginning with the January 2004 data, in order to comply with WMO Resolution 40, DSS has been removing some restricted data. This involves proprietary data from certain commercial sources. Also see the NCEP Restricted Data Information Site

You should notice that the volume of the ds353.4 aircraft and satellite data has increased significantly, by an order of magnitude, since the early 1990s. This trend may continue.


DSS UPDATING

Updates are usually done about three weeks after the end of a month.


USER PROCESSING MATTERS See:
DSS does not have a list of available stations, but DSS does have station receipt summaries here (updated yearly). Please refer to our USAF Station Libraries for station names, locations, elevations, etc. A map of the WMO regions is here Caution: the content of these libraries must not be taken as a list of what you may find in this dataset, they are just cross-references.



DS353.4 SUBSET NOTES

PERIOD OF RECORD AND VOLUME -

The period of record (POR) stated on this dataset's home page is misrepresentative. Please refer to the following subset list for accurate PORs. The POR on the home page is derived by the DSS webmaster from a database using an inflexible procedure. It is one of a very few faults of our modernization. The upgrade is briefly described at the bottom of this document, under GCMD


FILE STRUCTURE - see this .


NCEP'S TIME AND NAMING CONVENTIONS

BASIC - NCEP segregates the reports by dataset type in addition to the synoptic date/time, and calls the group a data file. The name of a data file is analogous to a directory of observation types. For example SFCSHP (surface ship observations) will include ships, moored buoys, drifting buoys, etc. A data file begins with a header block and is terminated by a trailer block. The header block identifies the dataset type (e.g. ADPUPA or SFCSHP) and synoptic date/time (which may also be called the header time). The actual observation times appear in the individual reports. The individual reports do not show the synoptic date/time stamp.

DSS converts NCEP's header blocks to the GATE format. This was originally intended just to support GARP projects in the mid-1970s, but has persisted and become our paradigm.

DATE/TIME STAMP - The collection of observations for a synoptic time begins before the synoptic time, as much as half of the time interval between synoptic times, but it should not be before the cutoff time of the previous data file. The header blocks will often, but not always, show a receipt time which indicates when the file was opened to begin the accumulation of reports. Likewise the cutoff time indicates when the file was closed, terminating the accumulation. DSS has found that these delineators have not been dependable, so users should just ignore them.

The synoptic time, header time or analysis time are all the same and are specified in Coordinated Universal Time (CUT or Z). The individual report times ares also given in UT and are the actual observation times, as reported by the stations. Stations may report observation times which look exactly like the synoptic times, but in practise these may not always be true.

For information about the time convention, see this.

WARNING: Many users have been accepting or mistakenly expecting the synoptic time to be the observation time. If their work is sensitive to the difference, then they should use the observation times. This is a very important consideration because often users will find more than one report from a station within a data file for a synoptic time. Moreover, some reports in a given data file may be duplicated in the previous or following file.

Users must be aware that the actual time of an observation should be obtained from within the individual report. One may also want to adjust the date shown in the header block, because for some 0Z (and even 06Z) data files the report times which show values between 18Z and 24Z actually belong to the previous date. A comparable thing may occur at 18Z, where the report may belong to the next day. Users need to be aware of this. Our latest access software, readupa2.f, allows a user to make the adjustment.

STATION IDENTIFICATION - Generally the SYNOP reports use WMO 5 digit numbers. Many other reports, such as ships, use call signs. So when attempting to extract all the reports made by a station, or stations nearby, one should use a latitude-longitude "window", which will get both the WMO and call sign labeled reports. Such a "window" will obtain neighboring station data that could be used when the desired station data is missing. See the code in readupa2.f. Also see our combined list of U.S.A.F. catalogs.

RADIOSONDE TOP LEVELS - .


DATA VALUES AND OBSERVATION TYPES-Users sometimes believe these are the same, but they are not. Observation type refers to the report type, or platform, such as land station, moving ship, etc.

NCEP'S DATA FILE STRUCTURE

NCEP's file organization (not the format) has varied over the years. We do not alter this. From 1 January 1978 - 31 March 1997 things were consistent. NCEP/NMC sent us this data on weekly tapes which sequenced all the week's data for each dataset type together. In other words, all the observation types for a given synoptic time were not contiguous. We collected these shipments in DS353.1., and then about every quarter we split them out to time series subsets. I.e., we reassemble all the raobs and pibals in one series of files, all the aircraft observations in another series, etc.

Beginning sometime before 1997, perhaps April 1996, NCEP began putting the GTS data directly into the BUFR format. Until October 1999, NCEP used their BUFRON29 conversion program to produce the ON29 formatted files from the BUFR files. This was done initially to verify content, and then for awhile as a service to users while they adapted to BUFR. NCEP ran this conversion on their Crays, and when they abandoned their Crays for a new system, they stopped making this conversion. In February 2000, we acquired NCEP's BUFRON29 in order to continue production of the ON29 format. NCEP has discontinued support for the ON29 format, and for the BUFRON29 software. We have been using BUFRON29 while knowing very little about it - the thing is a large black box.

NCEP made several changes in the spring of 1997, mostly in April. First, NCEP began putting the data in monthly files, inside huge tarfiles. We found it easier to split these directly to the 5 day, 15 day and 30 day files in the subsets, and continue to do so.

In April 1997 NCEP stopped shipping the ADP tapes directly to us. In February 1998 we obtained copies of the tapes which they continued to send to NCDC. Another batch was so obtained to bring us through March 1998, and then NCEP's shipments directly to us resumed. Then in June 2000, tape shipments were terminated in favor of using NCEP's FTP server.

DSS has been collecting the BUFR formatted data in DS353.5 and DS351.0 The operational changes at NCEP between 1996 and 2004 has created ongoing challenges to DSS for extending ds353.4 in a consistent format. We will continue to provide this data in the ON29 format for as long as we can run the BUFRON29 conversion.

THE DATA

GENERAL- This data is not available as a database. We also remind users that the data consists solely of the data which the stations reported over the GTS, and subsequently what NMC/NCEP saved on an operational basis. Dropouts are not unusual, and we can not recover them.

The data are provided in "packed ASCII" flat files containing a series of complete station reports. This has many implications. Being ASCII rather than binary significantly increases the volume. Being synoptically sorted, the expense for extracting a time series of reports from a handful of stations can be surprisingly high.

Our software, such as readupa2.f, opens individual reports and prints recognizable data values. Users who want to see just a few data values, or even just one, such as temperature, must still open each entire individual report to access them. Users will need to modify our access routines to obtain such special subsets.

REPORT COUNTS- Our inventories show report counts by synoptic time and station. The latter are updated annually, and show monthly totals. These may be accessed from the tab bar near the top of the main Web page.

There has been a huge increase in the number of satellite wind reports in the last few years.


CRITICAL INFORMATION ABOUT VARIABLES

QUALITY CONTROL - NCEP does some Quality Control (QC) on the data, and they set some QC flags in the reports to qualify, but not quantify their corrections. Radiosonde data corrections are described in Observational Data Processing at NCEP

Missing data values are usually indicated by a field of 9s.

DATA UNITS - The data units in this dataset involve both English and metric. See ON29 for details.

SEA LEVEL PRESSURE - This dataset has reports which include surface (station) pressure, sea level pressure, or both. From a scientific point of view, stations which report a sea level pressure, but not the observed station pressure, are being irresponsible. This is because it spoils the possibility of applying improved algorithms for the reduction of pressure from station elevation to sea level, among other things. For that matter, methods have changed over time.

The reduction of station surface pressure to sea level pressure is almost a black art. Basic meteorology texts describe theories which involve assumptions about the virtual temperature in an imaginary level between the station and sea level. An example is in the classic text "An Introduction to Theoretical Meteorology" by Seymour Hess (1959), on pages 88 - 90. In practise, the reduction algorithms are empirical, where local stations apply local statistical knowledge of weather and seasonal patterns.

Here are a couple of more recent texts on the subject: Chuang, et al and NOAA/NWS.

It is reasonable to expect to know exactly what was done in this dataset, but we do not have the answer.


WINDS - In ds464.0 we have notes about a wind problem that might also effect the radiosondes. The following is from the ds464.0 "more summary:"
DEWPOINT - The GCMD does not provide a keyword for dew point depression. We are forced by the GCMD rules to show "Dew Point Temperature" although the data values in the reports are actually dew point depression.


OTHER

Background information about the characteristics of this data may be found in An Introduction to Atmospheric and Oceanographic Datasets. There is general basic information in its Introduction.

More specific information may be found in these chapter(s) or section(s): Upper Air Observations.

One should always be aware of Observational Data Problems.

You may wish to refer to Other Sources of Data.

Here are some more upper air data resources (that could not be shown in the related datasets section on the home page):
For current North American charts, please see NCAR RAP Real-Time Weather Data. For access to single radiosonde reports, from a global database for 1973 to the present, see this University of Wyoming site.

GCMD In October 2006, all our dataset home pages were rewritten, and they will be subject to more changes. These actions accommodate a new dependency on a new DSS database and new rules based on the Global Change Master Directory (GCMD), which is the American Coordinating Node of the Committee on Earth Observation Satellites (CEOS) International Directory Network (IDN), which provides support for internet searches.

For ds353.4 all the list description tables and various variable notes had to be moved to this supplementary document. This included some critical information about certain variables. In the past, some scientists had ignored their responsibility to thoroughly examine this document, which adversely effected a few major projects. They requested that we show some of this critical information on the home page, so we accommodated them. The new DSS website rules require that this information be returned to this document.

The rewrite also updated a few things, corrected a couple of things, and led to a little reorganization.
Updated 08 June 2007 

© 2008, UCAR  |  Privacy Policy  |  Terms of Use  |  Webmaster