Email:  Password:  
  Forgot Password?  Register Now
UCAR > NCAR > CISL
Search:  
  
FAQs | Contact Us | For Staff
ds464.0  Go to Dataset: 
 

PAGE 2

Background information about the characteristics of this data may be found in An Introduction to Atmospheric and Oceanographic Datasets. There is general basic information in its Introduction.

More specific information may be found in these chapter(s) or section(s): Surface Observations

When NCEP began decoding the surface data from the GTS, they were using two internal documents to describe the eventual product. These were NMC Office Note 29 (ON29) and NMC Office Note 124 (ON124). ON124 was only a supplement to ON29, and provided information about surface data that was not in ON29. By the time final versions of these documents were prepared in 2001, the ON29 had become a document solely for upper air data files (as in our ds353.4) and ON124 was a complete stand alone document for surface data files.

In 1997 we got an email from Dennis Keyser (NCEP) about matters regarding ON29 and ON124, and the transition to BUFR. Please look at this. It is a good example of what can occur with operational data.


TYPES:

Operational
Synoptic
Surface observations
Sea observations
By land stations
By ship stations
By buoy
Bogus data
Oceanographic


COVERAGE:

Global coverage
Network
See this. DSS does not have a list of available stations, but DSS does have station receipt summaries here (updated yearly).

The stations are identified by 5 digit WMO number or call sign, and these can be found in our USAF Station Libraries, which has station names, locations, elevations, etc. Note that in this library, 6 digit WMO numbers are used, where [5 digits]0 applies. Caution: the content of these libraries must not be taken as a list of what you will find in this dataset, it is just a cross-reference.

A map of the WMO regions is here.

METAR reports (within the ADPSFC, which use call sign identification) disappear on 1993Sep28.09, and do not reappear until 1998Mar13.12, at which time a larger number appear, apparently from Europe, and then during 1998 a great portion of the northern hemisphere and perhaps more appear.

North American Surface Airway reports have been decoded by NCEP and included in this data. But DSS discovered in November 1995 that they ceased doing this on September 24, 1993. DSS recommends that you use DS472.0, "TDL surface hourlies," to get the surface airways data for 1993-1998. DSS notes that this data returned March 13, 1998, and as of this writing now appears to include most of the northern hemisphere.

ASOS reports were implemented in late 1994, and these do not include a complete set of cloud data. A news story about issues involving ASOS observations is here.

AWOS reports began to appear in March 1998, - these can occur as often as every 20 minutes.

Beginning with the January 2004 data, in order to comply with WMO Resolution 40, DSS has been removing some restricted data. This involves proprietary data from certain commercial sources.


CRITICAL INFORMATION LINKS

If you don't take the time to study all the information below, you may experience unexpected frustrations when using the data. Being an operational dataset, it may not be as clean as you may wish or assume.


INTRODUCTION

DS464.0 is maintained in NMC Office Note 124 (ON124) format. Originally NMC decoded the GTS data and reformatted it directly to ON29 and ON124. The surface data is not used in the NCEP models. Beginning sometime in the late 1990s, NCEP put it directly into BUFR format, and then, for testing, converted this back to ON29. When they stopped this conversion in October1999, DSS began doing it using NCEP's software.

NCEP retrieves the reports off of the GTS and collects them in files which have a synoptic date/time stamp, which corresponds to the analysis times of the NCEP models. This is an operational dataset, meaning that when stations fail to report over the GTS, or there are other data losses, neither NCEP nor DSS can recover the data.

The ongoing source of the data for the BUFR to ON29 conversion is NCEP's "dump" BUFR data files. NCEP has applied some interactive quality control, removed duplicate reports, and merged upper-air "parts". See Observational Data Processing at NCEP. More rigorous automated quality control has not been applied.

Be aware that the ON124 format does not provide for as much information as does the BUFR format, Layers 1&2 and Layer 3. (BUFR documents obtained from here, also see this information). The primary information is there, but some new data types carried in BUFR do not appear in ON124. Finally, BUFR also carries the original raw reports which NCEP collected from the GTS - ON124 does not.

NCEP uses their "dump" BUFR files to prepare the PREPBUFR files which are directly fed to the models. NCEP has not been using the surface data in their Perhaps this will change in the future.

For the curious, a history of U.S. weather data technology is here.


UPDATES:

Monthly

Updated usually about 2-3 weeks after the end of a month, but this lag time can be irregular. We use files that NCEP prepares specifically for us.


DS464.0 SUBSET NOTES

PERIODS OF RECORD AND VOLUME:

Continuous record begins on Feb 11, 1975.


You should notice that the volume of the ds464.0 data has increased significantly, by an order of magnitude, since the early 1990s. This trend may continue.


FILE STRUCTURE - see this .


NCEP'S TIME AND NAMING CONVENTIONS

BASIC - NCEP segregates the reports by dataset type in addition to the synoptic date/time, and calls the group a data file. The name of a data file is analogous to a directory of observation types. For example SFCSHP (surface ship observations) will include ships, moored buoys, drifting buoys, etc. A data file begins with a header block and is terminated by a trailer block. The header block identifies the dataset type (e.g. ADPUPA or SFCSHP) and synoptic date/time (which may also be called the header time). The actual observation times appear in the individual reports. The individual reports do not show the synoptic date/time stamp.

DSS converts NCEP's header blocks to the GATE format. This was originally intended just to support GARP projects in the mid-1970s, but has persisted and become our paradigm.

DATE/TIME STAMP - The accumulation of observations for a synoptic time begins before the synoptic time, as much as half of the time interval between synoptic times, but it should not be before the cutoff time of the previous data file. The header blocks will often, but not always, show a receipt time which indicates when the file was opened to begin the accumulation of reports. Likewise the cutoff time indicates when the file was closed, terminating the accumulation. DSS has found that these delineators have not been dependable, so users should just ignore them.

The synoptic time, header time or analysis time are all the same and are specified in Coordinated Universal Time (CUT or Z). The individual report times ares also given in UT and are the actual observation times, as reported by the stations. Stations may report observation times which look exactly like the synoptic times, but in practise these may not always be true.

For information about the time convention, see this.

WARNING: Many users have been accepting or mistakenly expecting the synoptic time to be the observation time. If their work is sensitive to the difference, then they should use the observation times. This is a very important consideration because often users will find more than one report from a station within a data file for a synoptic time. Moreover, some reports in a given data file may be duplicated in the previous or following file.

Users must be aware that the actual time of an observation should be obtained from within the individual report. One may also want to adjust the date shown in the header block, because for some 0Z (and even 06Z) data files the report times which show values between 18Z and 24Z actually belong to the previous date. A comparable thing may occur at 18Z, where the report may belong to the next day. Users need to be aware of this. Our latest access software, readsfc2.f, allows a user to make the adjustment.

STATION IDENTIFICATION - Generally the 6-hourly and 3-hourly SYNOP reports use WMO 5 digit numbers. Many other reports, such as METAR, or hourly and 20 minute reports, use call signs. So when attempting to extract all the reports made by a station, or stations nearby, one should use a latitude-longitude "window", which will get both the WMO and call sign labeled reports. Such a "window" will obtain neighboring station data that could be used when the desired station data is missing. See the code in readsfc2.f. Also see our combined list of U.S.A.F. catalogs.

DATA VALUES AND OBSERVATION TYPES - Users sometimes believe these are the same, but they are not. Observation type refers to the report type, or platform, such as land station, moving ship, etc.


NCEP'S DATA FILE STRUCTURE

NCEP's file organization (not the format) has varied over the years. DSS does not alter this. From January 1, 1978 - March 31, 1997 NCEP/NMC sent us this data on weekly tapes which sequenced all the week's data for each dataset type together. In other words, all the observation types for a given synoptic time were not contiguous. NCEP grouped all the land surface 6-hourly (00Z, 06Z, 12Z and 18Z) reports in one stream of tapes, and all the other data in another stream of tapes. The latter includes all ship reports plus the 03Z, 09Z, 15Z and 21Z land reports. Users are cautioned that what we often call the 3-hourly files for the 1978 - March 1997 period do not include the land 6-hourly reports.

Prior to January 1, 1978, we have two other structures. Between 1973 and 1975 DSS has only sample files in no particular order. From July 6, 1976 - December 31, 1977, the data files appear in one stream of files.

Beginning sometime before 1997, perhaps April 1996, NCEP began putting the GTS data directly into the BUFR format. Until October 1999, NCEP used their BUFRON29 conversion program to produce the ON29 formatted files from the BUFR files. This was done initially to verify content, and then for awhile as a service to users while they adapted to BUFR. NCEP ran this conversion on their Crays, and when they abandoned their Crays for a new system, they stopped making this conversion. In February 2000, DSS acquired NCEP's BUFRON29 in order to continue production of the ON29 format. NCEP has discontinued support for the ON29 format, and for the BUFRON29 software. DSS has been using BUFRON29 while knowing very little about it - the thing is a large black box.

NCEP made several changes in the spring of 1997, mostly in April. First, NCEP began putting the data in monthly files, inside huge tarfiles. DSS found it easier to split these directly to 5 day land and 30 day ship subsets, and continue to do so.

In April 1997, NCEP changed the streams: they put the land surface in one stream, all the ship surface in another. They also stopped making separate data files for 03Z, 09Z, 15Z and 21Z. So DSS is now getting 6-hourly files which have all available 3-hourly, hourly and even some 20-minute reports, within 3 hours of the 6-hourly synoptic times. There can be some duplication of reports, especially when station METAR reports fall on the same time as SYNOP reports.

In April 1997 NCEP stopped shipping the ADP tapes directly to us. In February 1998 DSS obtained copies of the tapes which they continued to send to NCDC. Another batch was so obtained to bring us through March 1998, and then NCEP's shipments directly to us resumed. Then in June 2000, tape shipments were terminated in favor of using NCEP's FTP server.

DSS has been collecting the BUFR formatted data in DS464.5 and DS461.0. The operational changes at NCEP between 1996 and 2004 has created ongoing challenges to DSS for extending ds464.0 in a consistent format. We will continue to provide this data in the ON29 format for as long as we can run the BUFRON29 conversion.




THE DATA

GENERAL - This data is not available as a database. DSS also reminds users that the files consist solely of the data which the stations reported over the GTS, and subsequently what NMC/NCEP saved on an operational basis. Dropouts are not unusual, and DSS can not recover them.

The data are provided in "packed ASCII" flat files containing a series of complete station reports. This has many implications. Being ASCII rather than binary significantly increases the volume. Being synoptically sorted, the expense for extracting a time series of reports from a handful of stations has become extraordinary to the point of being unrealistic.

Our software, such as readsfc2.f, opens individual reports and prints recognizable data values. Users who want to see just a few data values, or even just one, such as temperature, must still open each entire individual report to access them. Users will need to modify our access routines to obtain such special subsets.

You may also want to look at some general WMO Code Tables, and some special WMO tables for height of base of lowest cloud, and total cloud cover.

REPORT COUNTS - Our inventories show report counts by synoptic time and station. The latter are updated annually, and show monthly totals. These may be accessed from the tab bar near the top of the main Web page.

DSS recently added a series of annual sample printouts, found here. To obtain these, DSS made a pass through the entire dataset, to extract all the data from the first day of each month, at 00Z and 12Z, for about 10% of all available stations. These files are here. These "snapshots" of the data allow us and users to test the content of the reports among other things. In particular, it has enabled us to study the availability of the cloud and precipitation data.

DSS discovered a steady decline in the total number of reports, from about late 1991 until early 1994. Then there was a steady recovery, terminated by a large rebound in August 1994. This may be related to a similar behavior in the upper air data receipts, where NCEP had a problem with what they called a "flooded station table." Starting with the April 28, 1996 data, shipment cover letters from NCEP indicate data loss due to "too much data" which evidently overflowed their capture buffers. When NCEP moved their processing to larger and faster machines, this problem vanished.

There has been a huge increase in the number of land surface reports in the last few years. This involves the addition of hourly data and even 20 minute data.


VARIABLES:

QUALITY CONTROL - NCEP does some Quality Control (QC) on the data, and they set some QC flags in the reports to qualify, but not quantify their corrections. DSS believes that the QC for this surface data is not as rigorous as for the upper air data, which is likely due to its smaller role in the models. DSS does not do QC, but will investigate user complaints about data values.
Missing data values are usually indicated by a field of 9s.

DATA UNITS - The data units in this dataset involve both English and metric. See ON29 for details.

SEA LEVEL PRESSURE - This dataset has reports which include surface (station) pressure, sea level pressure, or both. From a scientific point of view, stations which report a sea level pressure, but not the observed station pressure, are being irresponsible. This is because it spoils the possibility of applying improved algorithms for the reduction of pressure from station elevation to sea level, among other things. For that matter, methods have changed over time.

The reduction of station surface pressure to sea level pressure is almost a black art. Basic meteorology texts describe theories which involve assumptions about the virtual temperature in an imaginary level between the station and sea level. An example is in the classic text "An Introduction to Theoretical Meteorology" by Seymour Hess (1959), on pages 88 - 90. In practise, the reduction algorithms are empirical, where local stations apply local statistical knowledge of weather and seasonal patterns.

Here are a couple of more recent texts on the subject: Chuang, et al and NOAA/NWS.

It is reasonable to expect to know exactly what was done in this dataset, but we do not have the answer.


WINDS - From at least Apr 2000 to May 2002, NCEP's GTS decoder generated incorrect wind speeds for certain parts of the world that were based on false assumptions about the units. The nature of the corrections is such that the information needed to make them does not appear in the ON29 data. Doug Schuster analyzed the BUFR data and prepared some documents about the problems: global, regional and more regional. Contact Doug Schuster for more information. We urge you to be careful about using the wind data for this period.

CLOUDS - In recent years the cloud data fields have become unreliable or even unavailable. Initially this was limited to just the United States, beginning in late 1994, when the U.S. observing network switched to automated measurements (ASOS) which could not determine cloud types. ON 2000 February 1, NCEP stopped decoding the cloud type data for the entire globe. This also occurred in DS470.0. A news story about issues involving ASOS observations is here.

DEWPOINT - The GCMD does not provide a keyword for dew point depression. We are forced by the GCMD rules to show "Dew Point Temperature" although the data values in the reports are actually dew point depression.

PRECIPITATION - DSS is frequently asked about the availability of precipitation data, in this data set, outside of the U.S. NCEP routinely decodes the GTS precipitation data only for North American stations, putting it in the report category 52 data. See this. NCEP does not decode the data for the remainder of the world, but does save it in the report category 08 data. Tables of numerous decoding schemes are required to decode these data, and DSS has just a few of these on paper. NCEP operations considers these data to be climatological data, of little use for their models and forecasts. DSS understands that NCEP's Climate Prediction Center (CPC) has been decoding these globally. Please refer to DS512.0, our global summary of day dataset, which is prepared by CPC, but note that it shows only daily totals.

WEATHER CODES - In late 1981, reports from automated stations began to appear. In 1987, the present and past weather codes for the automated stations were switched to a different code table. Users have reported a discontuity in the frequency of these codes denoting precipitation beginning in late 1981. See the final NMC ON124 document for more information.


FORMAT:

NMC Office Note 29 , NMC Office Note 124 , 85 and 108. Observational Data Processing at NCEP (BUFR)


The data is in a format which can be processed by the MM5, WRF and RAMS.

MM5 user advice...

RAMS user advice...

Also be aware of these processing issues.

A little advice is available for: GRIB and GrADs (display) software


OTHER

One should always be aware of Observational Data Problems.

You may wish to refer to Other Sources of Data.

GCMD In October 2006, all our dataset home pages were rewritten, and they will be subject to more changes. These actions accommodate a new dependency on a new DSS database and new rules based on the Global Change Master Directory (GCMD), which is the American Coordinating Node of the Committee on Earth Observation Satellites (CEOS) International Directory Network (IDN), which provides support for internet searches.

For ds464.0 all the list description tables and various variable notes had to be moved to this supplementary document. This included some critical information about certain variables. In the past, some scientists had ignored their responsibility to thoroughly examine this document, which adversely effected a few major projects. They requested that we show some of this critical information on the home page, so we accommodated them. The new DSS website rules require that this information be returned to this document.

The rewrite also updated a few things, corrected a couple of things, and led to a little reorganization.


Updated 8 June 2007 

© 2008, UCAR  |  Privacy Policy  |  Terms of Use  |  Webmaster