Processing and preservation of meteorological observation data
Processing and Preservation of Meteorological Observation Data
YUEN Chun-pong
December 2015
Have you ever wondered about the weather of the day you were born or other important days in the past? Were they cold, windy or stormy? The webpage for "Weather on a Special Day" on the Hong Kong Observatory’s website provides information on the weather conditions of the days you are interested. In fact, the webpage is supported by the weather data accumulated bit by bit by the Hong Kong Observatory since its inauguration more than a hundred years ago.
Since 1884, the HKO has been taking regular weather observations. In the old days, all weather observation data were recorded on paper manually due to the limited resources and technologies. As a result, the amount of data recorded were relatively meagre.
With technology advancements, volume of weather observation data is ever increasing. The Observatory started to use computers to record and store these observation data in plain text files. Due to establishment of an Automatic Weather Station Network in 1984, and more frequent data exchange with other weather centres, the volume of data to be handled increased rapidly. Storing data in plain text files became inappropriate. In response, relational database management system was first introduced in the mid 90's and used up to now.
After several generations of enhancements, the HKO currently has four main databases for different business areas. They are:
- Automatic Weather Station Database - handling observation data from automatic weather stations over the territory and adjacent areas.
- Global Telecommunications System Database - handling bulletins and data exchanged with weather centres in other parts of the world via the Global Telecommunications System.
- Meteorological Database - handling observation data other than those from automatic weather stations.
- Climate Database - handling local climate data.
- Automatic Weather Station Database - handling observation data from automatic weather stations over the territory and adjacent areas.
- Global Telecommunications System Database - handling bulletins and data exchanged with weather centres in other parts of the world via the Global Telecommunications System.
- Meteorological Database - handling observation data other than those from automatic weather stations.
- Climate Database - handling local climate data.
The existing relational database management system handles millions of weather observation data per day to support weather forecast and warning services, as well as research activities. The database employs partitioning technology to segregate the huge amount of data to facilitate processing. Older data are archived on magnetic tapes or other electronic media.
Information technology is changing fast. The ever increasing amount of weather observation data is bringing new challenges for the HKO. We are exploring and introducing new data processing and preservation methods and techniques to embrace challenges and seize opportunities in the imminent Big Data era.