Big Data and analytical systems. Data on air pollution for exact forecasts

Smog 1n Mexico DF

In recent years, the situation with air pollution is becoming more alarming and attracted the attention of leading media and politicians at the highest level. This is especially true of large cities such as Beijing, Los Angeles, Mexico City, Tehran, Johannesburg and other densely populated settlements, which are under the influence of various risk factors, such as dense traffic, the presence of air polluting enterprises located poorly in terms of ecology, where pollution levels sometimes exceed all standards many times.

Depending on the degree of interest of the authorities in cities, pollution control is more or less in place, but, unfortunately, the monitoring systems which are in use now, are ineffective. Technologies that allow to build forecasts already exist, but they do not provide the necessary level of accuracy and do not allow to predict rises to a dangerous levels of pollution. The basis for the creation of accurate predictions certainly are accurate data on key parameters in an amount, that is sufficient for analysis.

OpenWeatherMap provides unencrypted API data on basic parameters such as carbon CO2, ozone O3, nitrogen dioxide NO2 and sulfur dioxide S02, as well as data on the contents in the air of fine particles AOD (Aerosol optical depth), which are extremely dangerous for human health. OpenWeatherMap processes massive amounts of accurate data that can be processed for any geo point in the world. This data is obtained from tens of thousands of sensors worldwide from satellites and monitoring systems. On the basis of such data, using modern technology, such as the Internet of Things (IoT) and Big Data has now become possible to build the analytical system, capable of delivering high-precision forecasts for maximum contaminant levels for a few days in advance. With these forecasts, experts can formulate recommendations, and local authorities are able to take measures to reduce pollution, such as the optimization of the traffic, movement of industrial facilities outside the city, planting of forests and the creating green parks at key locations in major cities, the use of alternative sources energy. Similar measures have already been shown to be effective in some countries — in the UK (London) and in Japan (Tokyo).

OpenWeatherMap provides API to the historical data, as long as to the data obtained as a result of monitoring in real time, which allows a visual comparison, including the virtualized format and helps to identify the sources and periods of maximum and minimum levels of pollution, their possible recurrence, estimate the effectiveness of measures taken to combat the pollution and to understand the weaknesses of the monitoring infrastructure in the big cities.

With the help of a joint analysis of pollution data, historical and current weather and satellite data provided by OpenWeatherMap, one can find the cause of environmental degradation in any region or a particular village — to find out the location of the sources of pollution, to assess the impact of, for example, the formation and dispersion of smog, important factors such as temperature and humidity, wind direction, the level of ultraviolet radiation (UV index), to evaluate the pros and cons of geographic location.

OpenWeatherMap interactive maps

Currently, OpenWeatherMap, together with its partners, develops innovative solutions and models, which allow not only to deliver data and API to different data in the clear mode, as is the case now, but also to visualize all this global information clearly presenting it in the form of graphs and interactive maps which makes the work with data suitable for analysis.

Your business partner from space: How satellite vessel monitoring system OWM AIS helps to run business effectively

You never know what is enough unless you know what is more than enough.
- William Blake

As you know, who owns the information can either save the world or destroy it. Timely information can protect your health, trigger a stock market crisis or help to win the war.

In today’s world there are many different ways to retrieve data: around our planet circling satellites, drones are flying everywhere and all the sensors are used with complex signal processing, the people are implanting chips all over the world.

At the same time, with regard to satellites, the steady tendency is to reduce them in size, and make them cheaper, which significantly affects the cost, speed of data acquisition and volumes. And there were small satellites supplying huge amounts of data that need to be somehow handled. And handled as fast as possible. And here comes the help of modern technologies, such as cloud, the Internet of Things and Big data.

Behind all this is a real revolution in the field of satellite data. It is already underway and will continue, the further the more interesting things will become. The relatively young field of application of satellite data is collecting AIS data from space — AIS vessels monitoring — tracking the movement of ships along the coastlines and on the open sea and oceans.

This information is very useful for owners of any business, one way or another related to shipping:

• for port services — port owner can check the information provided by the ship on course and destination;

• for ship-owners, shipping agent, freight forwarders and charterers AIS is able to determine the current location of the vessel and monitoring the route on a map, find out complete list of parameters for each AIS vessel and reports to effectively manage small and medium sized fleet;

• for financiers, economists — analysis of the trends, risks, and helps to identify new business opportunities, determining the size of the market and the competition assessment. For example, information about how oil tankers move along the sea routes helps to make certain predictions, which makes it possible to go a little ahead of the market. Similarly, you can keep track of cargo ships, and after cargo carriage — is the blood of the world economy. 

Density map: density of vessels within a specific year or month in the past on interactive map to analyze global picture of marine traffic

But to use all this information, a complex analysis is required, which allows to receive it in visual form, and in a short period of time. For example, the maximum time passed from the moment of receiving the data from the satellites to the moment when the data is available through API OWM AIS services keeps within half an hour.

JSON API to current and historical satellite AIS data for vessel or group of vessels

OWM AIS service enables online mode to watch the movement of about 300,000 ships filtered by more than 50 types and create interactive maps for displaying movement of vessels for each fleet type — such as freight, passenger, and fishing or combined. The presence of historical satellite AIS data, which covers both coasts and oceans, broken down by month, also allows you to display the density of vessels on the interactive map within a particular year or month in the past, to analyze the global picture of maritime traffic.

A very interesting is application area of historical data and dynamic satellite maps. Their joint analysis can greatly facilitate the investigation of accidents, incidents have occurred in the open ocean, as well as help in tracking the pirates and violations of contracts.

 

 

Battle of modern technologies in open ocean: pirates start and win

Fifteen men on a dead man’s chest
 Yo ho ho and a bottle of rum
 Drink and the devil had done for the rest
 Yo ho ho and a bottle of rum.
“ Dead Man’s Chest” ( Robert Louis Balfour Stevenson)

When a talk comes about pirates, usually the images of Captain Jack Sparrow, the traditional “Jolly Roger”, views of barrels of rum and old wooden chests full of gold coins spring to mind. Or it can be stories about Queen Elizabeth of England, Sir Francis Drake who defeated the Spanish Armada, and Sir Henry Morgan notorious for his Caribbean adventures. However maritime piracy did not sink into oblivion. Nowadays this theme has lost its romantic glory, but it still has its significance for a present-day society.

Basically a goal of pirates remains the same that is capture of hostages, goods and gold. And there is a little distinction from ancient times, not gold but ‘black gold’ (i.e. petroleum) consists the main interest of modern pirates these days. Attacking oil tankers that carry both petroleum and money is common practice for pirates in this day and age. Along with that, means used by pirates have greatly changed, namely equipment and technology. The rise in modern maritime piracy started in Somali waters and for a rather short period of time of 10 years it has expanded extremely: armory of pirates has become diversified and their strategies of capturing vessels developed.

According to Tactical Intelligence International, a business providing maritime security, military training and government consulting for over 10 years, pirates have started to use heavy weapons and specific facilities to climb up vessels and conduct their operations from so-called mother ships. Recent tactics of pirates include using of remote bases that enables them to control and carry out operations at a greater distance of the shore, and for a longer time.

One of the main issues of the fight against piracy is pirates going further into the sea and thus making it harder for warships of the European Union to guard potentially dangerous areas. This fact decreases chances of assisting victimized vessels in time even if a message about an act of piracy is received.

Consequently, in spite of cooperation of naval forces of different states and the use of expensive preventive measures such as the presence of armed guards on board, the global problem of piracy has not been solved for several decades and it is still crucial. One of the recommended measures to prevent attacks is to limit access to information on the movement of ships. However unfortunately, international practice shows that these measures are useless or, at least, they work for some time.

Throughout the world, data become easy to access, monitoring systems can be used by the general public. Any attempt to restrict information flows is pointless like the Luddite movement or disapproval of Uber application by taxi drivers operating individually. As a result, digital technologies are widely employed by pirates and drug cartels: hackers break into IT systems of ports, while pirates use AIS monitoring system, and shipping containers with the most expensive goods are tracked by barcode.

One can assume that in the near future the way of the fight against piracy will obtain more technological characteristics.
 Modern technologies in the field of satellite data allow conducting of such measures as the global monitoring of the movement of vessels, not only near the coast, but also in the open ocean, including the detection of ships which hide their personal information, i.e. potential pirates.

 

Photo of the ship from satellites

Current information about vessels for any polygon in an API response from an analytical service OWM AIS and monitoring of suspicious areas on satellite images make it possible to conduct an investigation in a particular area and to detect vessels that do not report their location, as well as to determine where and when a missing ship transmitted its last signal. Owing to technology of OWM Platform Big Data and availability of a direct source of satellite data from the company SpaceQuest, OpenWeatherMap develops complex products such as, for instance, Density Maps where the routes of vessels like oil tankers and container ships are clearly observed via current and historical position of vessel. And thus, knowing the places where pirates are most likely to appear, one can choose areas to monitor the route, the type and the presence of signal from vessel, not across the sea in general, but in definite most dangerous areas.

Density Maps reflects the main ways and the density of the ships movement

Peculiarities of weather forecast for agriculture

No doubt weather is the most important factor determining whether an agricultural enterprise is successful or not. Changes in the weather greatly affect growth, state and amount of crops, its disease incidence, its need for water and fertilizers. Not only plants, but also a whole agricultural market highly depends on weather. Directly and due to some sideway reasons, weather is guilty of loss of the annual world crop in the amount of its three out of four! A mistake in weather forecast can result in crop loss.
Generally, as science and modern technologies evolve, the dependence of crop on weather seems to decrease but of course there is still some correlation of yield amount and weather conditions. Nevertheless, crop loss can be reduced significantly by adjusting taken measures that are based on well-timed and accurate weather forecast. Such weather forecast allows developing recommendations for long-term and seasonal planning and for selection of crops best suitable for expected climatic conditions.
Here one needs to take into consideration some specific features of weather forecast for agriculture which are quite different from those of weather forecast for daily living needs. Efficiency of weather forecasts is higher when attention of forecasters is emphasized on requests of farmers, and farmers know how to use these forecasts and what steps to take. The case is complicated by the fact that the reaction of various agricultural plants to type and intensity of peculiar weather phenomena is different; and this reaction also varies at different stages while plants grow. Furthermore, due to differences in climatic conditions, ways of cultivation of agricultural plants vary at certain world locations, even during the same season.

Thus forecasts for short and long periods of time are of the same great importance. For instance, making decisions based on forecast for late start of the season of growing plants requires significant alterations of ways of cultivation compared with ordinary start.
Organization and implementation of such strategies are considered as decisions of high cost, and usually they require quite a lot of time. That is why preliminary seasonal forecasts should be valid for at least 10 days and be received not less than a week beforehand.
Also measures of diminishing consequences of dangerous meteorological phenomena, of hazardous pests and diseases should be conducted properly in time and as they require some considerable time, so current seasonal forecasts should be received 5 days (at least not later than 3 days!) beforehand preferably.
One should mention here, as weather forecasts are processed for agricultural users, they should be distributed as soon as possible.
Since the season of growing plants starts, the only way to affect the situation is to minimize the impact of severe weather conditions by a timely adoption of preventive measures; the factor of time can be crucial here.
For example, crop loss due to night frosts can be prevented by irrigation, mulching or using smoke bombs.
The main weather phenomena timely information about which can significantly reduce commercial risks are:

• Precipitation, its amount and intensity. Warning of hail should be distributed in advance as there are agricultural plants that are to be protected (e.g., leafy and salad vegetables), and it takes additional expenses to have staff in the fields on constant basis.
• Temperature. In regions with temperate climate slight frosts represent a significant threat to crop, however lowering the temperature below zero has different impact on various kinds of agricultural plants: potato can perish while radish can withstand a drop in temperature up to -4C.
  In any cases current forecasts and chronological forecast data of previous seasons are both important.

So far as while making forecasts for agriculture, information on abnormal weather phenomena and its possibility for specific location should be emphasized; particular performance standards of weather regime and plant growth should be stated. Otherwise how abnormal weather phenomena can be recognized when a performance standard is not fixed. Thus chronological forecast data of previous seasons for this specific location represent particular interest; by scrutinizing which one can create a performance standard of ordinary weather conditions and can trace prevalence and frequency of abnormal weather phenomena.
In view of the above OpenWeatherMap company plans launching of new products designed for agriculture. Let’s see what we have in stores for our users: weather forecast for a specific region up to 10 days, temperature defining for any point worldwide (current and historical data), time, amount and location of precipitation (history and forecast), weather warnings of hail, frosts, storms and other abnormal weather phenomena.

Air pollution: ways to forecast and calculate it

Gazing through the window at the world outside
Wondering will mother earth survive
Hoping that mankind will stop abusing her sometime
- Ozzy Osbourne

In recent years, the fight against air pollution has become global. Humanity, at least when we are speaking about developed countries, starts to realize finally that the Earth is our common home which can get not suitable for living in the near future. Atmospheric pollution takes place when substances harmful for living creatures diffuse throughout the surrounding air, and then this destructive activity has its toxic impact later since it causes global changes of climate of the Earth.
 Processes of global warming, oxidation of world ocean, melting of the ice caps at the poles, thinning of the ozone layer, dense smog over metropolitan areas — these are only a part of the problems that have arisen as a result of air pollution. Generally speaking, air pollution is a penetration of physical, chemical and/or biological contents into air that are usually not common for air, or a change of natural concentration of these contents in the air.
 The most important indicators of air pollution are carbon monoxide (CO), ozone (O3), sulfur dioxide (SO2) and nitrogen dioxide (NO2). With the increase in their permissible concentration in the air these gases affect both the environment and human well-being.

Carbon monoxide (CO)
Any process during which the partial burning of carbon-containing substance happens is a possible source of producing of carbon monoxide.
Carbon monoxide occurs in the atmosphere as a byproduct of industrial plants; petroleum industry and chemical plants contribute to a large amount of carbon monoxide production. The main sources of carbon monoxide are the burning of fossil fuels and the exhaust gases of internal combustion engines. This gas is extremely toxic.
In a human body carbon monoxide is inhaled with air and comes to blood where it combines with hemoglobin to produce carboxyhemoglobin, which assumes the space in hemoglobin that normally carries oxygen, but carboxyhemoglobin is ineffective for delivering oxygen to bodily tissues. Thus it causes seizure, coma and fatality. The most significant damage is caused to central nervous system. Breathing a small concentration of carbon monoxide results in a feeling of heaviness, a feeling of squeezing in head, a heavy headache, severe pain in forehead and temples, dizziness, tremors, thirst, increased heart rate, nausea, vomiting, a rise in body temperature to 38–40C. Weakness in legs indicates influence of carbon monoxide on the spinal cord. High toxicity of carbon monoxide and its lack of colour, odour and taste make this gas extremely dangerous.

Ozone (O3)
Actually ozone is present in a low concentration throughout the Earth’s atmosphere. Ozone is an allotrope of oxygen and is vital for living creatures. Ozone gas forms the crucial ozone layer between about 10 km and 30 km above the Earth’s surface preventing damaging ultraviolet light from reaching the Earth’s surface, to the benefit of both plants and animals. But in close proximity to a human being ozone is much less salutary.
Ozone is a substance of the highest, the first class of hazard since it is a very strong oxidizer, which is extremely toxic to humans, and is the third largest greenhouse gas originating in human activity of environmental pollution.
As a component of smog, ozone is not discharged to the atmosphere, and is formed by the action of daylight UV rays on ozone precursors. Ozone precursors are a group of pollutants, predominantly those emitted during the combustion of fossil fuels; these are nitrogen oxides, volatile hydrocarbons and some other compounds.
Ozone is harmful to living beings, its action slow but serious. Long-term exposure to air rich in ozone has been shown to increase risk of cardiovascular diseases, respiratory problems, developing atherosclerosis.
Exposure to high ozone concentration above the permissible limit values is linked to headache, mucosal irritation, coughing, dizziness, fatigue, weak cardiac activity. Breathing of ozone leads to emergence or exacerbation of respiratory diseases, populations at risk are children, the elderly, and asthmatics.
Ozone adversely affects not only human health, but it is also harmful for agriculture: tons of crop of rice, wheat, soybeans and corn are lost annually due to toxicity of this gas.

Sulfur dioxide (SO2)
Sulfur dioxide is the most common air contaminant as many sorts of fuel contain sulfur. This gas has a general toxic effect and causes respiratory problems. Symptoms of poisoning are the following: a cough, rheum, hoarseness, strong sore throat and a peculiar taste in the mouth. Breathing of high concentration of sulfur dioxide results in difficulty swallowing, choking, speech disorder, nausea and vomiting, acute pulmonary edema may develop.

Nitrogen dioxide (NO2) Nitrogen dioxide (NO2) is a reddish-brown gas that has a characteristic sharp, biting odor. The main sources of nitrogen dioxide are internal combustion engines burning fossil fuels. Often this brown gas is called “Megapolis gas”. Its presence in the atmosphere contributes to acid rains and reduces the ozone layer. Nitrogen dioxide is highly toxic, and it adversely affects the sense of smell. Thus, a 10-minute inhalation of nitrogen dioxide is enough for a person to stop feeling smell; dryness in the throat appears, mucous membranes become irritated. Even small concentrations of nitrogen dioxide irritate the respiratory tract, and high concentrations cause pulmonary edema. Along with influence on the respiratory tract and lungs, nitrogen causes changes in blood composition, in particular, it reduces hemoglobin concentration in blood. Cancer and cardiovascular diseases are the results of the harmful effects of nitrogen dioxide.

One of the challenges of global and local assessment of the environmental situation consists in the fact that the values of the basic indicators vary widely not only from region to region but also even within a few kilometers. It depends on many factors such as anthropogenic (the proximity of fuel combustion sources, of industrial plants, actively ongoing agriculture and street traffic) and weather factors (wind direction, humidity and sunlight). Geographical location of sources of pollution and objects under consideration is not less important. All this must be taken into account in making forecasts, in development of measures to fix critical situations and in creating long-term plans for environmental improvement.
 One should understand that all these measures are unity of specific factors, which can be measured, calculated, forecast, and as a result, regulated. And modern technologies already allow doing that.
 Not wanting to stay away from the solution of world problems, OpenWeatherMap company has released a new product called “Air quality data” based on the technology of Big Data Platform in April 2016. There is a package of APIs for data on the main indicators of air pollution, on the basis of which it is possible to adequately and accurately evaluate the environmental situation anywhere in the world and create forecasts for the adoption of preventive measures.
 APIs for historical data (since November 2015) allow to create analytical services for the analysis of situation in the past, taking into account the impact of seasonal, weather and anthropogenic factors. It also helps making conclusions on necessary measures to improve the environmental situation across the countries and even continents.
 However no matter how advanced technology is, people and their determination to take the necessary steps have always been and will be the crucial power in preventing a potential disaster.

Use of satellite technology for decreasing risks in agriculture

The exploration of outer space and launching of satellites started in the 1960s. Having some romantic flair in the beginning, the satellite story went on and nowadays satellites are launched often and almost everywhere around the globe and their number is going to grow in the future. Together with that, a new question arose, what to do with a bunch of new information received? A huge lot of information has been gained in the satellite era, but its practical appliance can be seen only in certain spheres of economics.
One of the spheres where data received from satellites are used effectively is agriculture. There is a range of agricultural problems which can be solved with the help of these data. Satellites are classified into ones of high resolution, of medium and of small. Basically for agriculture satellites of medium resolution of 5-250m per pixel are used. What can be seen in such satellite images and what tasks can be done with the help of satellite data?
Using satellite data one can judge about the level of cultivation of soil in specified area, identify actively cultivated and abandoned fields and define their configuration. Also with the help of satellite data it’s possible to get information about the current state of things without visiting a specific field. Analysis of satellite data allows understanding whether agricultural events are properly conducted, whether the technologies of cultivation are not violated, and whether soil requires additional treatment for enhancing fertility. With satellite images it’s possible to evaluate condition of agricultural plants; the normalized difference vegetation index (NDVI) is designed for that. The normalized difference vegetation index is a simple graphical indicator of biomass active photosynthetically that is calculated via several spectral channels. Satellite possesses from 10 to 30 spectral channels. Any satellite has its own set of channels which can be visible, infrared, ultraviolet, thermal, etc. Satellites measure the reflectance of the planet in many spectral resolutions as Earth looks different in various spectral resolutions. Plants reflect light well in one spectrum while plants do absorb it in another spectrum. Correlation of these indicators allows defining live green plants from other objects. The NDVI is one of the most well-known and used index to detect live green plant canopies and to evaluate vegetated areas, their condition and quality.

Let’s see what tasks can be performed with the NDVI.

Monitoring of crop emergence.
Real-time monitoring allows supervising the current situation and forecasting crop yields. Using the NDVI it is possible to detect presence of plants, its dynamics of emergence and growth. Maps of the NDVI with dynamics (various seasons) let tracing peculiar features of seasonal vegetation, its deviations, allow planning, etc. For instance, knowing weather forecast, condition of plants and considering the current date, forecast about the amount of yield in the current yearcan be made. However, a specific feature of satellite monitoring should be taken in to consideration: satellite technology works for global tasks only, this technology is pointless for small farms. Small local farms do not need satellite technologies in their everyday operations. But this kind of information can be of great importance for wholesale purchasing companies. In this case information about extensive regions is of big interest: what region is more fertile, what plants grow widely and where wholesale can be done.
Analysis of historical data.
Historical data is information on the cultivation of fields, evaluation of their productivity and the efficiency of the technologies used. Currently a large archive on a daily survey of earth surface is gained. For example, satellites of MODIS system orbit earth on a daily basis since 2000. In this archive history on any field cultivation for the last 15 years can be obtained. This is a great opportunity for insurance companies to receive information about a future customer and to decrease potential risks.
Accurate land tenure.
Nowadays global manufacturers of fertilizers provide quite complex products. These are not only fertilizers simply, but also compound technologic processes for specific kind of plants, for peculiar weather conditions. Basically, a half of the cost of maintaining crop comes to expenses on fertilizers, and issue of proper use of fertilizers is an actual task for farmers. Evaluation of field condition based on satellite, weather and historical data is an available remote tool for analyzing the efficiency of land tenure, which lets significantly reduce commercial risks and expenses on fertilizers.
Evaluation of efficiency of water reclamation systems.
Condition of irrigation and drainage systems, their violation and failures can be estimated by comparative productivity of specific kinds of agricultural plants. Evaluation of pastures and hunting grounds. The NDVI allows effectively and at a low cost solving such tasks as assessment of quality of hunting grounds and of pasture productivity for planning of their usage and detecting violations of cattle grazing.