ARCADE

Destroyed buildings and infrastructure, temporary settlements, terrain disturbances and other signs of conflict can be seen in freely available satellite imagery. The ARtillery Crater Analysis and Detection Engine (ARCADE) is experimental computer vision software developed by Rudiment and the Centre for Visual Computing at the University of Bradford. ARCADE examines satellite imagery for signs of artillery bombardment, calculates the location of artillery craters, the inbound trajectory of projectiles to aid identification of their possible origins of fire. An early version of the tool that demonstrates the core capabilities is available here.

Figure 1: Automating the detection of artillery craters near Amvrosiivka, Donetsk Oblast, Ukraine, Image date: 14 September 2014. Imagery via Google © 2015 DigitalGlobe.

Why create ARCADE?
What is ARCADE and how does it work?
Does ARCADE work and what’s next?
Can I use or hack or contribute to ARCADE?
Acknowledgements

Why create ARCADE?

The ‘eye in the sky’ has created new opportunities for investigative reporting, human rights monitoring, documentation and advocacy. For example, the examination of satellite imagery for information suggestive of human rights violations has been performed successfully by civil society groups like Amnesty International. More specifically, investigators have sought to locate specific features in the imagery – terrain disturbances consistent with military emplacements, damage to human settlements and blast craters caused by artillery fire. It is this last class of feature that the ARCADE project is concerned with.

In 2015, investigative reporters Bellingcat and MapInvestigation analysed thousands of artillery craters visible in Google satellite imagery, as part of their exposé of alleged cross-border shelling from the Russian Federation into eastern Ukraine in July 2014. This built upon an approach pioneered by the American Association for the Advancement of Science (AAAS), which used similar techniques to evidence alleged shelling by the Sri Lankan Army into a Civilian Safety Zone in northeastern Sri Lanka in May 2009.

Bellingcat shared with us the full research method they used in their investigations. It is a digital version of  in-field artillery crater analysis techniques described in United States Army Field Manuals.  Rather than taking measurements from the actual blast crater to geolocate it and determine the angle of incoming fire, it is possible to perform the core aspects of this analysis upon the blast craters visible in satellite imagery.

Using  GIS tools like Google Earth or  ArcGIS, analysts must discover and geocode the each blast crater, before performing a trajectory analysis on it. The trajectory readings collected from the imagery must then be collated, and averaged out in order to more accurately determine the direction from which the artillery fire originated.

With thousands of potential blast craters any analyst doing this work will quickly find the task gruelling, repetitive and time consuming – three factors which increase the likelihood of fatigue and error. As more locations containing blast craters are found, and updated  satellite imagery becomes available, the analysis must be repeated. Over time, the process also becomes a significant data management challenge.

All this this  suggests there is high value in exploring new technology and methodological options.  However, most off-the-shelf desktop technologies do not have the object recognition capabilities central to this task. Nor is there a readily available pool of technologists with experience in this area. Rudiment started ARCADE to address the challenges and explore whether  key parts of the US Army blast crater analysis process, as used by Bellingcat and AAAS, could be automated whilst retaining comparable levels of precision and accuracy.

For this exploration phase, we have set ourselves five constraints:

• The prototype tool must perform the core parts of the analysis process automatically;
• The project must use freely available satellite imagery;
• It must be capable of integrating with a web service;
• It must use affordable and commonplace industry technologies and methods; and,
• The code of the prototype tool must be freely distributable, and usable by anyone.

The remainder of this page describes the tool we have created, how it functions, whether it works and what’s next.

What is ARCADE and how does it work?

ARCADE is a standalone MATLAB application which can be run on a computer running Windows.

It works by applying computer vision techniques to images drawn from Google Maps. Look at the Google satellite image below, to the left – it’s of a football field sized area in eastern Ukraine.  Per the information contained in the US Army Field Manuals, we can see the image contains  shapes consistent with “low-angle fuze quick craters” made by artillery shells such as those fired by Multiple Launch Rocket Systems (MLRS). There are other types of crater, but we focussed on this one. The process used by Bellingcat to geolocate and obtain trajectory information involves using a GIS tool to place a marker over the centre of the shape, and then rotating an arrow-shaped template over the blast crater until it matches the shape in the imagery. This creates data that can be extracted into a spreadsheet for further analysis.

Figure 2: Manual process of geolocating and obtain trajectory data from suspected blast craters near Amvrosiivka, Donetsk Oblast, Ukraine. Image date 14 September 2014. Source: Google ©2015 DigitalGlobe

ARCADE will examine the same imagery and work through equivalent steps to obtain similar data:

Figure 3: Automated process of geolocating and obtain trajectory data from suspected blast craters near Amvrosiivka, Donetsk Oblast, Ukraine. Image date 14 September 2014. Source: Google ©2015 DigitalGlobe

In more detail, the process used by ARCADE looks like this:

Figure 4: Overview of the steps used by ARCADE to inspect and assess satellite imagery

Each of these steps uses a spread of techniques to assist the process of analysis. In turn:

• Input of satellite imagery: the dataset we have used during development of ARCADE is of crater fields identified as a result of conflict in eastern Ukraine. Investigators plotted out on Google Maps and released the raw data. Additional processing in a GIS package has enabled us to extract the underlying satellite imagery.
• Crater detection: first, the image is pre-processed from colour to greyscale. Following this is an object detection step to flag areas of interest in the image. This uses the Viola-Jones detection algorithm, a framework developed originally to detect faces in imagery. The algorithm is trained using images containing the craters we are looking for (positive samples), and images which do not (negative samples). Using this data the algorithm passes a little window over every part of the image. It disregards areas which are more like the negative samples and the prints a box around those areas which match the positive samples.
• Annotated images: ARCADE then creates a version of the input imagery that shows the results of the crater detection. These boxes form the areas of interest for the next step.
• Image segmentation: this process takes each areas of interest flagged as a possible crater during the detection step and attempts to delineate what is a crater and what is not a crater. ARCADE uses a technique called Active contour segmentation, based on the Chan-Vese algorithm, to provide an outline of potential craters. Given this outline, the size and features like the centre, shape and extents can be calculated and analysed.
• Extract features: ultimately, ARCADE provides data on the latitude and longitude of a crater and estimates its trajectory. It does this through a process of feature extraction. By examining the shapes formed by the outlines created during image segmentation, ARCADE first looks for the precise centre of the crater (the centroid), its extents, area and perimeter.
• Plot trajectory: using data on the extracted features, ARCADE uses a process called template matching to estimate the inbound trajectory of the projectile that created the crater. Simply, ARCADE is given a set of ideal shapes which correspond to the shapes of craters, but rotated to every degree.  It computes the similarity between what it sees in the imagery and the template.
• Output imagery to file: finally, ARCADE pulls the results of the prior steps together. It creates two image files. The first, displaying the areas of the image that seem to correspond with its data on what a crater is; the second shows the image segmentations and outlines of the craters. In a separate file ARCADE lists out its calculations of the craters’ latitude and longitude, area and perimeter, and – when the feature is complete – its trajectory.

Rudiment and the Centre for Visual Computing at the University of Bradford are currently writing a full technical article about ARCADE, which will be published in due course.

Does ARCADE work and what’s next?

ARCADE is a successful prototype that will successfully identify a blast crater, detect its edges, extract its features and export data that can be taken and plotted on a map. However, there are – as ever – caveats to this.

We have tested ARCADE on over 70 crater fields from eastern Ukraine, and the results are mixed. For example, ARCADE’s analysis of the bombardment at  Savur Mohyla in Donetsk Oblast correctly makes positive identifications of a very large number of craters.

Figure 5: Output by ARCADE on crater field at Savur Mohyla, Donetsk Oblast, Ukraine. Image date 14 September 2014. Imagery via Google © 2015 CNES / Astrium, Cnes Spot Image, DigitalGlobe

In the segmented view below the overlap between ARCADE’s results (the white shapes) and the craters that Rudiment identified by eye (the pink dots) can be more clearly seen.

Figure 5: Segmented output of analysis by ARCADE on crater field at Savur Mohyla, Donetsk Oblast, Ukraine. Image date 14 September 2014. Imagery via Google © 2015 CNES / Astrium, Cnes Spot Image, DigitalGlobe

ARCADE can clearly correctly identify blast craters (true positives) but the rate of incorrect identification (false positives) is very high.  The challenge here is to improve the part of ARCADE which defines what a crater is and is not. Variations in the data used to train ARCADE produce different results – some better, some worse.

During the development process, we learned a great deal and encountered numerous other challenges which will need to be addressed for ARCADE to move forward:

• User interaction: the ARCADE prototype demonstrates a potentially useful capability. It is an open question of how end users can best bring this into their workflow. For example, ARCADE will always have false positives, so we need to think about ways the user can quickly review this data to exclude false positives from analysis. Should it be a standalone tool, simple browser bookmarklet, a plugin for a GIS package, or a web-based application? Could ARCADE be a monitoring service, which scans a defined areas for new activity?
• Georeferencing input imagery: during engineering we had difficulty feeding ARCADE with satellite imagery from Google at a consistent scale and workable resolution. This is particularly problematic for ARCADE, because its calculations rely on relating specific pixels in the input imagery to a set of geographical coordinates. In retrospect we would have invested more engineering time to give ARCADE input from the Google Maps Static API rather than imagery export through OpenLayers in QGIS. Beyond this, ARCADE may perform better on higher resolution imagery purchased directly from DigitalGlobe or others – however we have not had the resource to test this out during this exploration phase.
• Feature prioritisation dilemmas:  We mad a mid-project choice to invest developer time in wiring ARCADE to a simple web front-end, which we were not able to complete in the time available. The penalty we incurred was loss of developer time to complete the trajectory analysis function in time for the prototype release.
• Extend and improve ARCADE’s training we have trained ARCADE on a dataset from Ukraine, where the artillery blast crater type and imagery has particular characteristics. We would welcome being able to provide artillery crater imagery from different terrains, for example desert or urban areas. Further, ARCADE does not yet learn on a rolling basis, automatically incorporating human judgement about false positives.
• Participation and public resources: we see ARCADE as part of a free set of tools to help open source investigations. For example, we can imagine a free, persistent online resource containing data about every artillery blast crater that ARCADE identifies. Attached to that could be a public verification process. Further, platforms like CrowdCrafting could be used to help create training datasets which can feed back into ARCADE.

Despite the limitations this release of ARCADE is an important first step. We think it gives an alluring glimpse into the potential of computer vision to assist with investigative work. We hope that you do too and use it to kickstart the debate about the future toolsets investigators will need, and the types of skills and partnerships that we will need to create them.

Can I use or hack or contribute to ARCADE?

We hope so! If you live and breath computer vision and want a challenge, we know ARCADE could be just that.

ARCADE’s source code and a standalone application are available on Github, along with detailed installation and use guidance. ARCADE’s source and compiled application is licensed under the BSD simplified license. To adapt it you will access to a recent copy of MATLAB.

Should the tool prove useful we plan to move it to a free and open source toolset based around OpenCV. If you’d like to help us do that, get in touch.

Acknowledgements

ARCADE has been made possible through the skills and imagination of Professor Hassan Ugail, Ali M. Bukar and Shelina Jilani from the Centre for Visual Computing at the University of Bradford, and draws heavily on the investigative work of Eliot Higgins and the team at Bellingcat. Rudiment is also deeply grateful for support for our work as a Small Innovation Project from the Yorkshire Innovation Fund.