Image processing

The intended surveillance system is designed to operate outdoors in any condition of brightness and weather. However, testing a dataset is usually not enough to validate the performance on every condition and even less to validate it on every combination of conditions. The real life scenario might not be covered by the testing dataset. As a result the performance of the system may vary greatly with no explanation once in production.

To avoid such situations any engineer should increase its database coverage, either manually or through data augmentation. But this is very time consuming. With Saimple it is possible to test the robustness of your neural network type (for convolutional neural network or recurrent neural network for instance) against a variety of perturbation automatically. In this example let’s assume that the camera can suffer from blurry inputs (Fig 1.) or the presence droplets of water on the protective glass (Fig 2.). Normally an engineer would have to perturb all images in the dataset using multiple settings of the noise Normally an engineer would have to perturb the data training set by adding noise to all images (different settings of the blurring kernel, or different numbers of droplets). This would require an exponential number of tests, but it won’t be able to prove whether or not the system is performant beyond this test. For that you engineer need Saimple to cover all the space around the images he has with all the possible noise he can apply (Fig 3.).

Let us assume that the camera use can add up to a certain maximum intensity of blurring noise. With Saimple, the engineer can model this blurring noise using standard libraries. Then he uses Saimple to check the robustness of its networks on each image he has. Saimple will automatically prove whether or not the classifier will be correct for any noise up to a certain maximum value of intensity. He has no need to build an exponential testing process for each image, he just has to prove once the robustness of each image

(Fig.3 - Multiple noises in on one image contained in our Abstract Domain)

But not all noises are mathematical, for example droplets on images can be challenging to model. For those Saimple allow the use of masks to model specific noises applied onto an image, for example droplets (Fig 4). The tool will help prove that the classifier can handle the application of a mask. After setting up a mask the engineer can see how much the classifier resists the perturbation model by the mask (Fig 5).

(Fig. 1 - CCTV image with gaussian blur)

(Fig. 2 - CCTV image with droplet mask)

(Fig. 4 - Droplet mask)

Knowing when the network classification can fail gives the engineer the knowledge how much noise can the system handle (Fig 5). It is critical to know as soon as possible if a system will resist to avoid costly deployment phases that could fail

Using Saimple the engineer was able to check during the design phase if its system will handle the volume of noise the camera can add. He was also able to test before putting in production the limit of its system to environmental perturbations.

Changing hardware between the training phase and the production phase can imply severe cutbacks to the performance of neural networks. As artificial neural networks adapt their training to the precision they have, modifying it is bound to have an impact on their performance. Modern frameworks for training neural networks do not consider this impact since for them production and design environnement are on one and the same. To better manage your operational risks, Saimple helps your engineer measure the impact of the hardware on performance. Specifically it can check the robustness even when changing the underlying numerical precision. It allows engineers to anticipate and adapt both training and architecture to better match the hardware specifications down the road. 

Assuming that the classification system was trained on 64-bits architecture it takes a lot of memory space. To allow integration into the camera the system has to use only 16-bits operations. Changing just before going into production the precision causes a drop in the performance of the classifier of 3.9% (Fig. 1). Using Saimple earlier on this situation would have been avoided since the robustness of this neural network was not guaranteed upon a change that goes as down as to 16-bits operations. However using Saimple the engineer can find how mixed precision can be used to have both a reduction of the size of the neural network and the conservation of performance (Fig. 2). Thus avoid costly and time-consuming retrofit actions which are needed otherwise. 

(Fig. 1 - Evolution of the classification performance vs the precision used and memory usage)

(Fig. 2 - Evolution of the classification performance vs. also with a mixed precision)

Using Saimple your engineer can test ahead the integration in your system and avoid a retrofit. When the hardware constraint degrades too much the performance, Saimple helps by automatically finding a tradeoff between performance and memory size of the neural network. Assuming that the classification system was trained on 64-bits architecture it takes a lot a memory space. To allow integration into the camera the system have to use only 16-bits operations. Changing just before going into production the precision causes a drop in the performance of the classifier of 3.9% (Fig. 1).

By using Saimple earlier on this situation would have been avoided since the robustness of this neural network was not guarantee upon a change that goes as down as to 16-bits operations. However but using Saimple the engineer can found how mixed precision can be used to have both a reduction of the size of the neural network and the conservation of performance (Fig. 2). Using Saimple your engineer can test ahead the integration in your system and avoid a retrofit. When the hardware constraint degrade to much the performance Saimple helps by finding automatically a tradeoff between performance and memory size of the neural network.