Hands-on Coding Session on Metamorphic Testing of Neural Networks

Automating large-scale test case generation requires using multiple image transformations to build a large number of metamorphic transformations and their follow-up tests, in order to find DNN’s erroneous behaviors. In fact, the metamorphic transformation should be designed so that the transformed and original inputs are semantically equivalent. Thus, the first coding task is to implement metamorphic transformations on the input, assembling all the provided image-based transformations to guarantee the diversity of generated inputs. In order to estimate the level of logic explored by inputs, neuronal coverage criteria are employed, which estimate the amount of activation states covered by inputs. It is therefore essential that we store each valid synthetic input that fools the DNN under test. Hence, the second coding task is to develop the follow-up test that takes the logits returned by the DNN for the transformed data to check if the test fails or succeeds, then stores the synthetic images corresponding to the failed tests.