Categories

Follow Us

Share Post

Share on facebook
Share on twitter
Share on linkedin
Share on whatsapp

Using AutoML for fast and efficient Image Classification

Florian Marty

Written by:
Florian Marty (Sales Manager at Modulos)

A microcrack in a concrete surface.

Modulos AutoML allows building state-of-the-art machine learning solutions in a fully automated manner. To further illustrate this, we’ll share a use case where we performed an image classification experiment. Modulos AutoML handles image data, tabular data, as well as a combination of both.

Imbalanced data sets can result in deceptively good scores (e.g., accuracy) for binary classifications. Adjusting the training objective can therefore improve the score of a model on test data. 

For this blog post, we used a publicly available dataset described by the data source as:

divided into two as negative and positive crack images for image classification. Each class has 20000 images with a total of 40000 images with 227 x 227 pixels with RGB channels. The dataset is generated from 458 high-resolution images (4032×3024 pixel) with the method proposed by Zhang et al. (2016). High-resolution images have variance in terms of surface finish and illumination conditions. No data augmentation in terms of random rotation or flipping is applied.

Özgenel et al. (2018) http://doi.org/10.1109/ICIP.2016.7533052

Introduction 

Increased manufacturing automation leads to a growing number of produced parts that need to be screened for potential faults. Additionally, detecting faults using different imaging modalities (e.g., cracks in different materials) is becoming crucial in preventive maintenance and reduction of system downtime.

A major issue with detecting faulty parts is the massive amount of data where the vast majority of the components are satisfactory, and only a minute fraction (1:100 to 1:100 000) is faulty, or in our case, cracked. This is a challenge we have to overcome when creating our machine learning model.

Moreover, the faults or cracks are often smaller than the human eye can detect, which further raises the challenge. The so-called microcracks can, when left undetected, have a devastating effect. Imagine the consequences of missing a microcrack in a wing of a plane. As multiple incidents from the past have shown, such a fault can have fatal consequences. 

From a technical perspective, the two aforementioned issues – massive amounts of data and microcracks, can be successfully dealt with using artificial intelligence augmented decision support.

However, when setting up such a system from a perspective of a data scientist or a machine learning engineer, we should keep in mind these two possible threats:

  1. Highly imbalanced data may result in overly optimistic scores (e.g., accuracy) when the wrong training and evaluation objectives are chosen.
  2. False negatives can have catastrophic consequences, whereas false positives are often neglected.

Data Preparation and Model Training

We first downloaded the images from the source and manually resized them to 128×128 pixels. Manual resizing was used to overcome the limitations of the identity transformation feature extractor implemented in Modulos AutoML v.0.3.5.

While it may be counterintuitive, for this experiment, the positive class is defined as the class containing images with a crack, while the negative class is the one containing images without a crack. 

We randomly selected 19 478 negative (no crack) and 121 positive (crack) from all images to make up the training dataset, resulting in a 160:1 imbalance between the two classes. We applied no other data manipulation, preprocessing, or augmentation.

The resulting .tar archive of 93MB was uploaded to Modulos AutoML v.0.3.5 running on an Amazon AWS instance with 1 GPU. We set AutoML to use the Bayesian optimization strategy, with the F1 binary as the objective function. AutoML splits the data into an 80% training data set, and a 20% validation data set used to evaluate each machine learning solution.

Definitions:

  • True Positive (TP): Correctly predicting a class 1 (we predicted “crack,” and it’s “crack”),
  • True Negative (TN): Correctly predicting the other label (we predicted “no crack,” and it’s “no crack”),
  • False Positive (FP): Falsely predicting a label (we predicted “crack,” but it’s “no crack”),
  • False Negative (FN): Missing an incoming label (we predicted “no crack,” but it’s “crack”).

Results and Discussion

After 17 hours of training we manually paused the workflow to test the model performance on hold-out test data. For this test we used solution plan number 18. Investigation of solutions number 20 and 11 showed that “central cropping” as a feature engineering step removed cracks that were not centered on the image, producing many false negatives. As the same feature engineering is applied to the test data set, we only spotted this behaviour by manually investigating the results. This example highlights the need for domain knowledge when building ML solutions.

Figure 1: Overview of Modulos AutoML solutions tab. Inlet representing the solution insights for solution plan number 18 with the early stopping implementation. The dashed line at Epoch 20 shows the epoch with the best validation score.

The batch client (Jupyter Notebook) provided with each Modulos AutoML solution generated predictions. In a next step, we evaluated these predictions vs the known labels from the hold-out test data.
We achieved an F1 score binary of 0.9206 (validation score 0.982), indicating that our model generalizes well for the unseen test data. 

Further, as shown in Figure 2, the Accuracy (0.9956) is very high, despite 4 out of 33 true samples being classified incorrectly. As mentioned above, this is a result of the class imbalance of the data set. Using F1 binary during optimization helped us avoid choosing an inadequate model optimized for accuracy. 

Figure 2: Confusion matrix on hold-out test data (left). Calculation of different scores on the hold-out data (Upper Right). F1 binary of 0.9206 and Matthews Correlation Coefficient of 0.9195 indicating a well performing model that distinguishes positive from negative images. Example of a negative and a positive image from the train data set (Lower Right).

As stated in the introduction, in most similar real-life use cases, we would mainly be concerned about the False Negative Rate (0.1212) as it is often less costly to inspect a satisfactory part that is classified as faulty, than missing a faulty part. 

Of course, there are other situations where both classes are equally important and all four values (TP, FP, TN, FN) matter. For such cases, the Matthews Correlation Coefficient (MCC) can be a valuable additional evaluation criterion when dealing with imbalanced data sets for binary classifications. The MCC takes all four values (TP, FP, TN, FN) into account and ranges from -1 to +1. An MCC close to 1 means that both classes are predicted well, while an MCC close to -1 means that the classes cannot be predicted using the model, and a score close to 0 means that the classifier behaves like a fair-coin.

Key Learnings and Tips

  • First off all, it’s important to always keep test data separated from the training data when testing your model.
  • Choosing the correct objective is essential to generating machine learning models that generalize well for unseen test data.
  • Keep in mind that accuracy can be a misleading score when dealing with highly imbalanced data. If FNs or FPs are less critical, one can select precision, recall, or F1 as a better alternative to accuracy. The Matthews Correlation Coefficient (MCC) is a valuable alternative evaluation metric when all four values from the confusion matrix matter. 
  • Additionally, when dealing with imbalanced data sets, many techniques (e.g., random oversampling, SMOTE) for building better models are at your disposal. However, one should always carefully evaluate models trained on resampled data against the original distribution. Therefore, it is advisable to resample after the train/test split to keep the test data. Lastly, always report scores on the hold-out test data not used for training or evaluation.

In case you have questions on this use case, or you would like to get familiar with the Modulos AutoML platform, get in touch with us. 

If you found this post interesting and useful, feel free to share it!

Share Post

Share on facebook
Share on twitter
Share on linkedin
Share on whatsapp

Modulos Newsletter

Sign up for our newsletter to receive updates on our products and company.

Adrian Weber

Advisory Board

The bad news is time flies. The good news is you’re the pilot

Adrian has more than 18 years of working experience in top-tier management consulting. He has a track record with retail banks, wealth managers, technology providers as well as central banks and regulatory authorities and has worked in more than 20 countries. Recently, after having served as a Senior Partner and Head of Financial Services Switzerland and Head of Global Central Banking Practice at Roland Berger, he co-founded ADVYCE Switzerland, a boutique consulting firm based in Zurich.

Michael Röthlisberger

Michael Röthlisberger

Data Scientist

Data handling with structure

He started to take an interest in Data Science and Software Development during his master’s degree. For his master thesis he worked on the image reconstruction software for a new PET detector. Michael gained some first experience in an internship for Sensirion AG. There he was part of the R&D team, which was developing a new gas sensor. The participation of a machine learning hackathon was sparking the interest of Michael in ML and he decided to pursue a career in this field. He is now exited to face new challenges with modulos and experience working in a rising start-up.

Kevin Schawinski

Kevin Schawinski

CEO / Co-Founder

Running a startup is super relaxing, right?

While a Ph.D student, he co-founded the Galaxy Zoo citizen science project involving more than a million members of the public in scientific research because machines weren’t quite good enough yet to go map the cosmos and classify galaxies. He stayed in Oxford as the Henry Skynner Junior Research fellow at Balliol College before moving to Yale as a NASA Einstein Fellow. In 2012, he started the galaxy and black hole research group at ETH Zurich as an assistant professor and began a close collaboration with Ce Zhang from computer science to work on the space.ml project. He is now the CEO of Modulos.

Ce Zhang

Ce Zhang

Co-Founder

Random is best.

He believes that by making data—along with the processing of data—easily accessible to non-computer scientists, we have the potential to make the world a better place. His current research focuses on building data systems to support machine learning and help facilitate other sciences. Before joining ETH, Ce was advised by Christopher Ré. He finished his PhD round-tripping between the University of Wisconsin-Madison and Stanford University, and spent another year as a postdoctoral researcher at Stanford. His PhD work produced DeepDive, a trained data system for automatic knowledge-base construction. He participated in the research efforts that won the SIGMOD Best Paper Award (2014) and SIGMOD Research Highlight Award (2015), and was featured in special issues including the Science magazine (2017), the Communications of the ACM (2017), “Best of VLDB” (2015), and the Nature magazine (2015).

Alexandra Arvaniti

Alexandra Arvaniti

Operations Manager

“You miss 100% of the shots you don’t take.” – Wayne Gretzky

During the last twenty years, she worked in different roles, setting up and running PMOs, supporting the Executive Management Team or as Operations Manager for the DACH region. She loves all organizational challenges, which she can use well at Modulos, like set up and establish administrative business processes.

Rudolf Bar

Rudolf Bär

Chairman of the Advisory Board

After initially working for Dow Corning International in Zurich and Brussels (1964 to 1969), he held various management functions in the Private Banking Group Julius Baer, Zurich, lastly as CEO from 1993 to 2000 and retired from its Board of Directors in 2005. Since 2014 he has been studying at the Institute for Particle Physics and Astrophysics at the ETH, Zurich.

Marianne Chiesi

Marianne Chiesi

Administration

Marianne has worked in administration of various companies and the ETH.

Marianne Chiesi worked in the administration of various companies before taking time off to raise her children. She translated text books and literary works into Braille and joined the ETH Zurich as an administrative assistant. At ETH, she worked with professorships and researchers in many areas, including astrophysicists, particle physicists and biochemists. She now runs the administration at Modulos.

Bojan Karlaš

Bojan Karlaš

Software Engineer

Real engineers must be a little bit lazy.

After getting a bachelor’s degree in software engineering at the University of Belgrade, Serbia, Bojan spent 2 years working as a developer at Microsoft building distributed database solutions. He then went to Switzerland to pursue a computer science master’s degree at EPFL. He did his master thesis with Ce Zhang at ETH Zürich on the topic of time series forecasting, after which he joined Ce’s group as a PhD student. His industry experience also includes internships at Microsoft, Oracle and Logitech. His research interests revolve around systems and abstractions for making machine learning accessible to non-experts.

Romain Lencou

Romain Lencou

Head of Engineering

Deleted code is debugged code. (Jeff Sickel)

Romain Lencou graduated from the Grenoble Institut National Polytechnique with M.Sc in Computer Science in 2008. Growing up in France in the 90’s, he developed an enthusiasm for pop culture, technology and food. Always eager for technological challenges, Romain worked for companies like VMware, Intel and Logitech, covering various topics including cryptography, virtualization and computer vision. Bitten by the machine learning bug, he is looking forward to apply his problem solving skills in Modulos.

Dominic Stark

Dominic Stark

Data Scientist

Code quality correlates with food quality.

Dominic Stark studied physics at ETH Zürich. The transition of his career path to Data Science began when he was analyzing UV images of galaxies. Together with Kevin Schawinski an Ce Zhang, he worked on applying the latest advances of deep learning research to his problem. It turned out that the method itself was at least as interesting as the problem they designed it for. After publishing the results, his research project was about using Reinforcement Learning to develop novel ideas for data acquisition in astronomy. As a Data Scientist at Modulos, he keeps on solving problems, that require new ideas and technologies.

Andrei Văduva

Andrei Văduva

Senior Software Engineer

The trendsetter geek

He focused his attention on designing Architectures of Computer Systems. During university, he gained an excellent understanding of performance optimization and scalability on architectures such as distributed systems. Having a good experience in various Computer Science fields like big data analytics and Artificial Intelligence, he did his bachelor’s thesis designing a Machine Learning algorithm for social media platforms. After graduation, he joined the investment banking industry, in London, where he gained good experience in designing and building high-quality software. Andrei moved to Switzerland to explore new perspectives and found a great challenge in the startup world. Using his passion for technology and professional experience, he brings the best practices in software engineering to Modulos.

Dennis Turp

Dennis Turp

Data Scientist

Dennis Turp is the first employee of Modulos.

Prior to his work at Modulos he studied physics at ETH Zurich. During his Master studies he worked together with Kevin Schawinski and Ce Zhang on exploring machine learning related topics in astronomy. In these one and a half years they published three scientific papers together. Dennis Turp is currently employed as a Data Scientist. His main expertise lies in the fields of generative modeling and anomaly detection.

Elena Maran

Sales Executive

The bad news is time flies. The good news is you’re the pilot

Elena pursued her studies in Bologna University, UC Berkley and Bocconi University, where she obtained her master in Finance. In 2021, she was awarded Dean’s list honours in an Executive Master in Digital Transformation and Innovation Leadership at the IE, in Madrid. She worked for over twelve years in prestigious international financial institutions such as Citibank and China Construction Bank, in different functions and senior roles spanning from sales and trading to corporate relationship management. She worked and travelled in several countries with multicultural teams and clients. She is now living in Zurich with her husband and young daughter. An ex professional volleyball player, in her free time she now enjoys travelling and discovering off-track places, their traditions and culture.

Modulos appoints Anna Weigel as CTO

Anna Weigel

Chief Technology Officer

After acquiring Bachelor and Master degrees in Physics, Anna completed her PhD in Astrophysics in Kevin Schawinski’s group at ETH. Her work on the relationship between supermassive black holes and their host galaxies is summarized in five first-author papers. After exploring the depths of our Universe, Anna joined Modulos as the Head of Data Science. She has since been appointed the role of CTO and is now leading the overall technology development.

Claudio Bruderer

Claudio Bruderer

Head of Product

Give me coffee to function.

After obtaining a BSc and a MSc degree in physics at ETH Zurich, Claudio decided to continue his studies of the Universe as a PhD student in Prof. Refregier’s Cosmology research group. He studied the gravitational lensing effect, whereby he measured the shapes of several billions of galaxy images (mostly synthetic ones). After acquiring his PhD, Claudio then joined the consulting company AWK Group AG and worked as a project manager and associate for IT and communications projects in the logistics and mobility sectors and for the federal government. Determined to create cutting-edge IT solutions, he decided to join Modulos as a product manager.

Thank you for submitting this form.

Laura Guerrini

Data Science Intern

Laura Guerrini is the first intern of Modulos.

Laura is currently finishing her Master’s in Robotics, Systems and Control at ETH. During her studies, she focused on machine learning, control theory and optimization. She joined Modulos as a Data Science Intern to put theory into practice and boost her machine learning and programming skills.

Jérôme Fischer

Sales Development

“The only way to do great work, is to love what you do.” – Steve Jobs

Jérome Fischer is an expert on Sales. Apart from the successful build up of several companies like the Ad Interim Sales GmbH and the Sales4IT GmbH, he passes on his experience in various Sales and Marketing coachings. Jérome now supports Modulos in establishing the first contact with our customers.

In his free time, Mr. Fischer is an ambitious athlete with numerous awards.

Simone Severini

VP Revenue

“There are always three sides to every story!”

Simone graduated at the Politecnico di Torino in Electronics Engineering and recently gained an Executive MBA at MIP Politecnico di Milano. He has two decades of experience covering various management roles in Sales and Marketing in Fortune 500 companies such as Vodafone, Nokia, Qualcomm, Lear and US west coast tech startups. He travelled extensively for business across 40 countries and currently enjoys his time in Southern Switzerland with his wife and three daughters.

In his free time he enjoys taking off with his paraglider from anywhere high and coding for fun.

Matthias Eppenberger

Board Member

“There are always three sides to every story!”

Matthias studied law and economics at the University of St. Gallen HSG and finished with a PhD in law and bar exam. From 1989 to 1998 he worked for Goldman Sachs in New York, London and Zurich as Executive Director Wealth Management before founding his own firm “Private Client Bank, Zurich” which he ran as CEO from 2004 till 2018. He is still a shareholder of this bank and serves in the board.