TEL-830 - Pattern Recognition

The Pattern Recognition course introduces students to fundamental principles and advanced methodologies for automated decision-making systems. Students explore classification, feature extraction, dimensionality reduction, and clustering techniques, emphasizing both theoretical frameworks and practical applications. The course equips students to design and evaluate pattern recognition systems for diverse real-world problems, including image, speech, and biomedical signal analysis.

Key Topics Include:

  1. Classifiers Based on Bayes’ Decision Theory
    Bayes decision theory. Discriminant functions and decision surfaces. Bayesian classification for normal distributions. Estimation of unknown probability density functions: Maximum Likelihood (ML), Maximum A Posteriori (MAP), and Maximum Entropy methods. Mixture models and non-parametric estimation. Nearest neighbor rule. Bayesian networks.
  2. Linear Classifiers
    Linear discriminant functions and decision hyperplanes. Perceptron algorithm and least-squares methods. Logistic regression. Support Vector Machines (SVM) for linear classification.
  3. Non-Linear Classifiers
    The XOR problem and limitations of linear classifiers. Multi-layer perceptrons (two-layer and three-layer networks). Backpropagation algorithm and its variations. Generalized linear classifiers and polynomial classifiers. Radial Basis Function (RBF) networks and probabilistic neural networks. Non-linear SVMs. Decision trees and ensemble methods (combining classifiers).
  4. Feature Selection
    Preprocessing and feature selection techniques. Statistical hypothesis testing. ROC curves and class separability measures. Feature subset selection. Optimal feature generation using neural networks.
  5. Feature Generation and Dimensionality Reduction
    Data transformations and basis vectors. Karhunen-Loeve Transform (Principal Component Analysis – PCA). Independent Component Analysis (ICA). Non-negative Matrix Factorization (NMF). Non-linear dimensionality reduction techniques. Discrete Fourier Transform (DFT), Discrete Cosine Transform (DCT), Hadamard, and Haar transforms.
  6. Template Matching
    Optimal path search techniques. Dynamic Time Warping (DTW) for speech recognition. Measures based on correlations. Content-based information retrieval.
  7. Supervised Learning
    Error-counting approaches and case studies. Supervised learning applications in real-world problems, such as medical imaging.
  8. Clustering – Basic Concepts
    Introduction to clustering and its applications. Proximity measures between points, point-to-set, and set-to-set measures.
  9. Clustering Algorithms – Sequential Methods
    Sequential clustering algorithms. Estimation of the number of clusters and two-threshold sequential schemes. Refinement stages and neural network implementations for clustering.
  10. Clustering Algorithms – Optimization-Based Methods
    Mixture decomposition schemes. Fuzzy clustering algorithms. Hard clustering methods, including the K-Means algorithm.
  11. Clustering Algorithms – Advanced Methods
    Competitive learning algorithms. Self-Organizing Maps (SOM). Density-based clustering for large datasets. Clustering for high-dimensional data.
  12. Applications of Pattern Recognition
    Real-world applications of pattern recognition techniques, including: Speech recognition, Image recognition, Biomedical signal recognition.

Laboratory Topics

The laboratory component focuses on hands-on implementation of key algorithms using Python (scikit-learn), MATLAB, and other open-source tools. Key topics include:

  1. Classifiers Based on Bayes’ Decision Theory: Implementation of Bayesian classifiers and nearest neighbor rules.
  2. Classifiers Based on Cost Function Optimization: Design of linear and nonlinear classifiers such as perceptrons and support vector machines (SVM).
  3. Feature Generation and Dimensionality Reduction: Implementation of Karhunen-Loeve Transform (PCA), ICA, and other transformations.
  4. Feature Selection: Methods for identifying and optimizing relevant features.
  5. Template Matching: Optimal path search techniques, including Dynamic Time Warping (DTW).
  6. Clustering: Algorithms like K-Means, density-based clustering, and self-organizing maps (SOM).

The lab exercises emphasize practical applications such as object recognition, speech analysis, and unsupervised learning tasks.

Available Resources

Slides of Theory: Comprehensive lecture slides are available for download, covering all course topics:

  1. Lecture 1: Probabilistic Classifiers
    Classifiers based on Bayes’ decision theory, discriminant functions, and probability estimation.
  2. Lecture 2: The Perceptron
    Linear classifiers, perceptron algorithm, and decision hyperplanes.
  3. Lecture 3: Support Vector Machines (Linear)
    Linear Support Vector Machines (SVM) and optimization of decision boundaries.
  4. Lecture 4: Support Vector Machines (Non-Linear)
    Non-linear SVMs, kernel methods, and classification in higher dimensions.
  5. Lecture 5: The Perceptron Classifier
    Multi-layer perceptrons (MLP) and backpropagation for non-linear classification.
  6. Lecture 6: The RBF Classifier
    Radial Basis Function (RBF) networks and their applications for classification.
  7. Lecture 7: Feature Generation
    Data transformations, basis vectors, and dimensionality reduction techniques (PCA, ICA).
  8. Lecture 8: Feature Selection
    Techniques for feature selection, statistical testing, and class separability measures.

GitHub Repository: Access our dedicated repository for simulation programs in MATLAB/Octave and Python, categorized by chapter, to complement your understanding of theory and lab topics.

Student Projects

Students are encouraged to apply their knowledge through semester-long projects. Below are some examples of proposed projects for this course:

  1. Design and Implementation of an Automated Facial Recognition System:
    Create a system using OpenCV and TensorFlow to detect and recognize faces in real-time, including maintaining a database of authorized individuals for access control applications.
  2. Real-Time Object Detection for Autonomous Driving:
    Implement a system that uses pre-trained neural networks to identify and classify objects such as vehicles, pedestrians, and traffic signs for autonomous navigation.
  3. Health Monitoring for Plants:
    Develop a system using computer vision techniques to monitor plant health by detecting leaf diseases or stress. This project integrates image preprocessing and convolutional neural networks (CNNs).
  4. Hand Gesture Recognition and Finger Counting:
    Create a gesture-based control system using OpenCV and deep learning frameworks like TensorFlow to identify hand movements and count fingers in real-time.
  5. Traffic Light Recognition from Real-World Video Data:
    Build a system to detect and recognize traffic lights in video streams, addressing challenges such as motion blur and varying lighting conditions.
  6. Real-Time Pose Detection:
    Implement a body pose detection system to analyze and track human body movements using frameworks like OpenPose.

Equipment

The Pattern Recognition course leverages cutting-edge hardware and software to integrate theoretical learning with practical experience:

  • Google Colab: Provides cloud-based Python programming environments for developing and testing algorithms without requiring local resources.
  • Python and Libraries: Tools like scikit-learn, OpenCV, and NumPy are used for implementing supervised and unsupervised learning techniques, feature extraction, and classification tasks.
  • MATLAB: Used for prototyping and validating advanced algorithms for feature selection, dimensionality reduction, and clustering.
  • Raspberry Pi and NVIDIA Jetson Nano/Xavier NX: These devices enable students to test pattern recognition algorithms on edge-computing platforms for real-world applications like object recognition and real-time classification.
  • HuskyLens AI Camera: This AI-powered vision sensor supports practical applications such as gesture recognition, object classification, and facial detection.

These tools are used to implement algorithms, test performance on real-world data, and solve practical problems in domains like image processing, speech recognition, and biomedical signal classification.

Recommended Bibliography

The following resources are recommended for a deeper understanding of the concepts covered in the Pattern Recognition course:

  • S. Theodoridis, K. Koutroumbas, Pattern Recognition, Fourth Edition, Elsevier, Academic Press, 2008.
  • C. Bishop, Pattern Recognition and Machine Learning, Springer, 2007.
  • S. Theodoridis, A. Pikrakis, K. Koutroumbas, D. Cavouras, Introduction to Pattern Recognition: A MATLAB Approach, Academic Press, 2010.

These resources are available in the university library or can be accessed through online platforms for supplementary reading. Students are encouraged to refer to these books for additional examples, explanations, and problem-solving techniques.

Lectures are held weekly on Spring Semester, Mondays 09:00 – 12:00 at Room Η2.02.
Please refer to the official timetable on e-Class (access for registered users only) for the most up-to-date details

 
Weekly Schedule

Week 1 (Part 1)
Bayes Decision Theory and Classification.

Week 2 (Part 2)
Bayes Decision Theory – Estimation methods (ML and MAP).

Week 3 (Part 1)
Linear Classifiers – Perceptron, hyperplanes, and least-squares methods.

Week 4 (Part 2)
Linear Classifiers – Logistic regression and Support Vector Machines (SVM).

Week 5 (Part 1)
Non-Linear Classifiers – Multi-layer perceptrons and backpropagation.

Week 6 (Part 2)
Non-Linear Classifiers – Radial Basis Function (RBF) networks and decision trees.

Week 7 (Part 1)
Feature Selection – Statistical hypothesis testing, ROC curve, and separability measures.

Week 8 (Part 2)
Feature Selection – Subset selection and optimal feature generation.

Week 9 (Part 1)
Feature Generation – Karhunen-Loeve transform (PCA), ICA, and DFT/DCT transforms.

Week 10 (Part 2)
Dimensionality Reduction – Non-linear methods and matrix factorization.

Week 11
Template Matching – Optimal path searching and Dynamic Time Warping (DTW).

Week 12
Clustering – K-Means, probabilistic clustering, and density-based clustering.

Week 13
Applications – Real-world problems in image recognition, speech analysis, and biomedical signals.