Exploring the Fascinating World of Godil - Part 8

Exploring the Fascinating World of Godil - Part 8

Table of Contents:

1. Introduction
2. Overview of Active SDL
3. Purpose of Active SDL
4. Partnership and Progress
5. Types of Evaluations 5.1 Self-Reported Evaluation 5.2 Independent Evaluation
6. Challenges and Disadvantages
7. Tasks and Measures 7.1 Activity Detection Task 7.2 Localization and Confidence Score 7.3 Evaluation Metrics
8. Description of Datasets
9. Results of Active SDL 9.1 Ranking of Participants 9.2 System Performance across Activities 9.3 Real-Time Processing Speed 9.4 System Performance over Time 9.5 Ranking of Activities
10. Conclusion
11. Future Steps

Introduction

Hello, I am Andrew Delgado, and in this article, I'll be presenting the results of the Active SDL task that was conducted under the CVPR20 ActivityNet Workshop. This task was a collaborative effort between IARPA, Kitware, and NIST, with the aim of advancing the development of robust multi-camera automatic activity detection algorithms for forensic and real-time alerting applications.

Overview of Active SDL

Active SDL stands for Activities and Extended Video Sequester Data Leaderboard. It is an evaluation series that has been in progress since 2017. The main objective of Active SDL is to enhance the technology used for automatically detecting target activities and identifying the objects associated with those activities. This project is a partnership between NIST, IARPA, and Kitware.

Purpose of Active SDL

The purpose of Active SDL is to accelerate the development of robust multi-camera automatic activity detection algorithms. By evaluating different systems and their performance in detecting specified activities and localizing them across video sequences, Active SDL aims to drive innovation in this field and improve the efficiency and accuracy of activity detection systems.

Partnership and Progress

The partnership between NIST, IARPA, and Kitware has been fruitful in advancing the Active SDL evaluations and datasets. Over the years, the complexity of the datasets has increased, including more videos, activities, and known/unknown facilities where the activities occur. This progression allows for the evaluation of systems in different scenarios and ensures the development of algorithms that can handle a wide range of real-world situations.

Types of Evaluations

Active SDL conducts two types of evaluations: self-reported and independent. In the self-reported evaluations, participants run their software on their own hardware and submit the system outputs to NIST. In the independent evaluation, participants submit their runable system to NIST, which then runs the system on sequestered data using designated hardware. The results of the independent evaluation are then posted on the Active SDL leaderboard.

Challenges and Disadvantages

While Active SDL has been successful in advancing the field of activity detection, there have been some challenges and disadvantages. Some participants have faced difficulties in configuring their systems to work on the NIST infrastructure. However, NIST actively assists the teams in resolving any errors that occur by providing necessary data and support.

Tasks and Measures

The main task evaluated in Active SDL is the activity detection task, which tests the system's ability to detect the presence of a specified activity and localize it across the video sequence. The system's output includes the start and end frames, indicating the temporal location of the target activity, and a confidence score that reflects the system's level of certainty in detecting the activity. The primary metric used to evaluate systems is the normalized partial area under the curve (NADC). Additionally, real-time speed factor and time limit scoring are reported for systems that operate slower than real-time.

Description of Datasets

The Active SDL evaluation data is Based on the NEVA Multiview Extended Video with Activities (MAVEA) dataset, which was collected by Kitware. This dataset consists of synchronized multi-camera videos that include indoor and outdoor scenes captured by EO and IR sensors. The MAVEA dataset was collected for IARPA's DIVA program and is freely available for research purposes. It contains 37 different activities, including person activity, person-object interaction, and person-vehicle interaction.

Results of Active SDL

The results of the Active SDL evaluations reflect the performance of different systems across the 37 activities in the MAVEA dataset. The evaluation received a total of 40 submissions from 10 organizations and two different countries. The rankings of the systems' performance, relative runtime, and activity detection difficulty were analyzed.

Conclusion

In conclusion, Active SDL is a valuable evaluation series that drives the development of robust multi-camera automatic activity detection algorithms. The collaborative efforts of NIST, IARPA, and Kitware have resulted in significant progress in this field. The evaluation results highlight the strengths and areas for improvement in different systems, ultimately leading to advancements in forensic and real-time alerting applications.

Future Steps

The Active SDL evaluations are still ongoing, and the SDL and TREKVID evaluations are supported through 2022. These evaluations will Continue to push the boundaries of activity detection technology and foster innovation in this domain. The Active SDL team is committed to further improving the evaluation process and datasets to provide valuable insights for researchers and developers in the future.

Highlights:

• Active SDL is an evaluation series aimed at advancing the development of robust multi-camera automatic activity detection algorithms.
• The evaluation includes self-reported and independent evaluations to assess system performance.
• The primary metric used in the evaluation is the normalized partial area under the curve (NADC).
• The evaluation data is based on the NEVA Multiview Extended Video with Activities (MAVEA) dataset, containing 37 different activities.
• The evaluation results showcase the performance and improvements of different systems over time.

FAQ:

Q: What is Active SDL? A: Active SDL stands for Activities and Extended Video Sequester Data Leaderboard. It is an evaluation series focused on enhancing the development of robust multi-camera automatic activity detection algorithms.

Q: What is the purpose of Active SDL? A: The purpose of Active SDL is to accelerate the advancement of activity detection technology for forensic and real-time alerting applications.

Q: What types of evaluations are conducted in Active SDL? A: Active SDL conducts self-reported and independent evaluations to assess the performance of activity detection systems.

Q: What dataset is used in Active SDL evaluations? A: The evaluations are based on the NEVA Multiview Extended Video with Activities (MAVEA) dataset, which contains synchronized multi-camera videos with indoor and outdoor scenes and various activities.

Q: How are the systems evaluated in Active SDL? A: The systems are evaluated based on their ability to detect specific activities and localize them across video sequences. The evaluation metrics include normalized partial area under the curve, real-time speed factor, and time limit scoring.

Q: What are the future steps for Active SDL? A: The Active SDL evaluations will continue up to 2022, aiming to further advance and refine activity detection algorithms.