Introduction

The project focuses on the major scientific issues of the correlation of persona-behavior-scene visual representation and understanding, develops a visual understanding application platform for social governance intelligence. We form major scientific and technological achievements that benefit people's lives, such as health codes, finding and preventing human trafficking, and digital financial remote services, and provide a common technical guarantee for the intelligent innovation model of social governance.

Intelligent Social Governance based on the Correlation of Persona-Behavior-Scene Visual Representation

The team has overcome several international challenges in the field of intelligent social governance, achieving key technological innovations.

1) High Precision Face Verification Methods: The project pioneered high-precision face visual recognition algorithms through ultra-precision identity verification, multi-level subcontract across time and space efficient face verification, and fine-grained facial feature analysis. These innovations have supported identity verification for 100 million users in financial payment systems and 10 million cross-temporal person searches, establishing a global leadership position.

2)Behavioral Cognitive Mechanism and High Safety Understanding: The project introduced a novel concept of behavior understanding, proposed the cognitive mechanism of social behavior, and developed anomaly behavior detection and defense technologies, greatly enhancing behavior security and management precision in social governance.

3)Efficient Scene Correlation Analysis: The project built an efficient platform for scene association analysis, incorporating scene structure representation, 3D scene understanding, and reconstruction, situational tracking, etc., which effectively supported major applications in intelligent social governance and pandemic prevention and control.

The team has published 318 papers in Nature and top-tier journals, with 298 patents granted (including 127 international patents). Some of these achievements obtained the Special Prize of the 2020 Shanghai Science and Technology Progress Award and the First Prize of the 2024 Wu Wenjun 2024 Artificial Intelligence Natural Science of CAAI.

Intelligent Social Governance Services in Finance, Public Health, and Other Fields

The project has achieved widespread application and significant achievements in the field of human behavior and scene analysis technology, particularly in safeguarding people's lives and property. The ultra-high precision identity verification technology was first applied by WeBank in 2015, driving the development of new business models in Internet banking. It has been widely adopted by platforms like WeChat and QQ, spanning over 140 industries and successfully expanding into international markets. The cross-age facial recognition system for locating missing persons has helped China recover 1,713 individuals, marking the largest success case for the State Council's anti-trafficking office. The project's anti-spoofing technologies played a key role in the nationwide rollout of the first health code, which, by 2022, had served 1.4 billion users with over 81.5 billion health code scans. Behavior understanding and scene analysis technologies have enhanced social security, with the first China International Import Expo achieving seamless entry, earning high praise.

The project has generated an additional output value of 2.719 billion RMB, reduced costs by 7.566 billion RMB, and significantly improved social governance and public safety, delivering immense social and economic benefits.

This project has made significant breakthroughs in areas such as health codes, smart communities, Al-based missing person searches, and digital finance, driving technological advancements and optimizing industrial structure upgrades. During the COVID-19 pandemic, the project innovatively applied remote identity verification technology, contributing to the rapid adoption of health codes, and received letters of appreciation from various government departments. The AI missing person search system has successfully assisted local police in locating over a thousand missing individuals, addressing challenges related to identity verification for those unable to do so due to health or age, resulting in tremendous social benefits. The financial remote identity verification system developed by the project was commercially implemented in the banking sector for the first time, fostering the creation of new Internet finance models and promoting the intelligent transformation of the financial sector. Moreover, the smart community governance system has enhanced urban public safety, successfully being deployed in several large-scale events. The remote conferencing and education platforms, utilizing technologies such as portrait segmentation, have supported large-scale, contactless work models, with over 4 billion annual participants, making it the largest remote conferencing system in China. These achievements have significantly advanced technological progress, optimized industrial structures, and facilitated the transition toward a more intelligent and digital society.

Academic Evaluation and Technical Appraisal

The Science and Technology Achievement Appraisal Meeting, organized by the China Electronics Society, concluded that the project “Key Technologies and Applications for Visual Understanding in Intelligent Social Governance,” completed by Shanghai Jiao Tong University and other institutions, demonstrates high technical complexity and significant innovation, with substantial challenges in development. The overall technology has reached an advanced international level, with specific technologies such as remote liveness detection under varying lighting conditions and cross-temporal and multimodal human feature understanding achieving global leadership. The international academic community has highly praised several aspects of the project team's research, and some achievements have been awarded prestigious honors, including the Special Prize of the Shanghai Science and Technology Progress Award, the First Prize of Wu Wenjun Artificial Intelligence Natural Science of CAAl, the China Computer Graphics Excellence Award, and the Science Exploration Award.