APSD README

The project's README from github.com/hcliucs/APSD. As one markdown file: README.md.

APSD

Stealthiness Assessment of Adversarial Perturbation: From a Visual Perspective

Paper Dataset License

IEEE Transactions on Information Forensics and Security (TIFS), 2024

Hangcheng Liu · Yuan Zhou · Ying Yang · Qingchuan Zhao · Tianwei Zhang · Tao Xiang

Paper · Dataset · Citation

🔍 Overview

How stealthy is an adversarial perturbation to the human eye?

Widely used metrics such as (L_p) norms, PSNR, and SSIM were not designed specifically for subtle adversarial perturbations and can disagree with human visual perception. This work provides a systematic study of pixel-level adversarial stealthiness assessment from both subjective and objective perspectives.

We introduce:

✨ Highlights
👁️ Human-centric assessment A two-step protocol provides fine-grained subjective scores across the full stealthiness spectrum.
🗂️ 10,586 adversarial examples APSD contains 10,586 clean–adversarial pairs generated from 400 reference images.
⚔️ 12 attacks / 5 attack families Gradient-, optimization-, generative model-, pixel-, and block-based attacks are covered.
👥 60 human observers Every adversarial example is evaluated by 15 observers; Mean Opinion Score (MOS) is used as the label.
🧠 Human-inspired objective model A2SM combines global/local features with self-, cross-, and spatial-attention mechanisms.
📈 High HVS consistency A2SM achieves 0.984 PLCC / 0.978 SROCC on APSD.

👁️ Subjective Stealthiness Assessment

We propose a two-step subjective assessment protocol.

Step 1 — Rough assessment

Observers first inspect a clean/adversarial image pair for a fixed period and classify the perturbation as:

Step 2 — Fine-grained assessment

For strong stealthiness, observers use progressively amplified Scaled Grayscale Difference Maps (SGDM) to distinguish subtle pixel-level differences.

For weak stealthiness, observers assess the degree of visible degradation / annoyance.

The resulting scores are aggregated into a Mean Opinion Score (MOS) in the range ([0,1]), where a larger value means better stealthiness.

Subjective adversarial stealthiness assessment interface

Rating software.

🗃️ APSD Dataset

We construct the Adversarial Perturbation Stealthiness Assessment Dataset (APSD), a dedicated benchmark for human-aligned adversarial stealthiness evaluation.

Dataset statistics

Property APSD
Reference images 400
Clean–adversarial pairs 10,586
Image categories 200+
Attack methods 12
Attack families 5
Human observers 60
Ratings per adversarial example 15
Label MOS ∈ [0, 1]
Image size used for subjective study 512 × 512

Covered attack families

APSD contains perturbations spanning a broad range of perceptual stealthiness, allowing models to learn both subtle and visually obvious adversarial distortions.

Representative APSD adversarial examples

Distribution of adversarial examples generated by different attacks in APSD. The suffix of each attack indicates the corresponding parameter.

📥 Dataset Download

The APSD dataset is publicly available through the project repository:

➡️ Download APSD

The released dataset should contain the clean images, adversarial images, and their corresponding human subjective scores.

A recommended local organization is:

APSD/
├── REF-299 (Reference)/
│   ├── xxx.png
│   └── ...
└── AE_targetmodel0 (Adversarial example)/
    ├── 0 FGSM/
    ├── 1 MIFGSM/
    ├── 2 PGD/
    ├── 3 NES/
    ├── 4 CW/
    ├── 5C DP/
    ├── 6 AdvGAN/
    ├── 7 GAP/
    ├── 8 GUAP/
    ├── 9 SimBA/
    ├── 10 Pixel/
    └── 11 Square/
    └── score.csv

Note: the exact released directory/file names should follow the actual dataset package. Update the tree above if your final release uses a different structure.

📝 Citation

If you find APSD or A2SM useful in your research, please cite:

@article{liu2024stealthiness,
  title   = {Stealthiness Assessment of Adversarial Perturbation: From a Visual Perspective},
  author  = {Liu, Hangcheng and Zhou, Yuan and Yang, Ying and Zhao, Qingchuan and Zhang, Tianwei and Xiang, Tao},
  journal = {IEEE Transactions on Information Forensics and Security},
  volume  = {20},
  pages   = {898--913},
  year    = {2024},
  doi     = {10.1109/TIFS.2024.3520016}
}

📬 Contact

For questions about the paper, dataset, or code, please open a GitHub issue or contact the authors.

License

Please add the license that applies to the released code and dataset before public release.
If the dataset contains images inherited from ImageNet, COCO, or VOC2012, their original licenses/terms should also be respected.

APSD (hcliucs/APSD)

Searching Database…

897 ·GitHub
APSD
What is this?APSD is a large-scale dataset for studying how visually stealthy adversarial perturbations look to humans, containing 10,586 clean-adversarial image pairs with human opinion scores.
Is this a dataset only, or does it also include the A2SM model code?It is primarily the APSD dataset itself (images plus human MOS scores); the paper also describes an A2SM assessment model, but this repo is framed as the dataset release, not training code.
How do I get the dataset?Download APSD from the project repository link referenced in the README; the release should contain the clean images, the corresponding adversarial images, and their human subjective scores.
Is the released dataset structure finalized?The README itself warns that the shown folder layout is only a placeholder and the actual released files may be organized differently.
Which attacks are covered by APSD?APSD spans 12 attack methods across 5 families: gradient-based (FGSM, MIFGSM, PGD, NES), optimization-based (CW), generative-model-based (CDP, AdvGAN, GAP, GUAP), pixel-based (SimBA, Pixel), and block-based (Square).
How were the human stealthiness scores collected?60 observers rated each of the 10,586 adversarial examples in a two-step protocol (coarse hardly/readily perceptible classification, then fine-grained scoring with amplified difference maps), and 15 ratings per example are averaged into a Mean Opinion Score (MOS) between 0 and 1.
How good is the companion A2SM model at predicting human perception?A2SM, which combines global/local features with self-, cross-, and spatial-attention, reaches 0.984 PLCC and 0.978 SROCC against human MOS scores on APSD.
Where was this research published?It is the official dataset for the paper 'Stealthiness Assessment of Adversarial Perturbation: From a Visual Perspective,' published in IEEE Transactions on Information Forensics and Security (TIFS), 2024.
What are similar projects?reverse-SynthID, Engram, math, arc-task-gen, pointdit, transferlearning, Mira-Scene, High-Angle_Robust_Fast_FaceAlignment
Which topics is it in?adversarial-attacks, vision, security-research, ai-security, attention-mechanism, papers, paper, dataset, evaluation, neural-network, academic-research, image-processing
Creator
hcliucs
Size
1.3 MB
Stars
897
Updated
CommitSep 02
Age
3y
Commits · 90 Days