More info on attack / algorithm parameters #59
Replies: 2 comments
-
Hi @pstarconsult, In Counterfit, each attack has an attack definition file the determines the type of arguments an attack supports, default values, and documentation for the parameter. For example, in the case of Hop Skip Jump, we have the attack definition file https:/Azure/counterfit/blob/main/counterfit/frameworks/art/attacks/hop_skip_jump.yml # https:/Azure/counterfit/blob/f3203dc3cebb816ff877d0cb341501420e182d10/counterfit/frameworks/art/attacks/hop_skip_jump.yml
attack_category: evasion
attack_class: art.attacks.evasion.hop_skip_jump.HopSkipJump
attack_data_tags:
- image
- tabular
attack_docs: "\n Implementation of the HopSkipJump attack from Jianbo et al. (2019).\
\ This is a powerful closed-box attack that\n only requires final class prediction,\
\ and is an advanced version of the boundary attack.\n\n | Paper link: https://arxiv.org/abs/1904.02144\n\
\ "
attack_name: hop_skip_jump
attack_parameters:
batch_size:
default: 64
docs: The size of the batch used by the estimator during inference.
optimize:
uniform:
max: 1000
min: 1
clip_values:
default:
- 0.0
- 1.0
docs: Refer to attack file.
curr_iter:
default: 0
docs: Refer to attack file.
init_eval:
default: 100
docs: Initial number of evaluations for estimating gradient.
optimize:
uniform:
max: 1000
min: 1
init_size:
default: 100
docs: Maximum number of trials for initial generation of adversarial examples.
optimize:
uniform:
max: 200
min: 1
max_eval:
default: 1000
docs: Maximum number of evaluations for estimating gradient.
optimize:
uniform:
max: 10000
min: 1
max_iter:
default: 50
docs: Maximum number of iterations.
optimize:
uniform:
max: 1000
min: 1
norm:
default: 2
docs: 'Order of the norm. Possible values: "inf", np.inf or 2.'
optimize:
choice:
inf: "inf"
targeted:
default: false
docs: Should the attack target one specific class.
optimize:
bool:
"true": true
"false": false
verbose:
default: true
docs: Show progress bars.
attack_type: closed-box When running from the terminal, Counterfit will automatically pick up the attack definition file and display the information with the
Let me know if this helps! |
Beta Was this translation helpful? Give feedback.
-
Thank you, Gary, that does help a lot. There is one remaining ambiguity to the extent you can answer it (or know where to find the answer). The way I see it:
The ambiguity is if steps #1-#3 constitute one iteration. And, how does this reconcile with init_size (Maximum number of trials for initial generation of adversarial examples)? Is a trial somehow related to an iteration? Is it fair to assume that a trial is what steps #1-#5 consist of? And, last, I see 'average number of queries" in other attacks. I am left with confusion about the term 'query' and how that fits into the trial/iteration definitions as above. Sorry for the continued inquiry but I think it is important for users to know how the attacks are conducted so that we can estimate the risk properly. In other words, I am sure I will not be the only one with these issue(s) so your work here will be a "do once, reap many" effort, lol. Thank you so much for all that you do! Paul |
Beta Was this translation helpful? Give feedback.
-
I am trying to find definitions and distinctions between the algorithm parameters below (this is just an example that I copied out from something I am testing). Primarily, what is the difference between max number of iterations, number of queries (not listed below), sample size and number of trials? Also, what is the sample index? I am quite sure that batch_size, clip_values and (maybe) sample_size are related to settings for the DNN. I did not find anything on the wiki or elsewhere to get these answers. I have been separately (and previously) testing textattack and ART but I like the combined frameworks in Counterfit. I greatly appreciated whatever can be provided on these questions. Also, I am more than happy to contribute to include filing (and fixing?) issues or reporting test results (only significant results, lol). Thank you!
Algorithm Parameters │ │ │ │
│ -------------------- │ -- │ -- │ -- │
│ targeted (bool) │ False │ False │ │
│ delta (float) │ 0.01 │ 0.001 │ │
│ epsilon (float) │ 0.01 │ 0.001 │ │
│ step_adapt (float) │ 0.667 │ 0.1 │ │
│ max_iter (int) │ 5000 │ 100 │ 1000 │
│ num_trial (int) │ 25 │ 25 │ │
│ sample_size (int) │ 20 │ 20 │ │
│ init_size (int) │ 100 │ 100 │ │
│ batch_size (int) │ 64 │ 64 │ │
│ verbose (bool) │ False │ False │ │
│ clip_values (tuple) │ (0.0, 1.0) │ (0.0, 1.0) │ │
│ │ │ │ │
│ Attack Options │ │ │ │
│ -------------------- │ -- │ -- │ -- │
│ sample_index (int or expr) │ 0 │ 0 │ │
│ logger (str) │ default │ default │
Beta Was this translation helpful? Give feedback.
All reactions