You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/0_get_started/command_line_usage.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -102,9 +102,9 @@ Here are some models from transformers that have worked well for us:
102
102
-`roberta-base`
103
103
-`xlnet-base-cased`
104
104
105
-
## Evaluating Models with `textattack eval-model`
105
+
## Evaluating Models with `textattack eval`
106
106
107
-
Any TextAttack-compatible model can be evaluated using `textattack eval-model`. TextAttack-trained models can be evaluated using `textattack eval --num-examples <num-examples> --model /path/to/trained/model/`
107
+
Any TextAttack-compatible model can be evaluated using `textattack eval`. TextAttack-trained models can be evaluated using `textattack eval --num-examples <num-examples> --model /path/to/trained/model/`
Copy file name to clipboardExpand all lines: docs/0_get_started/installation.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,7 +1,7 @@
1
1
Installation
2
2
==============
3
3
4
-
To use TextAttack, you must be running Python 3.6 or above. A CUDA-compatible GPU is optional but will greatly improve speed.
4
+
To use TextAttack, you must be running Python 3.9 or above. A CUDA-compatible GPU is optional but will greatly improve speed.
5
5
6
6
We recommend installing TextAttack in a virtual environment (check out this [guide](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/)).
7
7
@@ -67,7 +67,7 @@ Besides, we highly recommend you to use virtual environment for textattack use,
67
67
see [information here](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#removing-an-environment). Here is one conda example:
TextAttack requires both the model and the tokenizer to be wrapped by a :class:`~transformers.models.wrapper.ModelWrapper` class that implements the forward pass operation given a list of input texts. For models provided by Transformers library, we can also simply use :class:`~transformers.models.wrapper.HuggingFaceModelWrapper` class which implements both the forward pass and tokenization.
17
+
TextAttack requires both the model and the tokenizer to be wrapped by a :class:`~textattack.models.wrappers.ModelWrapper` class that implements the forward pass operation given a list of input texts. For models provided by Transformers library, we can also simply use :class:`~textattack.models.wrappers.HuggingFaceModelWrapper` class which implements both the forward pass and tokenization.
Next, let's build the attack that we want to use. TextAttack provides prebuilt attacks in the form of :class:`~transformers.attack_recipes.AttackRecipe`. For this example, we will use :ref:TextFooler attack
24
+
Next, let's build the attack that we want to use. TextAttack provides prebuilt attacks in the form of :class:`~textattack.attack_recipes.AttackRecipe`. For this example, we will use the TextFooler attack.
Copy file name to clipboardExpand all lines: docs/1start/FAQ.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -145,7 +145,7 @@ The `attack_one` method in an `Attack` takes as input an `AttackedText`, and out
145
145
We formulate an attack as consisting of four components: a **goal function** which determines if the attack has succeeded, **constraints** defining which perturbations are valid, a **transformation** that generates potential modifications given an input, and a **search method** which traverses through the search space of possible perturbations. The attack attempts to perturb an input text such that the model output fulfills the goal function (i.e., indicating whether the attack is successful) and the perturbation adheres to the set of constraints (e.g., grammar constraint, semantic similarity constraint). A search method is used to find a sequence of transformations that produce a successful adversarial example.
146
146
147
147
148
-
This modular design unifies adversarial attack methods into one system, enables us to easily assemble attacks from the literature while re-using components that are shared across attacks. We provides clean, readable implementations of 16 adversarial attack recipes from the literature (see [our tool paper](https://arxiv.org/abs/2005.05909) and [our benchmark search paper](https://arxiv.org/abs/2009.06368)). For the first time, these attacks can be benchmarked, compared, and analyzed in a standardized setting.
148
+
This modular design unifies adversarial attack methods into one system, enables us to easily assemble attacks from the literature while re-using components that are shared across attacks. We provides clean, readable implementations of 24 adversarial attack recipes from the literature (see [our tool paper](https://arxiv.org/abs/2005.05909) and [our benchmark search paper](https://arxiv.org/abs/2009.06368)). For the first time, these attacks can be benchmarked, compared, and analyzed in a standardized setting.
Copy file name to clipboardExpand all lines: docs/1start/attacks4Components.md
+46-2Lines changed: 46 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,7 +2,7 @@
2
2
3
3
To unify adversarial attack methods into one system, We formulate an attack as consisting of four components: a **goal function** which determines if the attack has succeeded, **constraints** defining which perturbations are valid, a **transformation** that generates potential modifications given an input, and a **search method** which traverses through the search space of possible perturbations. The attack attempts to perturb an input text such that the model output fulfills the goal function (i.e., indicating whether the attack is successful) and the perturbation adheres to the set of constraints (e.g., grammar constraint, semantic similarity constraint). A search method is used to find a sequence of transformations that produce a successful adversarial example.
4
4
5
-
This modular design enables us to easily assemble attacks from the literature while re-using components that are shared across attacks. TextAttack provides clean, readable implementations of 16 adversarial attacks from the literature. For the first time, these attacks can be benchmarked, compared, and analyzed in a standardized setting.
5
+
This modular design enables us to easily assemble attacks from the literature while re-using components that are shared across attacks. TextAttack provides clean, readable implementations of 24 adversarial attacks from the literature. For the first time, these attacks can be benchmarked, compared, and analyzed in a standardized setting.
6
6
7
7
- Two examples showing four components of two SOTA attacks
@@ -44,7 +44,7 @@ A `SearchMethod` takes as input an initial `GoalFunctionResult` and returns a fi
44
44
45
45
### Four components in Attack Recipes we have implemented
46
46
47
-
- TextAttack provides clean, readable implementations of 16 adversarial attacks from the literature.
47
+
- TextAttack provides clean, readable implementations of 24 adversarial attacks from the literature.
48
48
49
49
- To run an attack recipe: `textattack attack --recipe [recipe_name]`
50
50
@@ -62,6 +62,14 @@ A `SearchMethod` takes as input an initial `GoalFunctionResult` and returns a fi
62
62
<tbody>
63
63
<tr><tdstyle="text-align: center;"colspan="6"><strong><br>Attacks on classification tasks, like sentiment classification and entailment:<br></strong></td></tr>
<td ><sub>Attack tuned for use in adversarial training, from (["Towards Improving Adversarial Training of NLP Models" (Yoo et al., 2021)](https://arxiv.org/abs/2109.00544))</sub></td>
<td ><sub>Greedy attack with word importance ranking , Reducing the input while maintaining the prediction through word importance ranking (["Pathologies of Neural Models Make Interpretation Difficult" (Feng et al., 2018)](https://arxiv.org/pdf/1804.07781.pdf))</sub></td>
<td ><sub>Levy-flight/adaptive-inertia variant of the `pso` search method, from (["LEAP: Efficient and Automated Test Method for NLP Software" (Ma et al., 2023)](https://arxiv.org/abs/2308.11284))</sub></td>
<tr><tdstyle="text-align: left;"colspan="6"><sub>Every other recipe here is locked to one task and one goal function, which is why the table splits into "classification" vs. "sequence-to-sequence" sections. <code>bad-characters</code> is different: its <code>build()</code> takes a <code>goal_function_type</code> argument and can be configured to attack classification models, NER models, or seq2seq/translation models. Since it spans both task types (and NER, which is neither), it doesn't fit either section above, so it's listed here under "General" instead.</sub></td></tr>
@@ -223,6 +240,33 @@ A `SearchMethod` takes as input an initial `GoalFunctionResult` and returns a fi
223
240
<td><sub>Uses imperceptible character-level perturbations including homoglyph substitutions, Unicode reordering, deletions, and invisibles. Based on (["Bad Characters: Imperceptible NLP Attacks" (Boucher et al., 2021)](https://arxiv.org/abs/2106.09898)).</sub></td>
224
241
</tr>
225
242
243
+
<tr><tdstyle="text-align: center;"colspan="6"><strong><br>Multi-lingual attacks on non-English classification models (Python API only, no CLI recipe name -- see <ahref="https://github.com/QData/TextAttack/issues/423">#423</a>):<br></strong></td></tr>
Copy file name to clipboardExpand all lines: docs/1start/what_is_an_adversarial_attack.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -53,7 +53,7 @@ Naturally, many have wondered about what adversarial examples for NLP models mig
53
53
54
54
55
55
56
-
*Two different ideas of adversarial examples in NLP. These results were generated using TextAttack on an LSTM trained on the Rotten Tomatoes Movie Review sentiment classification dataset. These are *real* adversarial examples, generated using the DeepWordBug and TextFooler attacks. To generate them yourself, after installing TextAttack, run ‘textattack attack — model lstm-mr — num-examples 1 — recipe RECIPE — num-examples-offset 19’ where RECIPE is ‘deepwordbug’ or ‘textfooler’.*
56
+
*Two different ideas of adversarial examples in NLP. These results were generated using TextAttack on an LSTM trained on the Rotten Tomatoes Movie Review sentiment classification dataset. These are *real* adversarial examples, generated using the DeepWordBug and TextFooler attacks. To generate them yourself, after installing TextAttack, run ‘textattack attack --model lstm-mr --num-examples 1 --recipe RECIPE --num-examples-offset 19’ where RECIPE is ‘deepwordbug’ or ‘textfooler’.*
57
57
58
58
Because two text sequences are never indistinguishable, researchers have proposed various alternative definitions for adversarial examples in NLP. We find it useful to group adversarial attacks based on their chosen definitions of adversarial examples.
0 commit comments