Documentation
Search…
Simple Transformers
This library is based on the Transformers library by HuggingFace. Simple Transformers lets you quickly train and evaluate Transformer models. Only 3 lines of code are needed to initialize a model, train the model, and evaluate a model. It supports Sequence Classification, Token Classification (NER),Question Answering,Language Model Fine-Tuning, Language Model Training, Language Generation, T5 Model, Seq2Seq Tasks , Multi-Modal Classification and Conversational AI.
The Weights & Biases framework
Weights and Biases is supported for visualizing model training. To use this, simply set a project name for W&B in the
wandb_project attribute of the args dictionary. This will log all hyperparameter values, training losses, and evaluation metrics to the given project.1
model = ClassificationModel('roberta', 'roberta-base', args={'wandb_project': 'project-name'})
Copied!
Any additional arguments that go into
wandb.init can be passed as wandb_kwargs.Structure
The library is designed to have a separate class for every NLP task. The classes that provide similar functionality are grouped together.
simpletransformers.classification- Includes all Classification models.ClassificationModelMultiLabelClassificationModel
simpletransformers.ner- Includes all Named Entity Recognition models.NERModel
simpletransformers.question_answering- Includes all Question Answering models.QuestionAnsweringModel
Here are some minimal examples
MultiLabel Classification
1
model = MultiLabelClassificationModel("distilbert","distilbert-base-uncased",num_labels=6,
2
args={"reprocess_input_data": True, "overwrite_output_dir": True, "num_train_epochs":epochs,'learning_rate':learning_rate,
3
'wandb_project': "simpletransformers"},
4
)
5
# Train the model
6
model.train_model(train_df)
7
​
8
# Evaluate the model
9
result, model_outputs, wrong_predictions = model.eval_model(eval_df)
Copied!
Here are some visualizations generated from the above training script after running a hyper-parameter sweep.
Question Answering
1
train_args = {
2
'learning_rate': wandb.config.learning_rate,
3
'num_train_epochs': 2,
4
'max_seq_length': 128,
5
'doc_stride': 64,
6
'overwrite_output_dir': True,
7
'reprocess_input_data': False,
8
'train_batch_size': 2,
9
'fp16': False,
10
'wandb_project': "simpletransformers"
11
}
12
​
13
model = QuestionAnsweringModel('distilbert', 'distilbert-base-cased', args=train_args)
14
model.train_model(train_data)
Copied!
Here are some visualizations generated from the above training script after running a hyper-parameter sweep.
SimpleTransformers provides classes as well as trainig scripts for all common natural language tasks. Here is the complete list of global arguments that are supported by the library, with their default arguments.
1
global_args = {
2
"adam_epsilon": 1e-8,
3
"best_model_dir": "outputs/best_model",
4
"cache_dir": "cache_dir/",
5
"config": {},
6
"do_lower_case": False,
7
"early_stopping_consider_epochs": False,
8
"early_stopping_delta": 0,
9
"early_stopping_metric": "eval_loss",
10
"early_stopping_metric_minimize": True,
11
"early_stopping_patience": 3,
12
"encoding": None,
13
"eval_batch_size": 8,
14
"evaluate_during_training": False,
15
"evaluate_during_training_silent": True,
16
"evaluate_during_training_steps": 2000,
17
"evaluate_during_training_verbose": False,
18
"fp16": True,
19
"fp16_opt_level": "O1",
20
"gradient_accumulation_steps": 1,
21
"learning_rate": 4e-5,
22
"local_rank": -1,
23
"logging_steps": 50,
24
"manual_seed": None,
25
"max_grad_norm": 1.0,
26
"max_seq_length": 128,
27
"multiprocessing_chunksize": 500,
28
"n_gpu": 1,
29
"no_cache": False,
30
"no_save": False,
31
"num_train_epochs": 1,
32
"output_dir": "outputs/",
33
"overwrite_output_dir": False,
34
"process_count": cpu_count() - 2 if cpu_count() > 2 else 1,
35
"reprocess_input_data": True,
36
"save_best_model": True,
37
"save_eval_checkpoints": True,
38
"save_model_every_epoch": True,
39
"save_steps": 2000,
40
"save_optimizer_and_scheduler": True,
41
"silent": False,
42
"tensorboard_dir": None,
43
"train_batch_size": 8,
44
"use_cached_eval_features": False,
45
"use_early_stopping": False,
46
"use_multiprocessing": True,
47
"wandb_kwargs": {},
48
"wandb_project": None,
49
"warmup_ratio": 0.06,
50
"warmup_steps": 0,
51
"weight_decay": 0,
52
}
Copied!
Checkout this Weights and Baises report that covers training transformers on some the most popular GLUE benchmark datasets. Try it out yourself on colab ​
​​
Last modified 3mo ago