MMS - Example training pipeline

!pip install datasets transformers==4.30.0 torch sacremoses scikit-learn evaluate accelerate

import os

import evaluate
import numpy as np
from datasets import load_dataset
from transformers import (
    AutoModelForSequenceClassification,
    AutoTokenizer,
    Trainer,
    TrainingArguments,
)

Our dataset is publicly available but we need to you to accept conditions. Please see this link, accept the terms

mms_dataset = load_dataset("Brand24/mms")

Downloading and preparing dataset mms/default to /root/.cache/huggingface/datasets/Brand24___mms/default/0.2.0/70532fdd01f149ff84a280b7d9cfb661643abf4837b4f0f3aa1128064e870d65...
Dataset mms downloaded and prepared to /root/.cache/huggingface/datasets/Brand24___mms/default/0.2.0/70532fdd01f149ff84a280b7d9cfb661643abf4837b4f0f3aa1128064e870d65. Subsequent calls will reuse this data.

There are 14 different dimensions which differentiate obtained datasets. In addition, there is a pre-calculated cleanlab self conficence score for each sample. All of them can be used to sample examples which suit our use case best

mms_dataset.column_names

{'train': ['_id',
  'text',
  'label',
  'original_dataset',
  'domain',
  'language',
  'Family',
  'Genus',
  'Definite articles',
  'Indefinite articles',
  'Number of cases',
  'Order of subject, object, verb',
  'Negative morphemes',
  'Polar questions',
  'Position of negative word wrt SOV',
  'Prefixing vs suffixing',
  'Coding of nominal plurality',
  'Grammatical genders',
  'cleanlab_self_confidence']}

Select only samples in polish and coming from social media

pl_sm = mms_dataset["train"].filter(lambda x: x["language"] == "pl" and x["domain"] == "social_media")

To achieve higher performance, we will select only samples with high self confidence score

pl_sm_high_confidence = pl_sm.filter(lambda x: x["cleanlab_self_confidence"] > 0.6)

len(pl_sm_high_confidence)

73227

We will use this examples to fine-tune Polish version of BERT model - HerBERT

tokenizer = AutoTokenizer.from_pretrained("allegro/herbert-base-cased")

def tokenize(batch):
    return tokenizer(batch["text"], padding="max_length", truncation=True)

tokenized_dataset = pl_sm_high_confidence.map(tokenize, batched=True, batch_size=512)

model = AutoModelForSequenceClassification.from_pretrained("allegro/herbert-base-cased", num_labels=3)

Some weights of the model checkpoint at allegro/herbert-base-cased were not used when initializing BertForSequenceClassification: ['cls.predictions.transform.dense.weight', 'cls.sso.sso_relationship.weight', 'cls.predictions.decoder.bias', 'cls.sso.sso_relationship.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias']
- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at allegro/herbert-base-cased and are newly initialized: ['classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

split_dataset = tokenized_dataset.train_test_split(test_size=0.1)
train_dataset = split_dataset["train"]
eval_dataset = split_dataset["test"]

training_args = TrainingArguments(
    output_dir="PL_SM_SENT",
    evaluation_strategy="epoch",
    num_train_epochs=1,
)
metric = evaluate.load("accuracy")


def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return metric.compute(predictions=predictions, references=labels)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    compute_metrics=compute_metrics,
)

trainer.train()

/opt/conda/lib/python3.10/site-packages/transformers/optimization.py:411: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning
  warnings.warn(