Pallet IMU data

The organizers provided annotated IMU timeseries data generated by an acceleration sensor (x,y,z-axis), a gyroscope (x,y,z-axis) and a barometer (x-axis). The annotated classes are

• Driving (straight and curve)
• Lifting (raising and lowering)
• Standing
• Docking
• Forks (from the front or side)
• Wrapping
• Wrapping preparation

In the provided preprocessing, wrapping preparation, docking, forks and standing were all mapped to “Stationary processes”.

Train, Validation and Test splitting

The data is segmented into 4 folds of roughly equal size. We used two folds for training, one for validation and one for testing. CNN and Transformer used the validation split during training for early stopping, the Random Forest used the validation split for training as well, which was explicitly allowed by the task organizers. Since the Random Forest does not have epochs, it does not need the validation split for early stopping.

Dataset investigation

In all folds, stationary processes are overrepresented, while lifting (both raising and lowering) is severely underrepresented. Training the underrepresented classes turned out to be our largest problem.

While we were able to learn the other classes well, classifying lifting correctly remained a problem. A closer investigation into how each feature behaves during these operations did also not reveal a lot of useful insight.

Dataset preprocessing

Preprocessing was partially provided by the task organizers and partially added by us. It consists of cleaning the data and turning the input stream into fixed-length 3-dimensional windows, together with a downsampling to 80 Hz. Each window has a length of 12, meaning the models receive 137.5 ms of historic data and have to classify the current action based of that. This is only a snapshot of what the pallet is experiencing. This way, we rarely have to worry about a window overlapping with another action which could cause wrong classification. But, in all honesty, if we would have had more compute, we would likely have used a higher resolution and longer windows.

Engineered features are also available, but we found that they did not improve model capabilities, so we left them out. The provided code likewise performed scaling during preprocessing, but we moved that to each model implementation to enable per-model control.

Evaluation of model quality

The evaluation metric MCC (Matthews correlation coefficient) was given by the organizers. MCC remains a robust measure even when classes are wildly off-balance, as is the case with the given dataset. MCC returns a value between -1 and +1. A coefficient of +1 represents a perfect prediction, 0 an average random prediction and -1 an inverse prediction.

The standard multiclass MCC is defined as:

where

• $t_k = \sum_{i=1}^{K} C_{ik}$ the number of times class $k$ truly occurred
• $p_k = \sum_{i=1}^{K} C_{ki}$ the number of times class $k$ was predicted
• $c = \sum_{k=1}^{K} C_{kk}$ the total number of samples correctly predicted
• $s = \sum_{i=1}^{K} \sum_{j=1}^{K} C_{ij}$ the total number of samples

Above formulas are from the Scikit-learn user guide ².

Since multiple labels can be true at the same time (e.g. lifting the forks while driving), the organizers implemented an unweighted average over all classes. As a consequence, a particular bad class can reduce the overall score, even if it does not occur often in the data. Both lifting classes dragged down our MCC considerably.

Testing different models

We tried different ML and DL models for classification, ranging from simple random forests to transformers.

Random forest classifier

The first classifier we tried was the RandomForestClassifier from sklearn using 50 estimators and 10,000 samples. Since the original data is timeseries data, we performed feature-engineering for the random forest. For each original feature (described above), we calculated the mean, standard deviation, maximum and minimum across a short time window. This flattened the original 3D data into 2 dimensions. The model had to predict the label of this time window using only the 27 engineered features.

We used this classifier as our baseline for future explorations. Nevertheless, problems surfaced by this model were indicative of our problems with more advanced models. More on the results in a later section.

CNN classifier

Our second and most realistic approach was a small CNN. The architecture is fairly standard: 3 convolution layers each followed by a normalization, ReLu and dropout layer. This section is followed by a dense $64 \cdot 64$ linear head with ReLu and dropout layers again. Finally, the CNN produces 6 output logits through a linear layer, one for each class. We considered a custom implementation for asymmetric focal loss (AFL) because the dataset is highly imbalanced. Ablation showed that the simpler BCE loss works just as well, so we switched to it as it requires less parameters and is in off-the-shelf solution.

During training, we use the AdamW optimizer. We allowed up to 50 epochs, but implemented early stopping with a patience of 8 based on the MCC on the validation dataset. Basing it of the MCC put the early stopping metric more in line with the final competition metric. The best checkpoint is restored. In the best run, the best checkpoint was reached in epoch 29.

After training, we perform per-class threshold optimization on the validation set. For every class, a specific cut-off maximizes the correlation between prediction and ground truth. The cut-off is not only different per class, but also different per model. We ran the model over validation data and collected the resulting probabilities. The candidate grid for thresholds is between 0.05 and 0.95 in steps of 0.01, so in total 91 possible thresholds for each class. We find the best threshold per class by calculating the MCC for each possible combination of class and threshold. Contrary to expectations, this optimization step has next to no impact for stronger models. The best CNN only gained 0.013 (MCC) through this step, while worse performing models gained up to 0.107. Since we developed this step while our CNNs were performing worse, we simply kept it during later runs, although it provided next to no benefit anymore.

A transformer for classification

Since I was learning about and using transformers in university at the time, I decided to try my luck with a transformer for classification. I knew that it would be absolutely overkill and that there are better models out there for this job, but I decided it would be a fun learning experience. After all, we had timeseries data, and transformers are built for sequences.

The transformer did not require engineered features, it only got the 7 original features. Due to us being compute restrained and transformers requiring a lot of compute, we tried different combinations of sequence length and sampling rate. We sampled down by only using every n-th measurement in the final input. Each timestep is encoded into one token, giving us 12 tokens per input. Attention mechanisms become largely trivial in our case, as all tokens are roughly equally important. In line with standard transformers for classification, e.g. BART, we used learned absolute positional encodings. For us, it has the advantage that positional embeddings turn into a lookup-table during inference, which is fast and simple. In practice, absolute embeddings are used, because they allow the model to learn a clear, well defined map of the sequence. The major trade-off is that our model can not deal with sequences longer than 12 tokens, which we find completely acceptable for our use-case.
We used 3 TransformerEncoderLayer (128 embedding dimensions, 8 attention heads, FFN width 512). No attention mask is applied, as we use a fixed size of input tokens, and every timestep can attend to every other. The encoder output is mean-pooled over time, followed by a single linear layer producing 6 independent logits for final classification. In total, our transformer has around 0.62M parameters. Training was performed across 20 epochs with early stopping (patience 5, using MCC on validation split as criterion), again with weighted BCE for optimization with a max pos_weight of 5.

Contrary to the CNN, we did not use post-training threshold tuning. As discussed above, this step proved unnecessary once the CNN improved, so we saw no reason to implement it in the transformer, which did perform better than the CNN.

Using domain knowledge to improve transformer performance

Often enough, the transformer was unable to distinguish between classes which can not be true simultaneously (i.e. driving straight and through a curve or raising and lowering at the same time). It can do so, because it treats every output class a independent Bernoulli decisions. To combat this problem, we used domain knowledge to post-process the logits. We manually created as list of impossible combinations. If they occur, the pipeline returns only the class with the highest probability. However, there are legal combinations of labels, for example one form of driving with one form of lifting. While keeping all single actions, we compute the log-probability of every legal combination derived from the training data and pick the best-scoring out of the single actions or legal combinations using the maximum-likelihood.

Results

Above table compares the best performance of our models with the model which would eventually go on to win the competition ³.

As the final results show, the transformer barely managed to beat the much simpler CNN. As alluded to earlier, we knew a transformer would not be the ideal model. Additionally, we probably made some mistakes along the way. Given all of that, I am surprised the transformer performed as good as it did.

We mainly struggled with the lifting classes, as they are severely underrepresented in the training data and not easy to distinguish from other classes. Per-class diagnostics make this obvious (table showing diagnostics for best model each):

Due to its unique characteristics, turntable wrapping is easy to distinguish. Stationary processes are both relatively easy and have a lot of training data, which enable high scores for this class as well. Driving and lifting are more problematic. Since the evaluation metric is unweighted average MCC, especially the lifting classes with their bad scores and low support decreased the result considerably. The CNN actually tends to outperform the transformer on easier classes. The transformer wins essentially only on lifting, mainly because it is better in recalling Lifting (raising), without losing much precision.

Personal take-aways

For one evening, I think we produced a fairly good model. I also learned a thing or two about analyzing and optimizing models. I especially enjoyed having the opportunity to experiment with a transformer, even though it is really small. I would love to work more with them on a larger scale. Given the lack of lifting data, I did consider experimenting with synthetic data generation, but was unsure if it was allowed by the competition rules and did not have time to dive deep, so ultimately I did not attempt it. This one is still on my bucket list.