Deep Learning in NLP (Winter 2019/2020)

Introduction and Overview

Python refresher

Recall how to program in Python and get familiar with the development environment (VSCode, IPython).

Python refresher and some hints on using VSCode and IPython.

Homework exercise (updated on 15/10), the partial solution (unpack it first), and the dataset with person names.

General feedback and solution to the first homework.

Basic end-to-end example

An end-to-end example of applying a neural network to a simple classification task. We will implement a feed-forward network using the basic PyTorch primitives.

Homework and the corresponding code (unpack it first).

Additional material on using tensors in PyTorch.

General feedback and solution to the second homework.

Lineare Regression

Language classification (I): Application design

Tackle the simple NLP task of classifying person names according to their language (English, German, French, …). Implement a couple of higher-level modules/classes on top of the basic primitives provided by the PyTorch framework, which will allow us to build more complex deep learning models.

Practical session, the corresponding code (original), as well as the version we worked on during the class.

(Link to the theoretical homework moved up, see 28 Oct)

Application design continued

We continue working on the practical session. Download the partial solution (zip) as we left it last week. The additional explanations on github may also be helpful.

Homework (updated on 11/11), based on the practical session. The solution can be found on github.

Das einfache Neuron und Tiefe Architekturen

Stochastic gradient descent

Implement stochastic gradient descent, learn about PyTorch optimizers (e.g. Adam).

Github material on SGD.

Full train/dev/test split. With SGD, we should be able to train on the entire training set. We have already seen the dev part (dev80.csv + dev20.csv). You should avoid doing any experiments with test.csv yet.

Batching

Batching is the technique of specifying neural computations over batches (i.e., sets) of dataset elements. It allows for better parallelization and, hence, faster computations.

Homework, the corresponding code, and the solution. Additionally, Ex. 1 solution notes.

Ideas for further improving the language prediction model.

Backpropagation (theory)

Backpropagation (practical aspects)

Manually specifying backpropagation procedures.

POS tagging (I): Embedding

Exercises and the corresponding code (also includes the UD dataset sample).

The code (without the dataset) is also on github.

UPDATE: version of the code with modifications implemented during the session.

LSTM (theory)

Theoretical homework

POS tagging (II): Scoring and Training

Exercises and the corresponding code (also includes the UD dataset sample).

UPDATE: version of the code with modifications implemented during the session.

POS tagging (III): Training and LSTMs

Exercises and the corresponding code (also includes the UD dataset sample).

UPDATE: version of the code with modifications implemented during the session.

UPDATE 31.12.2019: optimized version of the code. See also the description of the optimization steps.

POS tagging (IV): pre-trained word embeddings + dropout

Finalize the implementation of the POS tagger.

Exercises and the corresponding code (also includes the UD dataset sample).

Note that the code contains certain optimizations implemented during the break. These optimizations speed up training without changing the underlying model (the accuracy should be roughly the same).

English fastText word vectors: the 10^5 most frequent words and the words present in the English UD treebank (both files are based on wiki-news-300d-1M-subword.vec.zip).

UPDATE: version of the code after the session (with the fastText vectors included).

Dependency parsing (I)

Task description (updated on 21.01.2020) and the corresponding code (includes the UD dataset sample and the fastText vectors).

Version of the code after the session (WARNING: this is the last time the code is published on this webpage, make sure to keep track of your own code from now on!)

Dependency parsing (II)

Project proposal presentations

Dependency parsing (III)

Dependency-aware loss function (click raw at the top of the page to copy-and-paste the code)