blob: eb098368c132df6eb82cc9de38ffaeadec387e37 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
|
# Evaluating cross-lingual textual similarity on dictionary alignment
This repository contains the scripts to prepare the resources as well as open source implementations of the methods. Word Mover's Distance and Sinkhorn implementations are extended from [Cross-lingual retrieval with Wasserstein distance](https://github.com/balikasg/WassersteinRetrieval) and supervised implementation is extended from https://github.com/fionn-mac/Manhattan-LSTM.
```bash
git clone https://github.com/yigitsever/Evaluating-Dictionary-Alignment.git
cd Evaluating-Dictionary-Alignment
```
## Requirements
```bash
pip install -r pre_requirements.txt
pip install -r requirements.txt
```
- Python 3
- [nltk](http://www.nltk.org/)
- [lapjv](https://pypi.org/project/lapjv/)
- [POT](https://pypi.org/project/POT/)
- [mosestokenizer](https://pypi.org/project/mosestokenizer/)
- (Optional) If using VecMap
* NumPy
* SciPy
<details><summary>We recommend using a virtual environment</summary>
<p>
In order to create a [virtual environment](https://docs.python.org/3/library/venv.html#venv-def) that resides in a directory `.env` under home;
```bash
cd ~
mkdir -p .env && cd .env
python -m venv evaluating
source ~/.env/evaluating/bin/activate
```
After the virtual environment is activated, the python interpreter and the installed packages are isolated. In order for our code to work, the correct environment has to be sourced/activated.
In order to install all dependencies automatically use the [pip](https://pypi.org/project/pip/) package installer using `requirements.txt`, which resides under the repository directory.
```bash
# under Evaluating-Dictionary-Alignment
pip install -r requirements.txt
```
Rest of this README assumes that you are in the repository root directory.
</p>
</details>
## Acquiring The Data
nltk is required for this stage;
```python
import nltk
nltk.download('wordnet')
```
Then;
```bash
./get_data.sh
```
This will create two directories; `dictionaries` and `wordnets`.
Linewise aligned definition files are in `wordnets/ready`.
<details><summary>Language pairs and number of available aligned glosses</summary>
<p>
Source Language | Target Language | # of Pairs
--- | --- | ---:
English | Bulgarian | 4959
English | Greek | 18136
English | Italian | 12688
English | Romaian | 58754
English | Slovenian | 3144
English | Albanian | 4681
Bulgarian | Greek | 2817
Bulgarian | Italian | 2115
Bulgarian | Romaian | 4701
Greek | Italian | 4801
Greek | Romaian | 2144
Greek | Albanian | 4681
Italian | Romaian | 10353
Romaian | Slovenian | 2085
Romaian | Albanian | 4646
</p>
</details>
## Acquiring The Embeddings
We use [VecMap](https://github.com/artetxem/vecmap) on [fastText](https://fasttext.cc/) embeddings. You can skip this step if you are providing your own polylingual embeddings.
Otherwise;
* initialize and update the VecMap submodule;
```bash
git submodule init && git submodule update
```
* make sure `./get_data` is already run and `dictionaries` directory is present.
* run;
```bash
./get_embeddings.sh
```
Bear in mind that this will require around 50 GB free space.
## Quick Demo
`demo.sh` is included, downloads data for 2 languages.
```bash
./demo.sh
```
|
