The Incredible Machine: English-Finnish Machine Translation from Prefabricated Components

Lauri Carlson
University of Helsinki
Katina Bontcheva, Timo Järvinen, Mirkka Soininen, Pasi Tapanainen
Conexor Oy
Koetilantie 3, 00710 Helsinki, Finland
pasi.tapanainen@conexor.fi
Kimmo Huovila, Kaarina Hyvönen, Pekka Rapinoja
Kielikone Ltd.
Vattuniemenkatu 3
00211 Helsinki, Finland
kaarina@kielikone.fi
Introduction

We offer a system presentation of the English-Finnish machine translation system developed in the LINGMACHINE project. A novelty, for the Finnish scene at least, is that the setup involves the cooperative assembly of mature LT components by more than one provider. A principal goal is to develop existing language independent technologies of the partners into an MT solution which can be extended to further language pairs. The cooperation is funded by the EU MLIS program. 
We address individual screening translation, or browsing raw translations of foreign language documents in one's native or another more accessible language for information skimming purposes, by non-language professionals, using generic browsing or word processing tools.
In the English-Finnish side of the project we will
•	build up a sound core for an English-Finnish MT system 
•	demonstrate a translation browser for surveying EU related texts for content in English and  Finnish.
Partners
The system combines parser technology developed by Conexor Ltd, MT technology, Finnish morphology and English-Finnish dictionary databases developed by Kielikone Ltd., and the English-to-Finnish transfer and lexicon building experience of the University of Helsinki.
The user partners of the project are the Language Services of the Ministry of Foreign Affairs of Finland and Linguacubun Ltd, a British language consultancy company providing services to UK and EU administrations.
MT technology
The fact is that the commercial state of the art in MT builds on HLT innovations 10-15 years back. For instance, the HLT state of the art survey on MT (Varile/Zampolli 1997) references research done in the late eighties and early nineties. As Kay (op.cit. 249) notes, by and large the general perception remains that a linguistically based transfer approach offers best change for early success. This approach is followed in the LINGMACHINE as well.
The Kielikone Finnish-English MT system TranSmart® is the only commercially developed full MT system for the Finnish language. The English to Finnish direction is not yet available in any competitive products on the market. TranSmart is a fully implemented MT system which relies on KMTech, a language independent MT engine. The Finnish analysis has two parts: morphological analysis of word forms and dependency analysis of sentence structures. The KMTech-based transfer activates the MT engine nine times (of which lexical transfer takes three times) and synthesis three times. The Kielikone parser is deterministic (produces only one tree), and only one tree is progressively transformed into a target tree. The time spent translating a sentence is a linear function of the number of words in the sentence.
TranSmart's strengths include
•	sound technological basis
•	high quality Finnish parser
•	speed (4000-20 000 words per minute)
•	user-friendly lexical maintenance on two levels
•	preserves document formatting, facilitates post-editing
•	full integration to MS Windows/Word
TranSmart includes an intelligent Finnish-English general language dictionary with some 70 000 entries. Company-specific or domain specific dictionaries can be added at will and the dictionaries can be prioritized according to the user's needs.
Parser Technology
Conexor has a variety of morphological analyzers and parsers for English that have been used by developers and researchers worldwide. The full syntactic parser, Conexor Functional Dependency Grammar (FDG) has features that make it especially suitable for machine translation. 
The main modules of FDG are the lexical component and the parsing grammar. The lexical component produces the initial tokenization and assigns all possible morphosyntactic tags to all tokens. The parser proper builds functionally labeled dependencies between words and selects the contextually appropriate morphosyntactic tags. As a result, each token in the output contains the morphosyntactic analysis assigned to the tokens which are connected by labeled dependencies. If the morphosyntactic analysis remains ambiguous, there can be several alternative sets of tags assigned to a token. Furthermore, the output may contain partial trees if the whole structure of the sentence is not recovered. 
The FDG parser has various output formats (see Järvinen & Tapanainen, 1997). The word-based output, seen in Figure 2, is a commonly used form which shows the morphosyntactic information and the linear order of text tokens in a single representation. 
It is the nucleus output that is the most appropriate representation of the functional dependency model itself. The dependency model used in FDG is broadly Lucien Tesnière's structural syntax (Tesnière, 1959) though some modifications have been suggested (see Järvinen & Tapanainen, 1998). In Tesnière's model, the basic element in a syntactic description is not a word, but nucleus, a minimal element containing both the structural and semantic head. 
Functional dependencies represent utterances as a dependency structure where an element is related to another element as a dependent, with a certain function. Figure 1 below shows the nucleus analysis of the sentence ‘Did John Smith manage to do the work?’ in a visual format. The functional representation captures the fact that a statement ‘He managed to do his work’ contains the same number and type of the syntactic elements as the question sentence in Figure 1.

Figure 1: The nucleus output of the Conexor FDG parser

Most often the nucleus consists of the single word item, for example, words that belong to invariable part-of-speech classes (now, and) or synthetic (simple, single-word) forms of verbs, nouns etc. (went, water, lower). However, the verbs, but also nouns, some adjectives, adverbs, numerals etc. have analytic forms (complex, compound, multi-word). For example, ‘has wanted’  is the form for the past simple of the verb ‘to want’, and the boy is the definite form of the noun ‘boy’. Thus, the nucleus wraps together, in one (single- or multi-word) unit, the lexical and the grammatical semantics of the analyzed word. It is the level of nuclei where the best correspondence between syntactic representation and lexical description can be achieved, especially in a multilingual environment. Therefore, the nucleus is an excellent candidate for being the basic translation unit as well.

 #	text token	base form	dependency function	functional tags	surface tags	morphological tags	
1	Did	do	v-ch ® 4	@+FAUXV	%AUX	V PAST	
2	John	John	attr ® 3	@A>	%>N	N NOM SG	
3	Smith	Smith	subj® 1	@SUBJ	%NH	N NOM SG	
4	manage	manage	main® root	@-FMAINV	%VA	V INF	
5	to	to	pm® 6	@INFMARK	%AUX	INFMARK>	
6	do	do	obj® 4	@-FMAINV	%VA	V INF	
7	the	the	det ® 8	@DN	%N	DET	
8	work	work	obj ® 6	@OBJ	%NH	N NOM SG	
9	?	?				PUNCT	

Figure 2. The Conexor FDG parser output in text format.

What is new in LINGMACHINE?
Paradoxically, one relative novelty of LINGMACHINE is just that it is made out of prefabricated parts. Established modules of independent providers are cooperatively combined to produce a reusable, modular system. 
Consequently, the interesting questions center around language engineering aspects: interface standardization, documentation, and other forms of inter-provider communication as means of buffering discrepancies between modules and partners. 
This scenario can be expected to arise increasingly often as the LT field matures, especially in minor language areas. In fact, the LINGMACHINE team has seen this situation before: the MENTOR/F project (Carlson/Vilkuna 1990, Carlson 1990) had to interface a transfer module to two widely different parsers. 
Standardization of grammatical specifications remains unsettled, despite recent major efforts like EAGLES. While translation between grammar formalisms or migration of grammars is premature  (Backofen et al. 1996), there is a call for  language and technology independent parser output/transfer input specifications allowing modules to plug into another or to other front or back ends, whatever their internals. 
The direct use of academic lexicon standards such as PAROLE/SIMPLE (Villegas et al 2000) is impracticable given our engineering requirements (maximal simplicity and conformance to available human dictionary sources). However, we are in a good position to exploit them in an upward compatible way, with a member of the Finnish PAROLE/SIMPLE team on our staff of lexicographers.
One result of LINGMACHINE will thus be API specifications of this sort, for parser output, lexical entries and rules, and transfer input, respectively, which can be reused for new language pairs and modules. They allow decoupling development work at the different sites and better control over revisions of modules during and after the project. Specifications in preparation include the post-parser (Conexor), the feature structure specifications (Conexor) and the transfer rule formats (Kielikone).
In addition to explicit specifications, there are equally interesting if less tangible results in a joint project like LINGMACHINE. Let us look at some of the issues that the respective teams have encountered in the project.
The interface between the parser and the transfer
In order to obtain a human-readable and easily understandable interface, a feature structure representation (fs representation) was chosen as an interface between the transfer and the parser components. 
At the first stage, we transform the information produced by FDG with the name recognition module into a feature structure. Figure 3 below shows a (simplified) sample of the feature structure. The fs representation is still dependency-based where the original dependencies in FDG are described by using attributes. In addition, the syntactic information is in the nucleus form.
The feature structure conversion is not a simple rewrite module, but an additional component.
The grammar specifies three types of features – morphological, syntactic and semantic.
The list of the morphological features includes concepts that are grammaticized (they are expressed by such grammatical means as inflection or/and analytical forms that contain auxiliaries, other function words etc.) and refer to both the formal and the grammatico-semantic properties of the form. The syntactic features deal with sentence constituents such as subjects, objects etc. The semantic features specify the affiliation of the form to a certain lexico-grammatical class (e.g. semelfactive verb, mass noun, qualificative adjective). For example, we can specify the following for the hypercategory of tense and aspect of the verbs in the active voice, indicative:
Temporal distinctive features:
Past	values: True|False	marker: 
past tense of the finite verb component	
Future 	values: True|False	marker: 
auxiliary shall/will	
The categories that have two false values (i.e. non-past, non-future) will be called present.
Aspectual distinctive features:
Perfect 	values: True|False	marker: auxiliary HAVE	
Progressive 	values: True|False	marker: present participle (ING form)	
The categories that have two false values (i.e. non-perfect, non-progressive) will be called simple. Thus, we have:

Distinctive feature
Category	Past	Fut.	Perf.	Progr.	Example	
Present simple	F	F	F	F	he sees	
Present perfect	F	F	T	F	he HAS seen	
Present progressive	F	F	F	T	he is SEEING	
Present perfect progressive	F	F	T	T	he HAS been SEEING	
Past simple	T	F	F	F	he SAW	
Past perfect	T	F	T	F	he HAD seen	
Past progressive	T	F	F	T	he WAS SEEING	
Past perfect progressive	T	F	T	T	he HAD been SEEING	
Future simple	F	T	F	F	he WILL see	
Future perfect	F	T	T	F	he WILL HAVE seen	
Future progressive	F	T	F	T	he WILL be SEEING	
Future perfect progressive	F	T	T	T	he WILL HAVE been SEEING	
Past future simple	T	T	F	F	he WOULD see	
Past future perfect	T	T	T	F	he WOULD HAVE seen	
Past future progressive	T	T	F	T	he WOULD be SEEING	
Past future perfect progressive	T	T	T	T	he WOULD HAVE been SEEING	
Figure 3. The hypercategory of tense and aspect

Let us consider the sentence ‘Mr. Smith had been discussing the possibility to move out.’ The form ‘had been discussing’ has three distinctive features: past, perfect and progressive. Even if the target language does not allow their coexistence, still the meaning can be successfully transferred using pluperfect=past+perfect (e.g. Finnish) or imperfect=past(+progressive) (e.g. Bulgarian).
The fs-representation makes it possible to change the nature of the dependencies of FDG towards semantics. For example, FDG restricts that there is always one and only one head for each nucleus. In the fs-representation in Figure 4, the nucleus Smith is both the subject of the predicate did manage and the subject of the infinitive to do with a zero subject which is inherited from the governing predicate. 

[main [word "Did_manage"]
      [lemma "manage"]
      [morpho [cat verb]
              [voice active]]
      [sem [mood indicative]
           [form past_simple]]
      [syntax [subject W2: [word "John_Smith"]
                           [lemma "John_Smith"]
                           [morpho [cat noun]
                                   [number singular]
                                   [case nominative]]
                           [sem proper T]]
               [object [word "do"]
                       [lemma "do"]
                       [morpho [cat verb]
                               [voice active]]
                       [sem mood indicative]
                       [syntax [subject W2]
                               [object [word "work"]
                                       [lemma "work"]
                                       [morpho [cat noun]
                                               [number singular]
                                               [case nominative]]]]]]]

Figure 4: The feature structure representation of Did John Smith manage to do the work?

Transfer revisited
Componential MT development is not only a beautiful idea, it actually facilitates the cooperation between the companies involved as each company is free to concentrate on their area of expertise without having to know how the others work. This does not mean, however, that communication is not crucial or that it happens automatically. Each party needs to know enough to avoid extra difficulties in welding the components together. In a multi-company cooperative project communication needs to be scheduled and meticulously carried out.
The LINGMACHINE project presented Kielikone with a unique opportunity: to build a componential machine translation system from components provided by other developers and by us. We have a mature MT product, the Finnish-English MT system TranSmart® which we have built from scratch. LINGMACHINE gives us a chance to streamline our MT technology to better be able to import outside technology and domestic technology, to improve some of the features in our transfer, and to recycle out lexical databases and lexical transfer technology. 
Importing outside technology is always a challenge. However, the software work needed to connect the two systems turned out to be reasonable. The project benefited greatly from getting a mature, well tested parser from Conexor. Yet, the parser was not originally planned for machine translation, and some design decisions are less than optimal for Kielikone purposes. For example, the parser tagged all participial attributes as adjectives and lost all verbal information. This made it impossible or at least rather difficult to translate participial attributes productively. Furthermore, the parser outputs some useful information that the Kielikone transfer was not able to utilize, so we decided to improve our transfer engine to handle that additional data. Consider semantics: unlike our Finnish parser, the Conexor parser accommodates semantic information. Consequently, we do not need to have much semantic information in the lexical transfer. This is better for the overall system since now most of the semantic information is available to improve parsing results. The lexicon needs to give additional semantics only in cases where either the parser does not give all the needed semantics or the target language semantics differ from the source language semantics.
Bringing domestic technology into a new context can be as challenging as using outside technology. We have a Finnish morphological form generator, Formo®, that we integrated into the transfer engine as a new component. The actual software work was relatively easy but the integration of these two components was not without problems as the two were not originally planned to be part of the same system. In the context of reusing our own technology we also took a long critical look at our linguistic rule syntax for lexical transfer. Developing a mature Finnish-English MT system has given us a good grasp of what we could have done better the first time. For example, we revised our transfer attributes extensively. Part of the motivation was to make our system closer to the parser output to facilitate connecting the parser to our transfer, but it was also an opportunity to do things better the second time. Furthermore, we have been careful this time not to lose any information during the transfer. In our existing MT technology we had seemingly redundant semantic information pruned away in tree normalization. The problem is that redundancy tends to have a function in language, and this can sometimes affect the desired translation equivalent. There are sometimes important text linguistic cues coded in seemingly less relevant parts of the sentence, and these cues can help make heuristics better. Some of these features also help make guesses about the context of the sentence within a system that translates only one sentence at a time. The need for not pruning away information may not be apparent at the early stages of a machine translation system, but work with a mature one had taught us this. We devised a system of retaining all the information without doing away with the idea of tree normalization. However, we cannot yet evaluate whether the retention of all information is as important in an English-Finnish system as it is in a Finnish-English system.
Strangely enough, in the LINGMACHINE project compiling the initial MT lexicons (sometimes described as tedious and endless work) was almost fun. Kielikone has several large bilingual dictionary databases and in this case we were able to recycle our English-Finnish dictionary to automatically create a basic lexicon with some 30 000 entries.  This lexicon is currently being revised by our linguists to optimize translation equivalents. This basic lexicon will be augmented by some 300-500 intelligent lexical rules for this project. Our existing lexical transfer rule syntax is highly functional so we were able to recycle it almost as such.

Conclusions
Software development and glueing the components together takes quite a bit of time and communication, if it is to be done successfully. The end result will look slightly like a patchwork, but this need not be a big problem. It is still more feasible to use well-tested components and to build the APIs than to build these components. Doing things a second time gives us an opportunity to learn from past mistakes.

References
Backofen, R. et al. (1966). The EAGLES Formalisms Working Group Final Report.
Carlson, L. & M. Vilkuna. (1990). Independent transfer using graph unification. In COLING-90: Papers presented to 	the 13th International Conference on Computational Linguistics. Ed. H. Karlgren. Helsinki 1990, Col. 3., 
	pp. 60-65. 
Carlson, L. (1990). Design of unification based transfer lexicon. SKY 1990: Largely Lexical. Yearbook of the 	Linguistic Society of Finland 1990. Ed. K.Jokinen and J-O. Östman. Helsinki 1990, pp.
Varile, G., Zampolli A. eds. (1997). Survey of the State of the Art in Human Language Technology. In: 	http://cslu.cse.ogi.edu/HLTsurvey/HLTsurvey.html
Villegas, M. et al. (2000). Multilingual linguistic resources: from monolingual lexicons to bilingual interrelated 	lexicons. LREC2000, Athens.
Järvinen, T. & Tapanainen, P. (1997). 'Dependency Parser for English'. Technical Report No. TR-1 Helsinki: 	University of Helsinki, Department of General Linguistics.
Järvinen, T. & Tapanainen, P. (1998). Towards an implementable dependency grammar. In Proceedings of the 	Workshop "Processing of Dependency-Based Grammars", (eds.) Sylvain Kahane and Alain Polguère, 	Université de Montréal, Quebec, Canada, 15th August 1998, pp. 1-10. 
Tapanainen, P. (1999). Parsing in two frameworks: finite-state and functional dependency grammar. Academic 	Dissertation, Helsinki: University of Helsinki, Language Technology, Department of General Linguistics, 	Faculty of Arts.
Tapanainen, P. & Järvinen, T. (1997). A non-projective dependency parser. In Proceedings of the 5th Conference on 	Applied Natural Language Processing. Washington D.C.: Association for Computational Linguistics, 
	64-71.
Tapanainen, P. & Järvinen, T. (1998). Dependency Concordances. In International Journal of Lexicography, Vol. 11 	No. 3, Oxford: Oxford University Press, 187-203.
Tesnière, L. (1959). Éléments de syntaxe structurale. Editions Klincksieck, Paris. 

http://www.ling.helsinki.fi/kieliteknologia/tutkimus/lingmachine/Public/mtsummitix.rtf