Multi-Hop Transformer for Document-Level Machine Translationのまとめ

【論文まとめ】Multi-Hop Transformer for Document-Level Machine Translation

<p>本記事において使用される図表は，原著論文内の図表を引用しています．</p>
<p>また，本記事の内容は，著者が論文を読み，メモとして短くまとめたものになります．必ずしも内容が正しいとは限らないこと，ご了承ください．</p>
<h2>論文情報</h2>
<p>タイトル: Multi-Hop Transformer for Document-Level Machine Translation</p>
<p>研究会: NAACL</p>
<p>年度: 2021</p>
<p>キーワード: MT, transformer, Multi-Hop Transformer</p>
<p>URL: <a href="https://aclanthology.org/2021.naacl-main.309.pdf">https://aclanthology.org/2021.naacl-main.309.pdf</a></p>
<p>DOI: <a href="http://dx.doi.org/10.18653/v1/2021.naacl-main.309">http://dx.doi.org/10.18653/v1/2021.naacl-main.309</a></p>
<p>データセット: TED Talk, OpenSubtitles, Europarl7</p>
<h2>概要</h2>
<p>Document-level neural machine translationにおいて，Multi-Hopなアーキテクチャを導入することにより，従来手法と比べて精度の高い文脈を考慮した機械翻訳を実現</p>
<p>翻訳者のように，頭の中に翻訳のドラフトを作り，文脈に合わせて適切に修正する流れ（human-like draft-editing）を明示的にモデリング</p>
<p>大きな事前学習済みモデルを使うことなく，使用に足る機械翻訳モデルを実現</p>
<h2>提案手法</h2>
<p><img src="/images/article/Multi-Hop-Transformer-for-Document-Level-Machine-Translation/p7s4t8vi.png" alt=""></p>
<p>アーキテクチャ周りのこと</p>
<h3>Sentence Encoder</h3>
<p>source-sideとtarget-sideでそれぞれPretrained Encoderがあり，source contextとtarget draftの分散表現をそれぞれ得る</p>
<h3>Multi-Hop Encoder</h3>
<p>source-contextにおいて文章ごとのreasoningをして，現在の文章の分散表現を得る</p>
<h3>Multi-Hop Decoder</h3>
<p>target-side draftから情報を取得して，翻訳の確率分布を得る</p>
<p>そのほかアーキテクチャの工夫</p>
<h3>Contet Gating</h3>
<p>contextual informationを過剰にutilizeしすぎないように，context gating machanismを採用</p>
<p>contextと現在の文章間の重みを動的にコントロールする</p>
<p><span class="math math-inline"><mjx-container class="MathJax" jax="SVG"><svg style="vertical-align: -0.796ex;" xmlns="http://www.w3.org/2000/svg" width="25.003ex" height="3.196ex" role="img" focusable="false" viewBox="0 -1060.7 11051.5 1412.5" xmlns:xlink="http://www.w3.org/1999/xlink"><defs><path id="MJX-1-TEX-I-1D6FC" d="M34 156Q34 270 120 356T309 442Q379 442 421 402T478 304Q484 275 485 237V208Q534 282 560 374Q564 388 566 390T582 393Q603 393 603 385Q603 376 594 346T558 261T497 161L486 147L487 123Q489 67 495 47T514 26Q528 28 540 37T557 60Q559 67 562 68T577 70Q597 70 597 62Q597 56 591 43Q579 19 556 5T512 -10H505Q438 -10 414 62L411 69L400 61Q390 53 370 41T325 18T267 -2T203 -11Q124 -11 79 39T34 156ZM208 26Q257 26 306 47T379 90L403 112Q401 255 396 290Q382 405 304 405Q235 405 183 332Q156 292 139 224T121 120Q121 71 146 49T208 26Z"></path><path id="MJX-1-TEX-N-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path id="MJX-1-TEX-I-1D70E" d="M184 -11Q116 -11 74 34T31 147Q31 247 104 333T274 430Q275 431 414 431H552Q553 430 555 429T559 427T562 425T565 422T567 420T569 416T570 412T571 407T572 401Q572 357 507 357Q500 357 490 357T476 358H416L421 348Q439 310 439 263Q439 153 359 71T184 -11ZM361 278Q361 358 276 358Q152 358 115 184Q114 180 114 178Q106 141 106 117Q106 67 131 47T188 26Q242 26 287 73Q316 103 334 153T356 233T361 278Z"></path><path id="MJX-1-TEX-N-28" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"></path><path id="MJX-1-TEX-I-1D44A" d="M436 683Q450 683 486 682T553 680Q604 680 638 681T677 682Q695 682 695 674Q695 670 692 659Q687 641 683 639T661 637Q636 636 621 632T600 624T597 615Q597 603 613 377T629 138L631 141Q633 144 637 151T649 170T666 200T690 241T720 295T759 362Q863 546 877 572T892 604Q892 619 873 628T831 637Q817 637 817 647Q817 650 819 660Q823 676 825 679T839 682Q842 682 856 682T895 682T949 681Q1015 681 1034 683Q1048 683 1048 672Q1048 666 1045 655T1038 640T1028 637Q1006 637 988 631T958 617T939 600T927 584L923 578L754 282Q586 -14 585 -15Q579 -22 561 -22Q546 -22 542 -17Q539 -14 523 229T506 480L494 462Q472 425 366 239Q222 -13 220 -15T215 -19Q210 -22 197 -22Q178 -22 176 -15Q176 -12 154 304T131 622Q129 631 121 633T82 637H58Q51 644 51 648Q52 671 64 683H76Q118 680 176 680Q301 680 313 683H323Q329 677 329 674T327 656Q322 641 318 637H297Q236 634 232 620Q262 160 266 136L501 550L499 587Q496 629 489 632Q483 636 447 637Q428 637 422 639T416 648Q416 650 418 660Q419 664 420 669T421 676T424 680T428 682T436 683Z"></path><path id="MJX-1-TEX-I-1D44E" d="M33 157Q33 258 109 349T280 441Q331 441 370 392Q386 422 416 422Q429 422 439 414T449 394Q449 381 412 234T374 68Q374 43 381 35T402 26Q411 27 422 35Q443 55 463 131Q469 151 473 152Q475 153 483 153H487Q506 153 506 144Q506 138 501 117T481 63T449 13Q436 0 417 -8Q409 -10 393 -10Q359 -10 336 5T306 36L300 51Q299 52 296 50Q294 48 292 46Q233 -10 172 -10Q117 -10 75 30T33 157ZM351 328Q351 334 346 350T323 385T277 405Q242 405 210 374T160 293Q131 214 119 129Q119 126 119 118T118 106Q118 61 136 44T179 26Q217 26 254 59T298 110Q300 114 325 217T351 328Z"></path><path id="MJX-1-TEX-I-1D434" d="M208 74Q208 50 254 46Q272 46 272 35Q272 34 270 22Q267 8 264 4T251 0Q249 0 239 0T205 1T141 2Q70 2 50 0H42Q35 7 35 11Q37 38 48 46H62Q132 49 164 96Q170 102 345 401T523 704Q530 716 547 716H555H572Q578 707 578 706L606 383Q634 60 636 57Q641 46 701 46Q726 46 726 36Q726 34 723 22Q720 7 718 4T704 0Q701 0 690 0T651 1T578 2Q484 2 455 0H443Q437 6 437 9T439 27Q443 40 445 43L449 46H469Q523 49 533 63L521 213H283L249 155Q208 86 208 74ZM516 260Q516 271 504 416T490 562L463 519Q447 492 400 412L310 260L413 259Q516 259 516 260Z"></path><path id="MJX-1-TEX-I-1D45B" d="M21 287Q22 293 24 303T36 341T56 388T89 425T135 442Q171 442 195 424T225 390T231 369Q231 367 232 367L243 378Q304 442 382 442Q436 442 469 415T503 336T465 179T427 52Q427 26 444 26Q450 26 453 27Q482 32 505 65T540 145Q542 153 560 153Q580 153 580 145Q580 144 576 130Q568 101 554 73T508 17T439 -10Q392 -10 371 17T350 73Q350 92 386 193T423 345Q423 404 379 404H374Q288 404 229 303L222 291L189 157Q156 26 151 16Q138 -11 108 -11Q95 -11 87 -5T76 7T74 17Q74 30 112 180T152 343Q153 348 153 366Q153 405 129 405Q91 405 66 305Q60 285 60 284Q58 278 41 278H27Q21 284 21 287Z"></path><path id="MJX-1-TEX-N-29" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"></path><path id="MJX-1-TEX-I-1D460" d="M131 289Q131 321 147 354T203 415T300 442Q362 442 390 415T419 355Q419 323 402 308T364 292Q351 292 340 300T328 326Q328 342 337 354T354 372T367 378Q368 378 368 379Q368 382 361 388T336 399T297 405Q249 405 227 379T204 326Q204 301 223 291T278 274T330 259Q396 230 396 163Q396 135 385 107T352 51T289 7T195 -10Q118 -10 86 19T53 87Q53 126 74 143T118 160Q133 160 146 151T160 120Q160 94 142 76T111 58Q109 57 108 57T107 55Q108 52 115 47T146 34T201 27Q237 27 263 38T301 66T318 97T323 122Q323 150 302 164T254 181T195 196T148 231Q131 256 131 289Z"></path><path id="MJX-1-TEX-N-2B" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"></path><path id="MJX-1-TEX-I-1D44F" d="M73 647Q73 657 77 670T89 683Q90 683 161 688T234 694Q246 694 246 685T212 542Q204 508 195 472T180 418L176 399Q176 396 182 402Q231 442 283 442Q345 442 383 396T422 280Q422 169 343 79T173 -11Q123 -11 82 27T40 150V159Q40 180 48 217T97 414Q147 611 147 623T109 637Q104 637 101 637H96Q86 637 83 637T76 640T73 647ZM336 325V331Q336 405 275 405Q258 405 240 397T207 376T181 352T163 330L157 322L136 236Q114 150 114 114Q114 66 138 42Q154 26 178 26Q211 26 245 58Q270 81 285 114T318 219Q336 291 336 325Z"></path><path id="MJX-1-TEX-I-1D435" d="M231 637Q204 637 199 638T194 649Q194 676 205 682Q206 683 335 683Q594 683 608 681Q671 671 713 636T756 544Q756 480 698 429T565 360L555 357Q619 348 660 311T702 219Q702 146 630 78T453 1Q446 0 242 0Q42 0 39 2Q35 5 35 10Q35 17 37 24Q42 43 47 45Q51 46 62 46H68Q95 46 128 49Q142 52 147 61Q150 65 219 339T288 628Q288 635 231 637ZM649 544Q649 574 634 600T585 634Q578 636 493 637Q473 637 451 637T416 636H403Q388 635 384 626Q382 622 352 506Q352 503 351 500L320 374H401Q482 374 494 376Q554 386 601 434T649 544ZM595 229Q595 273 572 302T512 336Q506 337 429 337Q311 337 310 336Q310 334 293 263T258 122L240 52Q240 48 252 48T333 46Q422 46 429 47Q491 54 543 105T595 229Z"></path><path id="MJX-1-TEX-N-2212" d="M84 237T84 250T98 270H679Q694 262 694 250T679 230H98Q84 237 84 250Z"></path><path id="MJX-1-TEX-I-1D456" d="M184 600Q184 624 203 642T247 661Q265 661 277 649T290 619Q290 596 270 577T226 557Q211 557 198 567T184 600ZM21 287Q21 295 30 318T54 369T98 420T158 442Q197 442 223 419T250 357Q250 340 236 301T196 196T154 83Q149 61 149 51Q149 26 166 26Q175 26 185 29T208 43T235 78T260 137Q263 149 265 151T282 153Q302 153 302 143Q302 135 293 112T268 61T223 11T161 -11Q129 -11 102 10T74 74Q74 91 79 106T122 220Q160 321 166 341T173 380Q173 404 156 404H154Q124 404 99 371T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="scale(1,-1)"><g data-mml-node="math"><g data-mml-node="mi"><use data-c="1D6FC" xlink:href="#MJX-1-TEX-I-1D6FC"></use></g><g data-mml-node="mo" transform="translate(917.8,0)"><use data-c="3D" xlink:href="#MJX-1-TEX-N-3D"></use></g><g data-mml-node="mi" transform="translate(1973.6,0)"><use data-c="1D70E" xlink:href="#MJX-1-TEX-I-1D70E"></use></g><g data-mml-node="mo" transform="translate(2544.6,0)"><use data-c="28" xlink:href="#MJX-1-TEX-N-28"></use></g><g data-mml-node="msub" transform="translate(2933.6,0)"><g data-mml-node="mi"><use data-c="1D44A" xlink:href="#MJX-1-TEX-I-1D44A"></use></g><g data-mml-node="mi" transform="translate(977,-150) scale(0.707)"><use data-c="1D44E" xlink:href="#MJX-1-TEX-I-1D44E"></use></g></g><g data-mml-node="msubsup" transform="translate(4334.6,0)"><g data-mml-node="mi"><use data-c="1D434" xlink:href="#MJX-1-TEX-I-1D434"></use></g><g data-mml-node="TeXAtom" transform="translate(783,530.4) scale(0.707)" data-mjx-texclass="ORD"><g data-mml-node="mo"><use data-c="28" xlink:href="#MJX-1-TEX-N-28"></use></g><g data-mml-node="mi" transform="translate(389,0)"><use data-c="1D45B" xlink:href="#MJX-1-TEX-I-1D45B"></use></g><g data-mml-node="mo" transform="translate(989,0)"><use data-c="29" xlink:href="#MJX-1-TEX-N-29"></use></g></g><g data-mml-node="mi" transform="translate(783,-138.9) scale(0.707)"><use data-c="1D460" xlink:href="#MJX-1-TEX-I-1D460"></use></g></g><g data-mml-node="mo" transform="translate(6364.2,0)"><use data-c="2B" xlink:href="#MJX-1-TEX-N-2B"></use></g><g data-mml-node="msub" transform="translate(7364.5,0)"><g data-mml-node="mi"><use data-c="1D44A" xlink:href="#MJX-1-TEX-I-1D44A"></use></g><g data-mml-node="mi" transform="translate(977,-150) scale(0.707)"><use data-c="1D44F" xlink:href="#MJX-1-TEX-I-1D44F"></use></g></g><g data-mml-node="msubsup" transform="translate(8694.8,0)"><g data-mml-node="mi"><use data-c="1D435" xlink:href="#MJX-1-TEX-I-1D435"></use></g><g data-mml-node="TeXAtom" transform="translate(792,530.4) scale(0.707)" data-mjx-texclass="ORD"><g data-mml-node="mo"><use data-c="28" xlink:href="#MJX-1-TEX-N-28"></use></g><g data-mml-node="mi" transform="translate(389,0)"><use data-c="1D45B" xlink:href="#MJX-1-TEX-I-1D45B"></use></g><g data-mml-node="mo" transform="translate(989,0)"><use data-c="29" xlink:href="#MJX-1-TEX-N-29"></use></g></g><g data-mml-node="TeXAtom" transform="translate(792,-293.8) scale(0.707)" data-mjx-texclass="ORD"><g data-mml-node="mi"><use data-c="1D460" xlink:href="#MJX-1-TEX-I-1D460"></use></g><g data-mml-node="mo" transform="translate(469,0)"><use data-c="2212" xlink:href="#MJX-1-TEX-N-2212"></use></g><g data-mml-node="mi" transform="translate(1247,0)"><use data-c="1D456" xlink:href="#MJX-1-TEX-I-1D456"></use></g></g></g><g data-mml-node="mo" transform="translate(10662.5,0)"><use data-c="29" xlink:href="#MJX-1-TEX-N-29"></use></g></g></g></svg></mjx-container></span> where sigma is logistic sigmoid function</p>
<h2>新規性</h2>
<p>Docment-level NMTにおける従来手法の問題点</p>
<ol>
<li>文章間のreasoningの特徴づけを明示的に行うことなく，単純にcontextの分散表現を導入</li>
<li>推論時にはアクセスできないのに，訓練時には追加入力としてのtarget contextにground-truthなデータを入力
↑　訓練時と推論時において状況が異なる</li>
</ol>
<p>Document-level NMTにおいてMulti-Hop reasoningをモデリングしたMulti-Hop Transformerの提案と提案モデルによるDocument-level NMTの大きな性能改善</p>
<p>target contextにground-truthで訓練すると推論時にはアクセスできないため，他の翻訳モデルの翻訳結果を使用することで，訓練時と推論時の状況を同じにした</p>
<h2>実験</h2>
<p>Baseline</p>
<p>Transformer</p>
<p>CA-Transformer</p>
<p>CA-HAN</p>
<p>CADec</p>
<p>計算量のオーバーヘッドを改善するためSentence Encoderはそれぞれのsideでパラメータを共有</p>
<h2>まとめ</h2>
<p><img src="/images/article/Multi-Hop-Transformer-for-Document-Level-Machine-Translation/am19iape.png" alt=""></p>
<p>large-scaleな事前学習済み言語モデルを使用することなく，SoTA翻訳クオリティを達成</p>
<p><img src="/images/article/Multi-Hop-Transformer-for-Document-Level-Machine-Translation/z4jm21k3.png" alt=""></p>
<p>contextを付与するためのAttentionの構造は，ConcatやHierarchicalよりもMulti-HopなAttentionが効果があり</p>
<p><img src="/images/article/Multi-Hop-Transformer-for-Document-Level-Machine-Translation/tito19hb.png" alt=""></p>
<p>contextを考慮する幅のwindow sizeは大きくするほど効果が上がるわけではなく．3が最も良かった</p>
<p>4以上にすると悪化傾向らしく，本研究ではwindow size = 3 を採用</p>
<p><img src="/images/article/Multi-Hop-Transformer-for-Document-Level-Machine-Translation/pc76zuqn.png" alt=""></p>
<p>contextにおいてreasoningするときの方向は，一般的な読み順の通りleft-to-rightで順方向にreasoningさせた方が結果は良かった</p>
<p><img src="/images/article/Multi-Hop-Transformer-for-Document-Level-Machine-Translation/bx5z85gj.png" alt=""></p>
<p>訓練時と推論時にtarget draftに与える文章が異なる問題への対処に関する実験結果</p>
<p>Referenceはground-truthをtarget draftとして与えて訓練，Draftはpre-trained MT systemが生成した翻訳結果をtarget draftとして与えて訓練したモデル</p>
<p>Draftの方が結果がよく，pre-trained MT systemの生成結果をtarget draftとする方法によって訓練時と推論時のギャップの橋渡しになることを示唆する結果</p>
<h2>その他（なぜ通ったか？等）</h2>
<h2>次読みたい論文</h2>
<p><a href="/5955ca444629476ebf23e66629a2413f">Context-Aware Self-Attention Networks</a></p>
<h2>引用</h2>
<blockquote>
<p>@inproceedings{zhang-etal-2021-multi,
title = "Multi-Hop Transformer for Document-Level Machine Translation",
author = "Zhang, Long and
Zhang, Tong and
Zhang, Haibo and
Yang, Baosong and
Ye, Wei and
Zhang, Shikun",
booktitle = "Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
month = jun,
year = "2021",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "<a href="https://aclanthology.org/2021.naacl-main.309">https://aclanthology.org/2021.naacl-main.309</a>",
doi = "10.18653/v1/2021.naacl-main.309",
pages = "3953--3963",
abstract = "Document-level neural machine translation (NMT) has proven to be of profound value for its effectiveness on capturing contextual information. Nevertheless, existing approaches 1) simply introduce the representations of context sentences without explicitly characterizing the inter-sentence reasoning process; and 2) feed ground-truth target contexts as extra inputs at the training time, thus facing the problem of exposure bias. We approach these problems with an inspiration from human behavior {--} human translators ordinarily emerge a translation draft in their mind and progressively revise it according to the reasoning in discourse. To this end, we propose a novel Multi-Hop Transformer (MHT) which offers NMT abilities to explicitly model the human-like draft-editing and reasoning process. Specifically, our model serves the sentence-level translation as a draft and properly refines its representations by attending to multiple antecedent sentences iteratively. Experiments on four widely used document translation tasks demonstrate that our method can significantly improve document-level translation performance and can tackle discourse phenomena, such as coreference error and the problem of polysemy.",
}</p>
</blockquote><style>
mjx-container[jax="SVG"] {
  direction: ltr;
}

mjx-container[jax="SVG"] > svg {
  overflow: visible;
  min-height: 1px;
  min-width: 1px;
}

mjx-container[jax="SVG"] > svg a {
  fill: blue;
  stroke: blue;
}

mjx-container[jax="SVG"][display="true"] {
  display: block;
  text-align: center;
  margin: 1em 0;
}

mjx-container[jax="SVG"][display="true"][width="full"] {
  display: flex;
}

mjx-container[jax="SVG"][justify="left"] {
  text-align: left;
}

mjx-container[jax="SVG"][justify="right"] {
  text-align: right;
}

g[data-mml-node="merror"] > g {
  fill: red;
  stroke: red;
}

g[data-mml-node="merror"] > rect[data-background] {
  fill: yellow;
  stroke: none;
}

g[data-mml-node="mtable"] > line[data-line], svg[data-table] > g > line[data-line] {
  stroke-width: 70px;
  fill: none;
}

g[data-mml-node="mtable"] > rect[data-frame], svg[data-table] > g > rect[data-frame] {
  stroke-width: 70px;
  fill: none;
}

g[data-mml-node="mtable"] > .mjx-dashed, svg[data-table] > g > .mjx-dashed {
  stroke-dasharray: 140;
}

g[data-mml-node="mtable"] > .mjx-dotted, svg[data-table] > g > .mjx-dotted {
  stroke-linecap: round;
  stroke-dasharray: 0,140;
}

g[data-mml-node="mtable"] > g > svg {
  overflow: visible;
}

[jax="SVG"] mjx-tool {
  display: inline-block;
  position: relative;
  width: 0;
  height: 0;
}

[jax="SVG"] mjx-tool > mjx-tip {
  position: absolute;
  top: 0;
  left: 0;
}

mjx-tool > mjx-tip {
  display: inline-block;
  padding: .2em;
  border: 1px solid #888;
  font-size: 70%;
  background-color: #F8F8F8;
  color: black;
  box-shadow: 2px 2px 5px #AAAAAA;
}

g[data-mml-node="maction"][data-toggle] {
  cursor: pointer;
}

mjx-status {
  display: block;
  position: fixed;
  left: 1em;
  bottom: 1em;
  min-width: 25%;
  padding: .2em .4em;
  border: 1px solid #888;
  font-size: 90%;
  background-color: #F8F8F8;
  color: black;
}

foreignObject[data-mjx-xml] {
  font-family: initial;
  line-height: normal;
  overflow: visible;
}

mjx-container[jax="SVG"] path[data-c], mjx-container[jax="SVG"] use[data-c] {
  stroke-width: 3;
}
</style>

Highway Transformer: Self-Gating Enhanced Self-Attentive Networksのまとめ

【論文まとめ】Highway Transformer: Self-Gating Enhanced Self-Attentive Networks

<p>本記事において使用される図表は，原著論文内の図表を引用しています．</p>
<p>また，本記事の内容は，著者が論文を読み，メモとして短くまとめたものになります．必ずしも内容が正しいとは限らないこと，ご了承ください．</p>
<h2>論文情報</h2>
<p>タイトル: Highway Transformer: Self-Gating Enhanced Self-Attentive Networks</p>
<p>研究会: ACL</p>
<p>年度: 2020</p>
<p>キーワード: transformer, Highway Transformer, Gating Mechanism, Self-Dependency-Units (SDU)</p>
<p>URL: <a href="https://aclanthology.org/2020.acl-main.616.pdf">https://aclanthology.org/2020.acl-main.616.pdf</a></p>
<p>DOI: <a href="http://dx.doi.org/10.18653/v1/2020.acl-main.616">http://dx.doi.org/10.18653/v1/2020.acl-main.616</a></p>
<p>コード: <a href="https://github.com/cyk1337/Highway-Transformer">https://github.com/cyk1337/Highway-Transformer</a></p>
<p>データセット: Penn Tree Bank (PTB), enwik8</p>
<h2>概要</h2>
<p>LSTM-styleなSDUを提案</p>
<p>ゲートとしてSDUをTransformer内部に適用することにより，ハイパラをチューニングすることなく，Transformerの浅い層において，内在的な意味の重要性を捉え，より早い収束を可能に</p>
<h2>提案手法</h2>
<p><img src="/images/article/Highway-Transformer-Self-Gating-Enhanced-Self-Attentive-Networks/qrpajdel.png" alt=""></p>
<h3>Self-Dependency Units (SDU)</h3>
<p>sigmoid gatesを導入する</p>
<p><span class="math math-inline"><mjx-container class="MathJax" jax="SVG"><svg style="vertical-align: 0;" xmlns="http://www.w3.org/2000/svg" width="1.76ex" height="1.545ex" role="img" focusable="false" viewBox="0 -683 778 683" xmlns:xlink="http://www.w3.org/1999/xlink"><defs><path id="MJX-1-TEX-N-3A8" d="M340 622Q338 623 335 625T331 629T325 631T314 634T298 635T274 636T239 637H212V683H224Q248 680 389 680T554 683H566V637H539Q479 637 464 635T439 622L438 407Q438 192 439 192Q443 193 449 195T474 207T507 232T536 276T557 344Q560 365 562 417T573 493Q587 536 620 544Q627 546 671 546H715L722 540V515Q714 509 708 509Q680 505 671 476T658 392T644 307Q599 177 451 153L438 151V106L439 61Q446 54 451 52T476 48T539 46H566V0H554Q530 3 389 3T224 0H212V46H239Q259 46 273 46T298 47T314 48T325 51T331 54T335 57T340 61V151Q126 178 117 406Q115 503 69 509Q55 509 55 526Q55 541 59 543T86 546H107H120Q150 546 161 543T184 528Q198 514 204 493Q212 472 213 420T226 316T272 230Q287 216 303 207T330 194L339 192Q340 192 340 407V622Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="scale(1,-1)"><g data-mml-node="math"><g data-mml-node="mi"><use data-c="3A8" xlink:href="#MJX-1-TEX-N-3A8"></use></g></g></g></svg></mjx-container></span>はゲートとして作用し，logistic sigmoidもしくはtanhで実現</p>
<p>筆者らの認識</p>
<p>tanhはupdate gateとして作用し，重要度の幅を-1 to 1に制限</p>
<p>sigmoidはLSTMのinput gateと似ていて，feature-wise levelでどれくらいの情報を残すか決定</p>
<h3>Pseudo-highway Connection</h3>
<p>residual connectionされたgating-modified encodingsでMulti Head Dot Product Attention (MHDPA)の分散表現を豊かにするため，新たな計算グラフの枝を追加し，SDUとIdentityとMHDPAをpost LNを使用してresidual connectionする</p>
<h2>新規性</h2>
<p>本来，人にとって，読み物をよりよく理解するためには，global contextだけではなく，ここの単語の意味も必要</p>
<p>→ Self-gatingなアプローチを提案</p>
<ol>
<li>Transformerにおける浅い層において，trainingとvalidationでハイパラチューニングすることなく，より高速な収束を達成</li>
<li>Transformerでの低レイヤーにおいて，local-range encodingにフォーカスした層を実現</li>
<li>Self-gating mechanismは，R-TransformerやTransformer-XLのコンポーネントとしてRNN-likeなメカニズムを補完</li>
</ol>
<h2>実験</h2>
<p>SDUを導入し，PTBデータセットにおけるSDUの効果を検証</p>
<p>sigmoidとtanhを実験</p>
<h2>まとめ</h2>
<p><img src="/images/article/Highway-Transformer-Self-Gating-Enhanced-Self-Attentive-Networks/op874d4u.png" alt=""></p>
<p><img src="/images/article/Highway-Transformer-Self-Gating-Enhanced-Self-Attentive-Networks/e99q8luh.png" alt=""></p>
<p><img src="/images/article/Highway-Transformer-Self-Gating-Enhanced-Self-Attentive-Networks/4xk5l3fv.png" alt=""></p>
<p><img src="/images/article/Highway-Transformer-Self-Gating-Enhanced-Self-Attentive-Networks/gen0ole9.png" alt=""></p>
<p><img src="/images/article/Highway-Transformer-Self-Gating-Enhanced-Self-Attentive-Networks/bjrucnwe.png" alt=""></p>
<p><img src="/images/article/Highway-Transformer-Self-Gating-Enhanced-Self-Attentive-Networks/4fv3x3v3.png" alt=""></p>
<p><img src="/images/article/Highway-Transformer-Self-Gating-Enhanced-Self-Attentive-Networks/c6hfqjx9.png" alt=""></p>
<p><img src="/images/article/Highway-Transformer-Self-Gating-Enhanced-Self-Attentive-Networks/aptzk9jc.png" alt=""></p>
<p>sigmoidによるSDUが安定しているが，データとタスクによってはtanhの方がoutperformすることがある</p>
<p>いずれのactivationを使っても収束は早い</p>
<p>enwik8による大規模データでの追実験において，提案手法が浅いレイヤーには寄与することが確かめられた</p>
<p>SDUで計算量が増えるが，そこまで差はなかった</p>
<h2>その他（なぜ通ったか？等）</h2>
<h2>次読みたい論文</h2>
<h2>引用</h2>
<blockquote>
<p>@inproceedings{chai-etal-2020-highway,
title = "Highway Transformer: Self-Gating Enhanced Self-Attentive Networks",
author = "Chai, Yekun  and
Jin, Shuo  and
Hou, Xinwen",
booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics",
month = jul,
year = "2020",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "<a href="https://aclanthology.org/2020.acl-main.616">https://aclanthology.org/2020.acl-main.616</a>",
doi = "10.18653/v1/2020.acl-main.616",
pages = "6887--6900",
abstract = "Self-attention mechanisms have made striking state-of-the-art (SOTA) progress in various sequence learning tasks, standing on the multi-headed dot product attention by attending to all the global contexts at different locations. Through a pseudo information highway, we introduce a gated component self-dependency units (SDU) that incorporates LSTM-styled gating units to replenish internal semantic importance within the multi-dimensional latent space of individual representations. The subsidiary content-based SDU gates allow for the information flow of modulated latent embeddings through skipped connections, leading to a clear margin of convergence speed with gradient descent algorithms. We may unveil the role of gating mechanism to aid in the context-based Transformer modules, with hypothesizing that SDU gates, especially on shallow layers, could push it faster to step towards suboptimal points during the optimization process.",
}</p>
</blockquote><style>
mjx-container[jax="SVG"] {
  direction: ltr;
}

mjx-container[jax="SVG"] > svg {
  overflow: visible;
  min-height: 1px;
  min-width: 1px;
}

mjx-container[jax="SVG"] > svg a {
  fill: blue;
  stroke: blue;
}

mjx-container[jax="SVG"][display="true"] {
  display: block;
  text-align: center;
  margin: 1em 0;
}

mjx-container[jax="SVG"][display="true"][width="full"] {
  display: flex;
}

mjx-container[jax="SVG"][justify="left"] {
  text-align: left;
}

mjx-container[jax="SVG"][justify="right"] {
  text-align: right;
}

g[data-mml-node="merror"] > g {
  fill: red;
  stroke: red;
}

g[data-mml-node="merror"] > rect[data-background] {
  fill: yellow;
  stroke: none;
}

g[data-mml-node="mtable"] > line[data-line], svg[data-table] > g > line[data-line] {
  stroke-width: 70px;
  fill: none;
}

g[data-mml-node="mtable"] > rect[data-frame], svg[data-table] > g > rect[data-frame] {
  stroke-width: 70px;
  fill: none;
}

g[data-mml-node="mtable"] > .mjx-dashed, svg[data-table] > g > .mjx-dashed {
  stroke-dasharray: 140;
}

g[data-mml-node="mtable"] > .mjx-dotted, svg[data-table] > g > .mjx-dotted {
  stroke-linecap: round;
  stroke-dasharray: 0,140;
}

g[data-mml-node="mtable"] > g > svg {
  overflow: visible;
}

[jax="SVG"] mjx-tool {
  display: inline-block;
  position: relative;
  width: 0;
  height: 0;
}

[jax="SVG"] mjx-tool > mjx-tip {
  position: absolute;
  top: 0;
  left: 0;
}

mjx-tool > mjx-tip {
  display: inline-block;
  padding: .2em;
  border: 1px solid #888;
  font-size: 70%;
  background-color: #F8F8F8;
  color: black;
  box-shadow: 2px 2px 5px #AAAAAA;
}

g[data-mml-node="maction"][data-toggle] {
  cursor: pointer;
}

mjx-status {
  display: block;
  position: fixed;
  left: 1em;
  bottom: 1em;
  min-width: 25%;
  padding: .2em .4em;
  border: 1px solid #888;
  font-size: 90%;
  background-color: #F8F8F8;
  color: black;
}

foreignObject[data-mjx-xml] {
  font-family: initial;
  line-height: normal;
  overflow: visible;
}

mjx-container[jax="SVG"] path[data-c], mjx-container[jax="SVG"] use[data-c] {
  stroke-width: 3;
}
</style>