{"id":2684,"date":"2026-09-04T22:22:43","date_gmt":"2026-09-04T14:22:43","guid":{"rendered":"http:\/\/blog.xtaa.cn\/?p=2684"},"modified":"2026-09-05T00:30:34","modified_gmt":"2026-09-04T16:30:34","slug":"transformer%e5%b0%8f%e7%99%bd%e5%85%a5%e9%97%a8","status":"publish","type":"post","link":"http:\/\/blog.xtaa.cn\/index.php\/2026\/09\/04\/transformer%e5%b0%8f%e7%99%bd%e5%85%a5%e9%97%a8\/","title":{"rendered":"Transformer\u5c0f\u767d\u5165\u95e8"},"content":{"rendered":"\n

Transformer\u662f2017\u5e74\u8c37\u6b4c\u5728\u8bba\u6587\u300aAttention Is All You Need\u300b\u4e2d\u63d0\u51fa\u7684\u795e\u7ecf\u7f51\u7edc\u67b6\u6784\uff0c\u73b0\u5df2\u6210\u4e3aGPT\u3001BERT\u7b49\u6240\u6709\u5927\u8bed\u8a00\u6a21\u578b\u7684\u6838\u5fc3\u9aa8\u67b6\u3002<\/p>\n\n\n\n

\u65e9\u5e74\u7684AI\u600e\u4e48\u8bfb\u4e00\u53e5\u8bdd\uff1aRNN\u7684”\u4f20\u8bdd\u6e38\u620f”<\/strong><\/strong><\/h3>\n\n\n\n

\u5728Transformer\u51fa\u73b0\u4e4b\u524d\uff0cAI\u5904\u7406\u8bed\u8a00\u6700\u4e3b\u6d41\u7684\u65b9\u6cd5\u53eb\u505a\u5faa\u73af\u795e\u7ecf\u7f51\u7edc\uff0c\u7b80\u79f0RNN\uff0c\u540e\u6765\u53c8\u8fdb\u5316\u51faLSTM\u3001GRU\u7b49\u6539\u8fdb\u7248\u672c\uff0c\u4f46\u6838\u5fc3\u601d\u8def\u4e00\u8109\u76f8\u627f\u3002\u6211\u4eec\u53ef\u4ee5\u7528\u4e00\u4e2a\u7279\u522b\u751f\u6d3b\u5316\u7684\u573a\u666f\u6765\u7c7b\u6bd4\u2014\u2014\u5c0f\u5b66\u5e38\u73a9\u7684”\u4f20\u8bdd\u6e38\u620f”\uff1a\u5168\u73ed\u540c\u5b66\u6392\u6210\u4e00\u5217\uff0c\u8001\u5e08\u628a\u4e00\u53e5\u8bdd\u6084\u6084\u544a\u8bc9\u7b2c\u4e00\u4e2a\u4eba\uff0c\u7b2c\u4e00\u4e2a\u4eba\u542c\u5b8c\u518d\u5c0f\u58f0\u4f20\u7ed9\u7b2c\u4e8c\u4e2a\u4eba\uff0c\u7b2c\u4e8c\u4e2a\u4eba\u4f20\u7ed9\u7b2c\u4e09\u4e2a\uff0c\u4e00\u76f4\u4f20\u5230\u6700\u540e\u4e00\u4e2a\u4eba\uff0c\u7531\u4ed6\u5927\u58f0\u8bf4\u51fa\u542c\u5230\u7684\u5185\u5bb9\u3002<\/p>\n\n\n\n

RNN\u8bfb\u53e5\u5b50\u7684\u65b9\u5f0f\uff0c\u5c31\u8ddf\u8fd9\u4e2a\u6e38\u620f\u51e0\u4e4e\u4e00\u6a21\u4e00\u6837[3][4]\u3002\u5f53\u4f60\u8f93\u5165”\u4eca\u5929\u5929\u6c14\u771f\u597d”\u8fd9\u53e5\u8bdd\u65f6\uff0cRNN\u4f1a\u4ece\u5de6\u5230\u53f3\u4e00\u4e2a\u5b57\u4e00\u4e2a\u5b57\u5730\u8bfb\uff1a\u5148\u8bfb”\u4eca”\uff0c\u628a\u8111\u5b50\u91cc\u8bb0\u4f4f\u7684”\u76ee\u524d\u8bfb\u5230\u4e86’\u4eca'”\u8fd9\u4e2a\u4fe1\u606f\u6253\u5305\uff0c\u8fde\u540c\u4e0b\u4e00\u4e2a\u5b57”\u5929”\u4e00\u8d77\u4f20\u7ed9\u4e0b\u4e00\u4e2a\u5904\u7406\u5355\u5143\uff1b\u7136\u540e\u628a”\u4eca\u5929”\u7684\u4fe1\u606f\u7ee7\u7eed\u4f20\u7ed9\u8bfb”\u5929”\u7684\u5355\u5143\u2026\u2026\u76f4\u5230\u8bfb\u5b8c\u6700\u540e\u4e00\u4e2a\u5b57”\u597d”\u3002\u6bcf\u4e2a\u65f6\u523b\uff0cAI\u90fd\u53ea\u80fd\u770b\u5230”\u5f53\u524d\u8fd9\u4e2a\u5b57”\u52a0\u4e0a”\u4e0a\u4e00\u65f6\u523b\u4f20\u8fc7\u6765\u7684\u8bb0\u5fc6”\uff0c\u5c31\u50cf\u4f20\u8bdd\u6e38\u620f\u91cc\u7684\u6bcf\u4e00\u4e2a\u540c\u5b66\uff0c\u53ea\u80fd\u542c\u89c1\u524d\u4e00\u4e2a\u4eba\u8bf4\u7684\u8bdd\uff0c\u542c\u4e0d\u89c1\u66f4\u65e9\u7684\u540c\u5b66\u8bf4\u4ec0\u4e48\uff0c\u66f4\u4e0d\u80fd\u56de\u5934\u95ee\u3002<\/p>\n\n\n\n

\u8fd9\u79cd\u8bbe\u8ba1\u5728\u77ed\u53e5\u5b50\u4e0a\u52c9\u5f3a\u80fd\u5de5\u4f5c\uff0c\u5c31\u50cf\u4f20\u8bdd\u6e38\u620f\u53ea\u6709\u4e09\u4e94\u4e2a\u540c\u5b66\u65f6\uff0c\u4fe1\u606f\u504f\u5dee\u4e0d\u5927\u3002\u4f46\u4e00\u65e6\u53e5\u5b50\u53d8\u957f\uff0c\u95ee\u9898\u5c31\u66b4\u9732\u4e86\u3002\u7b2c\u4e00\u4e2a\u95ee\u9898\u53eb”\u4fe1\u606f\u8870\u51cf”\uff1a\u4e00\u53e5\u8bdd\u5982\u679c\u6709\u4e94\u516d\u5341\u4e2a\u5b57\uff0c\u4fe1\u606f\u7ecf\u8fc7\u4e94\u516d\u5341\u6b21\u4f20\u9012\uff0c\u6700\u65e9\u51fa\u73b0\u7684\u5173\u952e\u8bcd\u4f20\u5230\u6700\u540e\u51e0\u4e4e\u5df2\u7ecf”\u9762\u76ee\u5168\u975e”\u4e86\u3002\u6bd4\u5982\u53e5\u5b50”\u5c0f\u660e\u53bb\u5e74\u5728\u5df4\u9ece\u7559\u5b66\u65f6\u8ba4\u8bc6\u4e86\u4e00\u4f4d\u70ed\u7231\u70d8\u7119\u7684\u59d1\u5a18\uff0c\u4eca\u5e74\u590f\u5929\u56de\u5230\u5317\u4eac\u5f00\u4e86\u4e00\u5bb6\u5c0f\u5496\u5561\u9986\uff0c\u4ed6\u2026\u2026”\u5f53RNN\u8bfb\u5230\u540e\u9762\u7684”\u4ed6”\u65f6\uff0c\u524d\u9762\u7684”\u5c0f\u660e”\u65e9\u5c31\u5728\u51e0\u5341\u6b21\u4f20\u9012\u4e2d\u88ab\u7a00\u91ca\u5f97\u51e0\u4e4e\u6ca1\u5f71\u4e86\uff0cAI\u5f88\u96be\u5224\u65ad”\u4ed6”\u5230\u5e95\u6307\u8c01[13][14]\u3002\u7b2c\u4e8c\u4e2a\u95ee\u9898\u53eb”\u65e0\u6cd5\u5e76\u884c”\uff1a\u56e0\u4e3a\u6bcf\u4e2a\u6b65\u9aa4\u5fc5\u987b\u7b49\u4e0a\u4e00\u6b65\u4f20\u5b8c\u624d\u80fd\u5f00\u59cb\uff0c\u5c31\u50cf\u4f20\u8bdd\u6e38\u620f\u91cc\u7b2c10\u4e2a\u4eba\u5fc5\u987b\u7b49\u7b2c9\u4e2a\u4eba\u8bf4\u5b8c\uff0c\u7b2c9\u4e2a\u4eba\u53c8\u5fc5\u987b\u7b49\u7b2c8\u4e2a\u4eba\uff0c\u6574\u6761\u6d41\u6c34\u7ebf\u5b8c\u5168\u662f\u4e32\u884c\u7684\uff0c\u5c31\u7b97\u4f60\u6709\u4e00\u4e07\u4e2aCPU\u6838\u5fc3\u4e5f\u5e2e\u4e0d\u4e0a\u5fd9\uff0c\u8bad\u7ec3\u901f\u5ea6\u6162\u5f97\u8ba9\u4eba\u6293\u72c2[9][13]\u3002<\/p>\n\n\n\n

\u4e3a\u4e86\u7f13\u89e3\u4fe1\u606f\u8870\u51cf\uff0c\u7814\u7a76\u8005\u53d1\u660e\u4e86LSTM\u548cGRU\uff0c\u76f8\u5f53\u4e8e\u5728\u4f20\u8bdd\u961f\u4f0d\u91cc\u8bbe\u4e86\u51e0\u4e2a”\u8bb0\u6027\u597d\u7684\u540c\u5b66”\uff0c\u4e13\u95e8\u8d1f\u8d23\u628a\u5173\u952e\u4fe1\u606f\u5c3d\u91cf\u4fdd\u7559\u5230\u540e\u9762\u3002\u8fd9\u786e\u5b9e\u6709\u4e00\u5b9a\u6539\u5584\uff0c\u4f46\u672c\u8d28\u4e0a\u4f9d\u7136\u662f\u4ece\u5de6\u5230\u53f3\u4e32\u884c\u3001\u4f9d\u7136\u662f”\u4e00\u4e2a\u4f20\u4e00\u4e2a”\u7684\u6e38\u620f\u89c4\u5219\uff0c\u957f\u8ddd\u79bb\u4f9d\u8d56\u548c\u5e76\u884c\u6162\u8fd9\u4e24\u4e2a\u6839\u672c\u95ee\u9898\u5e76\u6ca1\u6709\u88ab\u771f\u6b63\u89e3\u51b3\u3002AI\u60f3\u771f\u6b63”\u8bfb\u61c2”\u8bed\u8a00\uff0c\u9700\u8981\u4e00\u79cd\u66f4\u806a\u660e\u7684\u6c9f\u901a\u65b9\u5f0f\u3002<\/p>\n\n\n\n

1.2 \u8f6c\u6298\u70b9\uff1a\u5982\u679c\u6240\u6709\u4eba\u90fd\u80fd\u540c\u65f6\u8bf4\u8bdd\u4f1a\u600e\u6837<\/strong><\/strong><\/h3>\n\n\n\n

\u6545\u4e8b\u5230\u8fd9\u91cc\uff0cTransformer\u7684\u6838\u5fc3\u7075\u611f\u5176\u5b9e\u53ea\u5dee\u4e00\u4e2a\u601d\u60f3\u4e0a\u7684\u8f6c\u5f2f\uff1a\u65e2\u7136”\u9010\u4eba\u4f20\u8bdd”\u4f1a\u5931\u771f\u3001\u4f1a\u6162\uff0c\u90a3\u6211\u4eec\u4e3a\u4ec0\u4e48\u4e0d\u8ba9\u4e00\u53e5\u8bdd\u91cc\u7684\u6240\u6709\u5b57\uff0c\u50cf\u5f00”\u5168\u5458\u7535\u8bdd\u4f1a\u8bae”\u4e00\u6837\uff0c\u5f7c\u6b64\u4e4b\u95f4\u53ef\u4ee5\u76f4\u63a5\u4ea4\u6d41\u5462[3][4]\uff1f<\/p>\n\n\n\n

\u60f3\u8c61\u8fd9\u6837\u4e00\u4e2a\u573a\u666f\uff1a\u516c\u53f8\u5f00\u4f1a\uff0c\u5982\u679c\u8fd8\u662f\u50cf\u4f20\u8bdd\u6e38\u620f\u90a3\u6837\u7531CEO\u4e00\u4e2a\u4f20\u4e00\u4e2a\u5730\u628a\u4fe1\u606f\u4f20\u5230\u57fa\u5c42\uff0c\u4e0d\u4ec5\u6162\uff0c\u4f20\u5230\u6700\u540e\u610f\u601d\u4e5f\u4f1a\u8d70\u6837\u3002\u4f46\u5982\u679c\u5f00\u4e00\u4e2a\u5168\u5458\u7535\u8bdd\u4f1a\u8bae\uff0cCEO\u5728\u8bf4\u65b9\u6848\u7684\u540c\u65f6\uff0c\u5e02\u573a\u90e8\u53ef\u4ee5\u7acb\u523b\u5bf9\u63a5\u4ea7\u54c1\u90e8\uff0c\u4ea7\u54c1\u90e8\u53c8\u80fd\u76f4\u63a5\u95ee\u6280\u672f\u90e8\u67d0\u4e2a\u7ec6\u8282\u662f\u5426\u53ef\u884c\uff0c\u4efb\u610f\u4e24\u4e2a\u4eba\u4e4b\u95f4\u90fd\u6709\u4e00\u6761\u76f4\u8fde\u7684\u7535\u8bdd\u7ebf\uff0c\u4e0d\u9700\u8981\u7ecf\u8fc7\u4e2d\u95f4\u4efb\u4f55\u4e00\u4e2a\u4eba\u4f20\u8bdd\u3002\u8fd9\u6837\u4e00\u6765\uff0c\u4fe1\u606f\u51e0\u4e4e\u96f6\u635f\u8017\uff0c\u800c\u4e14\u6240\u6709\u4eba\u53ef\u4ee5\u540c\u65f6\u53d1\u8a00\u3001\u540c\u65f6\u63a5\u6536\uff0c\u6548\u7387\u62c9\u6ee1\u3002\u8fd9\u5c31\u662fTransformer\u6700\u9769\u547d\u6027\u7684\u60f3\u6cd5\uff1a\u8ba9\u4e00\u53e5\u8bdd\u91cc\u7684\u6bcf\u4e00\u4e2a\u8bcd\uff0c\u90fd\u80fd”\u76f4\u63a5\u770b\u5230”\u8fd9\u53e5\u8bdd\u91cc\u7684\u5176\u4ed6\u6240\u6709\u8bcd\uff0c\u5e76\u4e14\u81ea\u5df1\u5224\u65ad”\u6211\u8be5\u91cd\u70b9\u5173\u6ce8\u8c01”[3][9]\u3002<\/p>\n\n\n\n

\u8fd8\u662f\u7528\u524d\u9762\u90a3\u4e2a\u4f8b\u5b50\uff1a”\u5c0f\u660e\u53bb\u5e74\u5728\u5df4\u9ece\u7559\u5b66\u65f6\u8ba4\u8bc6\u4e86\u4e00\u4f4d\u70ed\u7231\u70d8\u7119\u7684\u59d1\u5a18\u2026\u2026\u4ed6\u2026\u2026”\u5728Transformer\u7684\u4e16\u754c\u91cc\uff0c\u5f53AI\u5904\u7406\u5230”\u4ed6”\u8fd9\u4e2a\u5b57\u65f6\uff0c\u4e0d\u9700\u8981\u7b49\u4fe1\u606f\u7ecf\u8fc7\u51e0\u5341\u6b65\u6162\u6162\u4f20\u8fc7\u6765\uff0c\u800c\u662f\u53ef\u4ee5\u76f4\u63a5”\u6253\u7535\u8bdd”\u7ed9”\u5c0f\u660e”\u3001\u6253\u7ed9”\u59d1\u5a18”\u3001\u6253\u7ed9”\u4e00\u4f4d”\uff0c\u7136\u540e\u6839\u636e\u8bed\u4e49\u81ea\u52a8\u5224\u65ad\u2014\u2014”\u4ed6”\u6700\u5e94\u8be5\u8fde\u63a5\u7684\u662f”\u5c0f\u660e”\uff0c\u4e8e\u662f\u628a\u6ce8\u610f\u529b\u96c6\u4e2d\u5728”\u5c0f\u660e”\u8eab\u4e0a\uff0c\u5176\u4ed6\u8bcd\u53ea\u662f\u987a\u5e26\u53c2\u8003[14]\u3002\u66f4\u5999\u7684\u662f\uff0c\u8fd9\u79cd”\u6253\u7535\u8bdd”\u662f\u6240\u6709\u8bcd\u540c\u65f6\u8fdb\u884c\u7684\uff1a\u5904\u7406”\u4ed6”\u7684\u65f6\u5019\uff0c”\u5c0f\u660e”\u4e5f\u5728\u8ddf\u5176\u4ed6\u8bcd\u8fde\u7ebf\uff0c”\u5df4\u9ece”\u4e5f\u5728\u8ddf”\u59d1\u5a18”\u8fde\u7ebf\uff0c\u6574\u5f20\u5173\u7cfb\u7f51\u662f\u4e00\u6b21\u6027\u7b97\u51fa\u6765\u7684\uff0c\u800c\u4e0d\u662f\u50cfRNN\u90a3\u6837\u4e00\u4e2a\u5b57\u4e00\u4e2a\u5b57\u6392\u961f\u3002\u8fd9\u5c31\u662f\u8bba\u6587\u6807\u9898”Attention Is All You Need”\uff08\u4f60\u53ea\u9700\u8981\u6ce8\u610f\u529b\uff09\u60f3\u8981\u8868\u8fbe\u7684\u6838\u5fc3\uff1a\u4ee5\u524d\u7684AI\u8981\u9760\u8bb0\u5fc6\u4e00\u6b65\u6b65\u4f20\uff0c\u73b0\u5728\u5e72\u8106\u8ba9\u6bcf\u4e2a\u8bcd\u90fd\u62e5\u6709”\u4e3b\u52a8\u5173\u6ce8\u8c01”\u7684\u80fd\u529b\uff0c\u9760\u6ce8\u610f\u529b\u76f4\u63a5\u8fde\u7ebf\uff0c\u5c31\u591f\u4e86[9]\u3002<\/p>\n\n\n\n

\u8fd9\u4e2a\u60f3\u6cd5\u57282017\u5e74\u521a\u63d0\u51fa\u65f6\u76f8\u5f53\u6fc0\u8fdb\uff0c\u56e0\u4e3a\u5b83\u5f7b\u5e95\u4e22\u6389\u4e86RNN\u548cCNN\u7684\u7ed3\u6784\uff0c\u53ea\u7528\u6ce8\u610f\u529b\u673a\u5236\u6765\u505a\u5e8f\u5217\u5efa\u6a21\u3002\u4f46\u6548\u679c\u60ca\u4eba\uff1a\u4e0d\u4ec5\u7ffb\u8bd1\u51c6\u786e\u7387\u5927\u5e45\u8d85\u8d8a\u5f53\u65f6\u6700\u597d\u7684\u6a21\u578b\uff0c\u8bad\u7ec3\u901f\u5ea6\u8fd8\u56e0\u4e3a\u53ef\u4ee5\u5e76\u884c\u8ba1\u7b97\u800c\u5feb\u4e86\u51e0\u5341\u500d\u3002\u4e00\u573aAI\u5e95\u5c42\u67b6\u6784\u7684\u9769\u547d\uff0c\u5c31\u4ece\u8fd9\u4e2a”\u5168\u5458\u4f1a\u8bae”\u7684\u7b80\u5355\u7c7b\u6bd4\u5f00\u59cb\u4e86\u3002<\/p>\n\n\n\n

1.3 \u4ece\u4e00\u53e5\u8bdd\u5230ChatGPT\uff1aTransformer\u5982\u4f55\u6539\u53d8\u4e86AI\u7684\u547d\u8fd0<\/strong><\/strong><\/h3>\n\n\n\n

Transformer\u8bde\u751f\u4e4b\u540e\uff0cAI\u7406\u89e3\u8bed\u8a00\u7684\u80fd\u529b\u8fdb\u5165\u4e86\u5feb\u8f66\u9053\u3002\u7814\u7a76\u8005\u5f88\u5feb\u53d1\u73b0\uff0c\u8fd9\u4e2a”\u5168\u5458\u5f00\u4f1a”\u7684\u67b6\u6784\u4e0d\u4ec5\u80fd\u5904\u7406\u7ffb\u8bd1\uff0c\u8fd8\u51e0\u4e4e\u9002\u5408\u6240\u6709\u8bed\u8a00\u4efb\u52a1\uff1a\u53ea\u8981\u7ed9\u5b83\u8db3\u591f\u591a\u7684\u6587\u672c\uff0c\u5b83\u5c31\u80fd\u5b66\u4f1a\u8bed\u6cd5\u3001\u5e38\u8bc6\u3001\u903b\u8f91\u751a\u81f3\u7b80\u5355\u7684\u63a8\u7406\u3002\u539f\u56e0\u5176\u5b9e\u85cf\u5728\u6211\u4eec\u524d\u4e24\u8282\u8bb2\u7684\u4e24\u4e2a\u7279\u6027\u91cc\uff1a\u4e00\u662f\u4efb\u610f\u4e24\u8bcd\u53ef\u76f4\u8fde\uff0c\u957f\u8ddd\u79bb\u4fe1\u606f\u4e0d\u4f1a\u4e22\u5931\uff0c\u8fd9\u8ba9AI\u8bfb\u957f\u6587\u7ae0\u3001\u5199\u957f\u56de\u590d\u6210\u4e3a\u53ef\u80fd\uff1b\u4e8c\u662f\u9ad8\u5ea6\u5e76\u884c\uff0c\u8fd9\u8ba9\u7814\u7a76\u8005\u6562\u4e8e\u628a\u6a21\u578b\u5806\u5230\u51e0\u5341\u4ebf\u3001\u51e0\u767e\u4ebf\u751a\u81f3\u4e0a\u5343\u4ebf\u53c2\u6570\uff0c\u7528\u6d77\u91cf\u6570\u636e\u53bb\u8bad\u7ec3\uff0c\u800c\u4e0d\u7528\u62c5\u5fc3\u8bad\u7ec3\u65f6\u95f4\u957f\u5230\u4e0d\u53ef\u63a5\u53d7[9][13]\u3002<\/p>\n\n\n\n

\u57fa\u4e8eTransformer\uff0c2018\u5e74\u8c37\u6b4c\u63a8\u51fa\u4e86BERT\uff0c\u64c5\u957f\u9605\u8bfb\u7406\u89e3\uff1b\u540c\u5e74OpenAI\u63a8\u51fa\u4e86GPT-1\uff0c\u8d70\u4e0a\u4e86\u751f\u6210\u5f0f\u9884\u8bad\u7ec3\u7684\u8def\u7ebf\u3002\u4e4b\u540eGPT-2\u3001GPT-3\u4e00\u8def\u6f14\u8fdb\uff0c\u52302022\u5e74\u5e95ChatGPT\u6a2a\u7a7a\u51fa\u4e16\uff0c\u80fd\u8ddf\u4eba\u6d41\u7545\u5bf9\u8bdd\u3001\u5199\u4ee3\u7801\u3001\u505a\u65b9\u6848\uff0c\u80cc\u540e\u5168\u662fTransformer\u8fd9\u5957\u9aa8\u67b6\u5728\u652f\u6491\u3002\u53ef\u4ee5\u8bf4\uff0c\u4eca\u5929\u4f60\u7528\u5230\u7684\u6bcf\u4e00\u4e2a\u5927\u8bed\u8a00\u6a21\u578b\uff0c\u672c\u8d28\u4e0a\u90fd\u662f\u4e00\u4e2a\u628aTransformer\u5806\u6df1\u3001\u5806\u5927\u3001\u5582\u8db3\u6570\u636e\u4e4b\u540e\u7684\u7ed3\u679c\u3002\u66f4\u6709\u610f\u601d\u7684\u662f\uff0cTransformer\u7684\u80fd\u529b\u5e76\u4e0d\u5c40\u9650\u4e8e\u8bed\u8a00\uff1a\u89c6\u89c9\u9886\u57df\u7684ViT\u628a\u56fe\u7247\u5207\u6210\u5c0f\u5757\u5f53”\u8bcd”\u9001\u8fdbTransformer\uff0c\u5c31\u80fd\u505a\u56fe\u50cf\u5206\u7c7b\uff1b\u8bed\u97f3\u3001\u591a\u6a21\u6001\u3001\u751a\u81f3\u86cb\u767d\u8d28\u7ed3\u6784\u9884\u6d4b\u91cc\uff0c\u90fd\u80fd\u770b\u5230\u5b83\u7684\u8eab\u5f71\u3002\u5b83\u4ece\u4e00\u4e2a”\u66f4\u597d\u7684\u7ffb\u8bd1\u6a21\u578b”\u8fdb\u5316\u6210\u4e86\u901a\u7528AI\u7684\u57fa\u7840\u5e95\u5ea7\u3002<\/p>\n\n\n\n

\u5f53\u7136\uff0cTransformer\u4e5f\u4e0d\u662f\u5b8c\u7f8e\u7684\u3002\u6bd4\u5982”\u5168\u5458\u5f00\u4f1a”\u867d\u7136\u6c9f\u901a\u987a\u7545\uff0c\u4f46\u5982\u679c\u4e00\u53e5\u8bdd\u6709\u51e0\u5343\u4e2a\u5b57\uff0c\u6bcf\u4e2a\u4eba\u90fd\u8981\u8ddf\u5176\u4ed6\u6240\u6709\u4eba\u6253\u4e00\u6b21\u7535\u8bdd\uff0c\u8ba1\u7b97\u91cf\u4f1a\u968f\u53e5\u5b50\u957f\u5ea6\u5e73\u65b9\u7ea7\u589e\u957f\uff0c\u8fd9\u662f\u5927\u6a21\u578b\u5904\u7406\u8d85\u957f\u6587\u672c\u65f6\u6700\u5934\u75bc\u7684\u95ee\u9898\u4e4b\u4e00\uff1b\u518d\u6bd4\u5982\uff0c\u56e0\u4e3a\u6240\u6709\u8bcd\u540c\u65f6\u5230\u573a\uff0c”\u987a\u5e8f”\u8fd9\u4ef6\u4e8b\u53cd\u800c\u6210\u4e86\u96be\u9898\u2014\u2014”\u732b\u8ffd\u72d7”\u548c”\u72d7\u8ffd\u732b”\u5728\u5168\u5458\u4f1a\u8bae\u91cc\u770b\u8d77\u6765\u662f\u540c\u4e00\u62e8\u4eba\u5728\u5f00\u4f1a\uff0c\u610f\u601d\u5374\u5b8c\u5168\u76f8\u53cd\uff0c\u8fd9\u5c31\u9700\u8981\u6211\u4eec\u540e\u9762\u8981\u8bb2\u5230\u7684”\u4f4d\u7f6e\u7f16\u7801”\u6765\u7ed9\u6bcf\u4e2a\u8bcd\u8d34\u5ea7\u4f4d\u53f7[12]\u3002\u4f46\u8fd9\u4e9b\u90fd\u662f\u540e\u7eed\u7ae0\u8282\u8981\u6162\u6162\u62c6\u89e3\u7684\u5185\u5bb9\u3002<\/p>\n\n\n\n

\u8bfb\u5230\u8fd9\u91cc\uff0c\u4f60\u53ea\u9700\u8981\u8bb0\u4f4f\u4e00\u53e5\u8bdd\uff1aRNN\u662f”\u9010\u4eba\u4f20\u8bdd”\uff0c\u53c8\u6162\u53c8\u5bb9\u6613\u5931\u771f\uff1bTransformer\u662f”\u5168\u5458\u7535\u8bdd\u4f1a\u8bae”\uff0c\u4efb\u610f\u4e24\u8bcd\u76f4\u8fde\uff0c\u53c8\u5feb\u53c8\u80fd\u770b\u6e05\u5168\u5c40\u3002\u7406\u89e3\u4e86\u8fd9\u4e24\u79cd\u6c9f\u901a\u65b9\u5f0f\u7684\u5dee\u522b\uff0c\u4f60\u5c31\u5df2\u7ecf\u63a8\u5f00\u4e86Transformer\u5927\u95e8\u7684\u7b2c\u4e00\u9053\u7f1d\u3002\u63a5\u4e0b\u6765\u7684\u51e0\u7ae0\uff0c\u6211\u4eec\u4f1a\u8d70\u8fdb\u8fd9\u573a”\u4f1a\u8bae”\u7684\u5185\u90e8\uff0c\u770b\u770b\u6bcf\u4e2a\u8bcd\u5230\u5e95\u662f\u600e\u4e48”\u6253\u7535\u8bdd”\u7684\u2014\u2014\u90a3\u5c31\u662f\u5927\u540d\u9f0e\u9f0e\u7684\u81ea\u6ce8\u610f\u529b\u673a\u5236\u3002<\/p>\n\n\n\n

\u7b2c2\u7ae0 \u5f7b\u5e95\u641e\u61c2\u81ea\u6ce8\u610f\u529b\u673a\u5236\u2014\u2014\u6838\u5fc3\u539f\u7406\u7684\u751f\u6d3b\u5316\u89e3\u8bfb<\/strong><\/strong><\/h2>\n\n\n\n

\u5b66\u5b8c\u4e0a\u4e00\u7ae0\uff0c\u6211\u4eec\u5df2\u7ecf\u77e5\u9053Transformer\u662f\u4e3a\u4e86\u89e3\u51b3RNN”\u4f20\u8bdd\u6e38\u620f”\u91cc\u4fe1\u606f\u8870\u51cf\u3001\u4e0d\u80fd\u5e76\u884c\u4e24\u5927\u75db\u70b9\u624d\u88ab\u53d1\u660e\u51fa\u6765\u7684[3][4]\u3002\u4f46\u5b83\u5230\u5e95\u51ed\u4ec0\u4e48\u4e00\u53e5\u8bdd\u5c31\u80fd\u505a\u5230”\u5168\u5458\u540c\u65f6\u5728\u7ebf”\uff1f\u8fd9\u4e00\u7ae0\u7684\u4e3b\u89d2\u2014\u2014\u81ea\u6ce8\u610f\u529b\uff08Self-Attention\uff09<\/strong>\u2014\u2014\u5c31\u662f\u8fd9\u4e00\u5207\u9b54\u529b\u7684\u6e90\u5934\u3002\u5b83\u7684\u6838\u5fc3\u601d\u60f3\u542c\u4e0a\u53bb\u4e0d\u53ef\u601d\u8bae\uff1a\u5f53\u673a\u5668\u8bfb\u4e00\u53e5\u8bdd\u91cc\u7684\u67d0\u4e00\u4e2a\u8bcd\u65f6\uff0c\u5b83\u4f1a\u540c\u65f6”\u56de\u5934\u770b”\u8fd9\u53e5\u8bdd\u91cc\u7684\u6240\u6709\u5176\u4ed6\u8bcd<\/strong>\uff0c\u5e76\u81ea\u52a8\u5224\u65ad\u8c01\u548c\u5f53\u524d\u8fd9\u4e2a\u8bcd\u6700\u6709\u5173\u7cfb\uff0c\u518d\u628a\u90a3\u4e9b\u76f8\u5173\u8bcd\u7684\u4fe1\u606f”\u501f\u8fc7\u6765”\u5e2e\u52a9\u7406\u89e3\u5f53\u524d\u8bcd[7][14]\u3002\u4e0b\u9762\u6211\u4eec\u7528\u4e09\u4e2a\u6700\u751f\u6d3b\u5316\u7684\u573a\u666f\uff0c\u628a\u8fd9\u4e2a\u770b\u4f3c\u7384\u4e4e\u7684\u6982\u5ff5\u5f7b\u5e95\u8bb2\u900f\u3002<\/p>\n\n\n\n

2.1 “\u5b83”\u5230\u5e95\u6307\u8c01\uff1a\u4e00\u4e2a\u4ee3\u8bcd\u63ed\u5f00\u81ea\u6ce8\u610f\u529b\u7684\u771f\u672c\u4e8b<\/strong><\/strong><\/h3>\n\n\n\n

\u8981\u611f\u53d7\u81ea\u6ce8\u610f\u529b\u6709\u591a\u5389\u5bb3\uff0c\u6700\u7b80\u5355\u7684\u529e\u6cd5\u662f\u770b\u4e00\u53e5\u5e26\u4ee3\u8bcd\u7684\u8bdd\u3002Jay Alammar\u5728\u90a3\u7bc7\u88ab\u5168\u7403\u516c\u8ba4\u4e3a\u5165\u95e8\u795e\u6587\u7684\u300aThe Illustrated Transformer\u300b\u91cc\uff0c\u4e3e\u8fc7\u4e00\u4e2a\u7ecf\u5178\u4f8b\u53e5\uff1a“The animal didn’t cross the street because it was too tired.”<\/em>\uff08\u90a3\u53ea\u52a8\u7269\u6ca1\u6709\u7a7f\u8fc7\u8857\u9053\uff0c\u56e0\u4e3a\u5b83\u592a\u7d2f\u4e86\uff09[7][14]\u3002\u8fd9\u53e5\u8bdd\u91cc\u7684”it\uff08\u5b83\uff09”\u5230\u5e95\u6307\u8c01\uff1f\u4efb\u4f55\u4e00\u4e2a\u5c0f\u5b66\u751f\u90fd\u80fd\u79d2\u7b54\uff1a\u6307”animal\uff08\u52a8\u7269\uff09”\u3002\u4f46\u4f60\u6709\u6ca1\u6709\u60f3\u8fc7\uff0c\u4e3a\u4ec0\u4e48\u4e0d\u662f”street\uff08\u8857\u9053\uff09”\uff1f\u56e0\u4e3a\u4f60\u5728\u770b\u5230”it”\u7684\u77ac\u95f4\uff0c\u5927\u8111\u4f1a\u81ea\u52a8\u56de\u5934\u626b\u4e00\u904d\u524d\u9762\u51fa\u73b0\u8fc7\u7684\u6240\u6709\u540d\u8bcd\uff0c\u7136\u540e\u6839\u636e”\u592a\u7d2f”\u8fd9\u4e2a\u7ebf\u7d22\uff0c\u628a\u5b83\u548c”\u52a8\u7269”\u7ed1\u5b9a\uff0c\u800c\u4e0d\u662f\u548c”\u8857\u9053”\u7ed1\u5b9a\u2014\u2014\u8857\u9053\u662f\u4e0d\u4f1a\u7d2f\u7684\u3002<\/p>\n\n\n\n

\u53ef\u8fd9\u4ef6\u4e8b\u5bf9\u4f20\u7edfRNN\u6765\u8bf4\u5374\u76f8\u5f53\u5403\u529b\u3002RNN\u662f\u6309”\u4f20\u8bdd\u6e38\u620f”\u7684\u65b9\u5f0f\u4ece\u5de6\u8bfb\u5230\u53f3\uff1aanimal\u2192didn’t\u2192cross\u2192\u2026\u2192it\u3002\u5f53\u5b83\u8bfb\u5230”it”\u65f6\uff0c\u5173\u4e8eanimal\u7684\u4fe1\u606f\u5df2\u7ecf\u7ecf\u8fc7\u4e03\u516b\u6b21\u4f20\u9012\uff0c\u8870\u51cf\u5f97\u4e03\u4e03\u516b\u516b\u4e86\uff0c\u5b83\u5f88\u53ef\u80fd\u7cca\u91cc\u7cca\u6d82\u5730\u628a”it”\u548c\u6700\u8fd1\u7684”street”\u626f\u4e0a\u5173\u7cfb[3][13]\u3002\u800c\u81ea\u6ce8\u610f\u529b\u7684\u505a\u6cd5\u5b8c\u5168\u4e0d\u540c\uff1a\u5f53\u6a21\u578b\u5904\u7406”it”\u8fd9\u4e2a\u8bcd\u7684\u65f6\u5019\uff0c\u5b83\u4f1a\u540c\u65f6<\/strong>\u7ed9\u53e5\u5b50\u91cc\u7684\u6bcf\u4e00\u4e2a\u8bcd\u90fd\u6253\u4e00\u4e2a”\u76f8\u5173\u5ea6\u5206\u6570”\u2014\u2014animal\u5f97\u5206\u5f88\u9ad8\uff0cstreet\u5f97\u5206\u5f88\u4f4e\uff0ctired\u5f97\u5206\u4e5f\u4e0d\u4f4e\u2014\u2014\u7136\u540e\u628a\u8fd9\u4e9b\u8bcd\u7684\u610f\u601d\u6309\u7167\u5206\u6570\u52a0\u6743\u6df7\u5408\uff0c\u62fc\u6210”it”\u6b64\u523b\u5728\u673a\u5668\u8111\u6d77\u91cc\u7684\u771f\u5b9e\u542b\u4e49[7][14]\u3002<\/p>\n\n\n\n

\u52a0\u6743\u4ec0\u4e48\u610f\u601d\uff1f<\/p>\u52a0\u6743 = \u7ed9\u6bcf\u4e2a\u4e1c\u897f\u5206\u914d\u4e00\u4e2a”\u91cd\u8981\u7a0b\u5ea6\u7cfb\u6570”\uff0c<\/strong> \u6309”\u91cd\u8981\u7a0b\u5ea6”\u6765\u5206\u914d\u6bd4\u4f8b\uff0c\u91cd\u8981\u7684\u5360\u6bd4\u5927\uff0c\u4e0d\u91cd\u8981\u7684\u5360\u6bd4\u5c0f\uff0c\u6700\u540e\u6df7\u5408\u5728\u4e00\u8d77<\/strong><\/cite><\/blockquote><\/figure>\n\n\n\n

B\u7ad9\u4e0a\u6709\u4e2a\u66f4\u63a5\u5730\u6c14\u7684\u4e2d\u6587\u4f8b\u5b50\uff1a”\u5b83\u574f\u4e86\uff0c\u670b\u53cb\u79d2\u56de\u6362\u65b0\u7684\u3002”\u4f60\u542c\u5230\u8fd9\u53e5\u8bdd\uff0c\u7acb\u523b\u77e5\u9053”\u5b83”\u8bf4\u7684\u662f\u670b\u53cb\u521a\u4e70\u7684\u67d0\u4ef6\u4e1c\u897f\uff0c\u800c\u4e0d\u662f\u670b\u53cb\u672c\u4eba\u574f\u6389\u4e86\u2014\u2014\u8fd9\u5c31\u662f\u4f60\u7684\u5927\u8111\u5728\u4e00\u77ac\u95f4\u5b8c\u6210\u4e86\u81ea\u6ce8\u610f\u529b\u8fd0\u7b97\uff1a\u628a”\u5b83”\u548c\u524d\u6587\u91cc\u51fa\u73b0\u8fc7\u7684\u7269\u54c1\u3001\u548c”\u6362\u65b0\u7684”\u8fd9\u4e2a\u52a8\u4f5c\u81ea\u52a8\u5bf9\u9f50[15]\u3002\u53e6\u4e00\u4f4dUP\u4e3b\u7528”\u5b83\u6307\u4ee3\u82f9\u679c\u8fd8\u662f\u6c34\u679c”\u7684\u4f8b\u5b50\u6f14\u793a\u4e86\u540c\u6837\u7684\u9053\u7406\uff1a\u5f53\u4e00\u53e5\u8bdd\u91cc\u51fa\u73b0\u591a\u4e2a\u5019\u9009\u540d\u8bcd\u65f6\uff0c\u81ea\u6ce8\u610f\u529b\u4f1a\u50cf\u805a\u5149\u706f\u4e00\u6837\uff0c\u81ea\u52a8\u628a\u5149\u675f\u6253\u5230\u6700\u8be5\u5173\u6ce8\u7684\u90a3\u4e2a\u8bcd\u8eab\u4e0a[2]\u3002\u6362\u53e5\u8bdd\u8bf4\uff0c\u81ea\u6ce8\u610f\u529b\u505a\u7684\u4e8b\uff0c\u672c\u8d28\u4e0a\u5c31\u662f\u4eba\u7c7b\u9605\u8bfb\u65f6“\u8054\u7cfb\u4e0a\u4e0b\u6587\u731c\u610f\u601d”<\/strong>\u8fd9\u4e2a\u672c\u80fd\u52a8\u4f5c\u7684\u6570\u5b66\u5316\u7248\u672c\u3002\u7406\u89e3\u4e86\u8fd9\u4e00\u70b9\uff0c\u4f60\u5c31\u5df2\u7ecf\u6293\u4f4f\u4e86Transformer\u6700\u6838\u5fc3\u7684\u7075\u9b42\u3002<\/p>\n\n\n\n

2.2 \u4ece”\u9010\u8bcd\u9ed8\u8bfb”\u5230”\u6574\u9875\u626b\u8bfb”\uff1a\u81ea\u6ce8\u610f\u529b\u662f\u600e\u4e48”\u540c\u65f6\u770b\u6240\u6709\u8bcd”\u7684<\/strong><\/strong><\/h3>\n\n\n\n

\u5149\u77e5\u9053”\u5b83\u4f1a\u770b\u6240\u6709\u8bcd”\u8fd8\u4e0d\u591f\uff0c\u6211\u4eec\u9700\u8981\u66f4\u76f4\u89c2\u5730\u611f\u53d7\u8fd9\u79cd”\u540c\u65f6\u770b”\u548c\u6211\u4eec\u719f\u6089\u7684\u9605\u8bfb\u65b9\u5f0f\u5dee\u522b\u5230\u5e95\u6709\u591a\u5927\u3002\u4e0a\u4e00\u7ae0\u628aRNN\u6bd4\u4f5c”\u4f20\u8bdd\u6e38\u620f”\uff0c\u662f\u56e0\u4e3a\u5b83\u5fc5\u987b\u4e00\u4e2a\u8bcd\u4e00\u4e2a\u8bcd\u6309\u987a\u5e8f\u5904\u7406\uff1a\u8bfb\u5b8c\u7b2c\u4e00\u4e2a\u8bcd\uff0c\u628a\u4fe1\u606f\u6253\u5305\u4f20\u7ed9\u7b2c\u4e8c\u4e2a\u8bcd\uff1b\u8bfb\u5b8c\u7b2c\u4e8c\u4e2a\u8bcd\uff0c\u518d\u628a\u524d\u4e24\u4e2a\u8bcd\u7684\u4fe1\u606f\u6253\u5305\u4f20\u7ed9\u7b2c\u4e09\u4e2a\u8bcd\u2026\u2026\u6574\u6761\u94fe\u6761\u91cc\uff0c\u4efb\u4f55\u4e24\u4e2a\u8bcd\u4e4b\u95f4\u90fd\u4e0d\u80fd\u76f4\u63a5\u5bf9\u8bdd<\/strong>\uff0c\u5fc5\u987b\u7ecf\u8fc7\u4e2d\u95f4\u6240\u6709\u8bcd\u7684”\u4f20\u8bdd”[3][4][9]\u3002\u8fd9\u5c31\u50cf\u4f60\u8bfb\u4e00\u884c\u5b57\uff0c\u5fc5\u987b\u7528\u624b\u628a\u5176\u4ed6\u5b57\u5168\u906e\u4f4f\uff0c\u53ea\u9732\u51fa\u5f53\u524d\u8fd9\u4e00\u4e2a\u5b57\uff0c\u9760\u8bb0\u5fc6\u56de\u60f3\u524d\u9762\u8bf4\u4e86\u4ec0\u4e48\u2014\u2014\u8bfb\u77ed\u53e5\u5b50\u8fd8\u884c\uff0c\u4e00\u5230\u957f\u6587\u7ae0\u5c31\u5bb9\u6613\u5fd8\u3002<\/p>\n\n\n\n

\u81ea\u6ce8\u610f\u529b\u5f7b\u5e95\u6253\u7834\u4e86\u8fd9\u4e2a\u9650\u5236\u3002\u5b83\u66f4\u50cf\u4f60\u628a\u6574\u5f20\u7eb8\u5b8c\u5168\u644a\u5f00<\/strong>\u6446\u5728\u684c\u4e0a\uff0c\u6bcf\u4e2a\u8bcd\u90fd\u548c\u5176\u4ed6\u6240\u6709\u8bcd\u9762\u5bf9\u9762\u5750\u7740\uff0c\u5f00\u4e00\u573a”\u5168\u5458\u7535\u8bdd\u4f1a\u8bae”[3][4]\u3002\u5f53\u4f60\u9700\u8981\u7406\u89e3\u7b2c5\u4e2a\u8bcd\u7684\u65f6\u5019\uff0c\u4f60\u4e0d\u9700\u8981\u7b49\u7b2c1\u5230\u7b2c4\u4e2a\u8bcd\u628a\u8bdd\u63a5\u529b\u4f20\u8fc7\u6765\uff0c\u800c\u662f\u53ef\u4ee5\u76f4\u63a5<\/strong>\u62ff\u8d77\u7535\u8bdd\uff0c\u62e8\u7ed9\u7b2c1\u4e2a\u8bcd\u3001\u7b2c2\u4e2a\u8bcd\u2026\u2026\u76f4\u5230\u6700\u540e\u4e00\u4e2a\u8bcd\uff0c\u6328\u4e2a\u95ee\u4e00\u53e5\uff1a”\u4f60\u548c\u6211\u73b0\u5728\u5904\u7406\u7684\u8fd9\u4ef6\u4e8b\u5173\u7cfb\u5927\u5417\uff1f”\u5173\u7cfb\u5927\u7684\uff0c\u5c31\u591a\u542c\u4e24\u53e5\uff1b\u5173\u7cfb\u5c0f\u7684\uff0c\u5c31\u5c11\u542c\u751a\u81f3\u4e0d\u542c\u3002\u6574\u4e2a\u8fc7\u7a0b\u5bf9\u53e5\u5b50\u91cc\u6240\u6709\u4f4d\u7f6e\u7684\u8bcd\u540c\u65f6\u53d1\u751f\uff0c\u6240\u4ee5\u624d\u6709\u4e86Transformer\u6700\u8457\u540d\u7684\u4f18\u52bf\u2014\u2014\u9ad8\u5ea6\u5e76\u884c<\/strong>\uff1aGPU\u53ef\u4ee5\u4e00\u53e3\u6c14\u7b97\u51fa\u6574\u53e5\u8bdd\u91cc\u4efb\u610f\u4e24\u4e2a\u8bcd\u4e4b\u95f4\u7684\u5173\u7cfb\uff0c\u800c\u4e0d\u5fc5\u50cfRNN\u90a3\u6837\u82e6\u54c8\u54c8\u5730\u7b49\u4e0a\u4e00\u4e2a\u8bcd\u7b97\u5b8c[9][13]\u3002<\/p>\n\n\n\n

\u674e\u5b8f\u6bc5\u8001\u5e08\u5728\u4ed6\u90a3\u95e8\u88ab\u516c\u8ba4\u4e3a\u4e2d\u6587\u4e16\u754c\u6700\u901a\u4fd7\u7684Transformer\u8bfe\u4e0a\uff0c\u7528\u4e00\u4e2a\u66f4\u5f62\u8c61\u7684\u6bd4\u55bb\u89e3\u91ca\u8fd9\u4ef6\u4e8b\uff1a\u81ea\u6ce8\u610f\u529b\u7684\u8f93\u5165\u662f”\u4e00\u6392\u5411\u91cf”\uff0c\u8f93\u51fa\u4e5f\u662f”\u4e00\u6392\u5411\u91cf”\uff0c\u4f46\u8f93\u51fa\u4f4d\u7f6e\u4e0a\u7684\u6bcf\u4e00\u4e2a\u5411\u91cf\uff0c\u90fd\u5df2\u7ecf”\u878d\u5408\u4e86\u6574\u6392\u6240\u6709\u8f93\u5165\u5411\u91cf\u7684\u4fe1\u606f”[5]\u3002\u6253\u4e2a\u751f\u6d3b\u5316\u7684\u6bd4\u65b9\uff0c\u8fd9\u5c31\u50cf\u4e00\u4e2a\u73ed\u7ea7\u91cc\u7684\u540c\u5b66\u4e00\u5f00\u59cb\u5404\u60f3\u5404\u7684\uff0c\u8001\u5e08\u558a\u4e86\u4e00\u58f0”\u5c0f\u7ec4\u8ba8\u8bba”\uff0c\u6240\u6709\u4eba\u7acb\u523b\u540c\u65f6\u548c\u6240\u6709\u4eba\u4ea4\u6362\u610f\u89c1\uff1b\u8ba8\u8bba\u7ed3\u675f\u540e\uff0c\u6bcf\u4e2a\u540c\u5b66\u8111\u5b50\u91cc\u7684\u60f3\u6cd5\uff0c\u90fd\u5df2\u7ecf\u5438\u6536\u4e86\u5168\u73ed\u540c\u5b66\u7684\u89c2\u70b9\u2014\u2014\u53ea\u662f\u5438\u6536\u591a\u5c11\uff0c\u53d6\u51b3\u4e8e\u4ed6\u548c\u5bf9\u65b9\u804a\u5f97\u6709\u591a\u6295\u673a\u3002Gal Lahat\u5236\u4f5c\u7684\u53ef\u89c6\u5316\u52a8\u753b\u76f4\u63a5\u628a\u8fd9\u4e2a\u8fc7\u7a0b\u753b\u5728\u4e86\u5c4f\u5e55\u4e0a\uff1a\u4f60\u80fd\u770b\u5230\u4e00\u4e2a\u4e2a\u5f69\u8272\u7684Q\u3001K\u3001V\u7bad\u5934\u5728\u8bcd\u4e0e\u8bcd\u4e4b\u95f4\u98de\u6765\u98de\u53bb\uff0c\u4eae\u5ea6\u8d8a\u9ad8\u4ee3\u8868\u4e24\u4e2a\u8bcd\u7684\u5173\u7cfb\u8d8a\u7d27\u5bc6[6]\u3002\u8fd9\u79cd”\u4e24\u4e24\u4e4b\u95f4\u90fd\u80fd\u76f4\u63a5\u8fde\u7ebf”\u7684\u7ed3\u6784\uff0c\u6b63\u662f\u81ea\u6ce8\u610f\u529b\u80fd\u591f\u8f7b\u677e\u6355\u6349\u957f\u8ddd\u79bb\u4f9d\u8d56\u7684\u6839\u672c\u539f\u56e0\u2014\u2014\u4e00\u53e5\u8bdd\u5f00\u5934\u7684\u8bcd\u548c\u7ed3\u5c3e\u7684\u8bcd\uff0c\u4e5f\u80fd\u4e00\u6761\u76f4\u7ebf\u8fde\u8d77\u6765\uff0c\u4fe1\u606f\u4e0d\u518d\u9700\u8981\u7ecf\u8fc7\u957f\u9014\u4f20\u8bdd\uff0c\u81ea\u7136\u5c31\u4e0d\u4f1a\u8870\u51cf[3][13]\u3002<\/p>\n\n\n\n

2.3 \u56fe\u4e66\u9986\u67e5\u4e66\u4e0e\u505a\u9605\u8bfb\u7406\u89e3\uff1a\u4e24\u4e2a\u6bd4\u55bb\u5e2e\u4f60\u5efa\u7acb\u7ec8\u8eab\u76f4\u89c9<\/strong><\/strong><\/h3>\n\n\n\n

\u7406\u89e3\u4e86\u81ea\u6ce8\u610f\u529b”\u5728\u5e72\u4ec0\u4e48”\uff0c\u6700\u540e\u4e00\u6b65\u662f\u628a\u5b83”\u600e\u4e48\u5e72”\u7684\u76f4\u89c9\u6c38\u4e45\u523b\u8fdb\u8111\u5b50\u91cc\u3002\u8fd9\u91cc\u7ed9\u4f60\u4e24\u4e2a\u88ab\u5404\u8def\u6559\u7a0b\u53cd\u590d\u9a8c\u8bc1\u8fc7\u3001\u6700\u9002\u5408\u5c0f\u767d\u7684\u751f\u6d3b\u6bd4\u55bb\uff0c\u4f60\u53ea\u8981\u8bb0\u4f4f\u5176\u4e2d\u4efb\u4f55\u4e00\u4e2a\uff0c\u4ee5\u540e\u518d\u542c\u5230”\u81ea\u6ce8\u610f\u529b”\u56db\u4e2a\u5b57\u90fd\u4e0d\u4f1a\u53d1\u6035\u3002<\/p>\n\n\n\n

\u7b2c\u4e00\u4e2a\u6bd4\u55bb\uff1a\u56fe\u4e66\u9986\u67e5\u4e66\uff08\u5bf9\u5e94QKV\u68c0\u7d22\u7684\u76f4\u89c9\uff09\u3002<\/strong> \u60f3\u8c61\u4f60\u8d70\u8fdb\u4e00\u5bb6\u8d85\u5927\u56fe\u4e66\u9986\u60f3\u627e\u4e00\u672c\u4e66\u3002\u4f60\u5fc3\u91cc\u6709\u4e00\u4e2a\u660e\u786e\u7684\u95ee\u9898\uff1a”\u6211\u60f3\u4e86\u89e3\u732b\u54aa\u7684\u4e60\u6027\u3002”\u2014\u2014\u8fd9\u5c31\u662f\u4f60\u7684Query\uff08\u67e5\u8be2\uff09<\/strong>\u3002\u4e66\u67b6\u4e0a\u6bcf\u672c\u4e66\u7684\u4e66\u810a\u4e0a\u90fd\u8d34\u7740\u4e00\u4e2a\u5c0f\u6807\u7b7e\uff0c\u5199\u7740\u4e66\u540d\u3001\u5173\u952e\u8bcd\u2014\u2014\u8fd9\u5c31\u662fKey\uff08\u952e\/\u7d22\u5f15\uff09<\/strong>\u3002\u4f60\u62ff\u7740\u81ea\u5df1\u7684\u95ee\u9898\uff0c\u53bb\u548c\u6bcf\u4e00\u672c\u4e66\u7684\u6807\u7b7e\u505a\u6bd4\u5bf9\uff0c\u8d8a\u8d34\u5408\u4f60\u95ee\u9898\u7684\u4e66\uff0c\u4f60\u8d8a\u53ef\u80fd\u628a\u5b83\u62bd\u51fa\u6765\uff1b\u800c\u4e66\u91cc\u9762\u771f\u6b63\u5199\u7740\u7684\u5185\u5bb9\u2014\u2014\u8bb2\u732b\u54aa\u600e\u4e48\u7761\u89c9\u3001\u600e\u4e48\u6355\u730e\u2014\u2014\u5c31\u662fValue\uff08\u503c\/\u5b9e\u9645\u4fe1\u606f\uff09<\/strong>[2][11]\u3002\u6700\u540e\uff0c\u4f60\u5e76\u4e0d\u662f\u53ea\u62ff\u8d70\u6700\u76f8\u5173\u7684\u90a3\u4e00\u672c\uff0c\u800c\u662f\u628a\u6240\u6709\u4e66\u7684\u5185\u5bb9\u6309\u7167”\u548c\u4f60\u95ee\u9898\u7684\u8d34\u5408\u7a0b\u5ea6”\u52a0\u6743\u6c47\u603b\uff0c\u5f97\u5230\u4e00\u4e2a\u6700\u80fd\u56de\u7b54\u4f60\u95ee\u9898\u7684\u7b54\u6848\u3002\u81ea\u6ce8\u610f\u529b\u505a\u7684\u4e8b\uff0c\u5c31\u662f\u8ba9\u53e5\u5b50\u91cc\u7684\u6bcf\u4e00\u4e2a\u8bcd\u90fd\u5f53\u4e00\u6b21”\u67e5\u4e66\u4eba”<\/strong>\uff1a\u5b83\u62ff\u7740\u81ea\u5df1\u7684Query\uff0c\u53bb\u548c\u6240\u6709\u8bcd\uff08\u5305\u62ec\u81ea\u5df1\uff09\u7684Key\u6bd4\u5bf9\u76f8\u4f3c\u5ea6\uff0c\u518d\u6309\u76f8\u4f3c\u5ea6\u52a0\u6743\u6c47\u603b\u6240\u6709\u8bcd\u7684Value\uff0c\u5f62\u6210\u8fd9\u4e2a\u8bcd\u6b64\u523b\u5728\u4e0a\u4e0b\u6587\u4e2d\u7684\u65b0\u542b\u4e49\u3002\u4e00\u4f4dUP\u4e3b\u7528”\u67e5\u5c0f\u7ea2\u8eab\u9ad8”\u7684\u5c0f\u6e38\u620f\u628a\u8fd9\u4e2a\u8fc7\u7a0b\u6f14\u4e86\u4e00\u904d\uff1a\u5f53\u4f60\u95ee”\u5c0f\u7ea2\u591a\u9ad8”\u65f6\uff0c\u7cfb\u7edf\u4f1a\u81ea\u52a8\u53bb\u6240\u6709\u4eba\u7269\u6863\u6848\u91cc\u627e\u6807\u7b7e\u5e26”\u5c0f\u7ea2””\u8eab\u9ad8”\u7684\u90a3\u6761\u8bb0\u5f55\uff0c\u518d\u628a\u8eab\u9ad8\u6570\u5b57\u8bfb\u7ed9\u4f60\u542c\u2014\u2014\u8fd9\u5c31\u662f\u6700\u6734\u7d20\u7248\u672c\u7684\u6ce8\u610f\u529b[2]\u3002<\/p>\n\n\n\n

\u7b2c\u4e8c\u4e2a\u6bd4\u55bb\uff1a\u505a\u9605\u8bfb\u7406\u89e3\uff08\u5bf9\u5e94\u6574\u4e2a\u81ea\u6ce8\u610f\u529b\u5c42\u7684\u5de5\u4f5c\u6d41\uff09\u3002<\/strong> \u60f3\u8c61\u4f60\u6b63\u5728\u505a\u4e00\u7bc7\u82f1\u8bed\u9605\u8bfb\u7406\u89e3\uff0c\u8bd5\u5377\u95ee\u4f60\uff1a”\u7b2c\u4e8c\u6bb5\u7b2c\u4e09\u884c\u7684’it’\u6307\u7684\u662f\u4ec0\u4e48\uff1f”\u4f60\u4f1a\u600e\u4e48\u505a\uff1f\u4f60\u4f1a\u5148\u7528\u7b14\u628a”it”\u5708\u51fa\u6765\u2014\u2014\u8fd9\u5c31\u662f\u5f53\u524d\u8981\u5904\u7406\u7684\u8bcd\uff1b\u7136\u540e\u4f60\u7684\u773c\u775b\u4f1a\u5feb\u901f\u626b\u8fc7\u5168\u6587\uff0c\u5c24\u5176\u662f”it”\u9644\u8fd1\u4ee5\u53ca\u903b\u8f91\u4e0a\u53ef\u80fd\u76f8\u5173\u7684\u53e5\u5b50\u2014\u2014\u8fd9\u5c31\u662f\u5728\u7b97\u76f8\u4f3c\u5ea6\uff1b\u4f60\u4f1a\u5728\u5fc3\u91cc\u7ed9\u6bcf\u4e2a\u5019\u9009\u540d\u8bcd\u6253\u4e2a\u5206\uff0c\u770b\u8c01\u6700\u80fd\u8ba9”it was too tired”\u8fd9\u53e5\u8bdd\u8bb2\u5f97\u901a\u2014\u2014\u8fd9\u5c31\u662f\u52a0\u6743\uff1b\u6700\u540e\u4f60\u628a\u6700\u76f8\u5173\u7684\u90a3\u4e2a\u540d\u8bcd\u5708\u51fa\u6765\u4f5c\u4e3a\u7b54\u6848\u2014\u2014\u8fd9\u5c31\u662f\u6c47\u603b\u540e\u7684\u8f93\u51fa[11]\u3002\u6ce8\u610f\u8fd9\u4e2a\u8fc7\u7a0b\u6709\u4e09\u4e2a\u5173\u952e\u7279\u70b9\uff1a\u7b2c\u4e00\uff0c\u4f60\u770b\u7684\u662f\u6574\u7bc7\u6587\u7ae0<\/strong>\u800c\u4e0d\u662f”it”\u524d\u9762\u4e00\u4e2a\u8bcd\uff1b\u7b2c\u4e8c\uff0c\u4f60\u7ed9\u4e0d\u540c\u53e5\u5b50\u5206\u914d\u7684\u6ce8\u610f\u529b\u6743\u91cd\u662f\u4e0d\u4e00\u6837\u7684<\/strong>\uff0c\u76f8\u5173\u7684\u591a\u770b\u51e0\u773c\uff0c\u4e0d\u76f8\u5173\u7684\u626b\u8fc7\u5c31\u5fd8\uff1b\u7b2c\u4e09\uff0c\u8fd9\u4e2a\u52a8\u4f5c\u4f60\u53ef\u4ee5\u5bf9\u6587\u7ae0\u91cc\u6bcf\u4e00\u4e2a\u8bcd<\/strong>\u90fd\u91cd\u590d\u505a\u4e00\u904d\u2014\u2014\u5f53\u4f60\u5206\u6790\u5b8c”it”\uff0c\u4f60\u8fd8\u53ef\u4ee5\u53bb\u5206\u6790”animal”\u3001”street”\u3001”tired”\uff0c\u6bcf\u4e2a\u8bcd\u90fd\u4f1a\u5f97\u5230\u5c5e\u4e8e\u5b83\u81ea\u5df1\u7684\u90a3\u4efd”\u5168\u73ed\u8ba8\u8bba\u7ed3\u679c”\u3002\u8fd9\u6b63\u662f\u81ea\u6ce8\u610f\u529b\u5c42\u91cc\u53d1\u751f\u7684\u4e8b\u60c5\uff1a\u8f93\u5165\u4e00\u6574\u53e5\u8bdd\uff0c\u8f93\u51fa\u4e00\u6574\u53e5\u8bdd\uff0c\u8f93\u51fa\u7684\u6bcf\u4e2a\u8bcd\u90fd\u5df2\u7ecf”\u8bfb\u8fc7\u5168\u6587”\uff0c\u5e76\u4e14\u628a\u6ce8\u610f\u529b\u7cbe\u51c6\u5730\u6295\u7ed9\u4e86\u6700\u8be5\u5173\u6ce8\u7684\u540c\u4f34[7][8][10]\u3002<\/p>\n\n\n\n

\u5230\u6b64\u4f60\u5e94\u8be5\u660e\u767d\uff0c\u81ea\u6ce8\u610f\u529b\u5e76\u4e0d\u662f\u4ec0\u4e48\u9ad8\u6df1\u83ab\u6d4b\u7684\u9b54\u6cd5\uff0c\u5b83\u53ea\u4e0d\u8fc7\u662f\u628a\u6211\u4eec\u4eba\u7c7b\u7406\u89e3\u8bed\u8a00\u65f6\u90a3\u4e2a”\u8054\u7cfb\u4e0a\u4e0b\u6587\u3001\u6293\u91cd\u70b9\u3001\u878d\u4f1a\u8d2f\u901a”\u7684\u672c\u80fd\uff0c\u7528\u5411\u91cf\u548c\u77e9\u9635\u8fd0\u7b97\u4e25\u8c28\u5730\u590d\u523b\u4e86\u51fa\u6765\u3002\u4e0b\u4e00\u7ae0\u6211\u4eec\u5c31\u987a\u7740\u56fe\u4e66\u9986\u67e5\u4e66\u8fd9\u4e2a\u6bd4\u55bb\uff0c\u628aQuery\u3001Key\u3001Value\u4e09\u5144\u5f1f\u7684\u6765\u9f99\u53bb\u8109\uff0c\u4ee5\u53ca\u5b83\u4eec\u8ba1\u7b97\u76f8\u5173\u5ea6\u7684\u56db\u6b65\u6d41\u7a0b\uff0c\u8fde\u5c0f\u5b66\u751f\u90fd\u80fd\u61c2\u5730\u4e00\u6b65\u6b65\u62c6\u5f00\u3002<\/p>\n\n\n\n

\u7b2c3\u7ae0 QKV\u4e09\u5144\u5f1f\u4e0e\u8ba1\u7b97\u56db\u6b65\u2014\u2014\u8fde\u5c0f\u5b66\u751f\u90fd\u80fd\u61c2\u7684\u6d41\u7a0b<\/strong><\/strong><\/h2>\n\n\n\n

\u7406\u89e3\u4e86\u81ea\u6ce8\u610f\u529b”\u4e00\u4e2a\u8bcd\u540c\u65f6\u770b\u6240\u6709\u8bcd”\u7684\u6838\u5fc3\u601d\u60f3\u540e\uff0c\u5f88\u591a\u4eba\u4f1a\u7acb\u523b\u8ffd\u95ee\uff1a\u5b83\u5177\u4f53\u662f\u600e\u4e48”\u770b”\u7684\uff1f\u6ce8\u610f\u529b\u5206\u6570\u662f\u600e\u4e48\u7b97\u51fa\u6765\u7684\uff1f\u7b54\u6848\u5c31\u662fQ\u3001K\u3001V\u4e09\u4e2a\u89d2\u8272\u52a0\u4e0a\u4e00\u5957\u53ea\u6709\u56db\u6b65\u7684\u6d41\u6c34\u7ebf\u3002\u672c\u7ae0\u5c31\u7528\u67e5\u4e66\u3001\u8003\u8bd5\u3001\u505a\u9605\u8bfb\u7406\u89e3\u8fd9\u4e9b\u4f60\u6bcf\u5929\u90fd\u5728\u505a\u7684\u4e8b\uff0c\u628a\u8fd9\u5957\u6d41\u7a0b\u5f7b\u5e95\u8bb2\u900f\u3002<\/p>\n\n\n\n

3.1 QKV\u4e09\u5144\u5f1f\uff1a\u95ee\u9898\u3001\u6807\u7b7e\u4e0e\u7b54\u6848<\/strong><\/strong><\/h3>\n\n\n\n

Q\u3001K\u3001V\u5206\u522b\u4ee3\u8868Query\u3001Key\u3001Value\uff0c\u7ffb\u8bd1\u6210\u4e2d\u6587\u5c31\u662f”\u67e5\u8be2””\u952e””\u503c”\u3002\u8fd9\u4e09\u4e2a\u8bcd\u542c\u8d77\u6765\u5f88\u552c\u4eba\uff0c\u5176\u5b9e\u5b83\u4eec\u7684\u5173\u7cfb\u5c31\u662f\u4f60\u6bcf\u5929\u505a\u9605\u8bfb\u7406\u89e3\u65f6\u7684\u4e09\u4e2a\u4e1c\u897f\u3002<\/p>\n\n\n\n

\u505a\u9605\u8bfb\u7406\u89e3\u7684\u6bd4\u55bb<\/strong>\u3002\u5047\u8bbe\u4f60\u6b63\u5728\u8bfb\u4e00\u7bc7\u82f1\u8bed\u77ed\u6587\uff0c\u624b\u91cc\u62ff\u7740\u4e00\u9053\u9898\u76ee\u2014\u2014”\u6587\u4e2d\u7684\u5b83\u6307\u7684\u662f\u4ec0\u4e48\uff1f”\u3002\u4f60\u8111\u5b50\u91cc\u5e26\u7740\u7684\u8fd9\u4e2a\u95ee\u9898\uff0c\u5c31\u662fQ\uff08Query\uff09\u3002\u4f60\u626b\u8bfb\u6587\u7ae0\u65f6\uff0c\u6bcf\u53e5\u8bdd\u65c1\u8fb9\u5176\u5b9e\u90fd\u6709\u4e00\u4e2a”\u5c0f\u6807\u7b7e”\uff0c\u6709\u7684\u5199\u7740”\u8bb2\u5c0f\u732b”\u3001\u6709\u7684\u5199\u7740”\u8bb2\u8857\u9053”\u3001\u6709\u7684\u5199\u7740”\u8bb2\u7d2f\u4e86”\uff0c\u8fd9\u4e9b\u6807\u7b7e\u5c31\u662fK\uff08Key\uff09\u3002\u4f60\u6700\u7ec8\u5728\u67d0\u53e5\u8bdd\u91cc\u627e\u5230\u7684\u5177\u4f53\u4fe1\u606f\u3001\u771f\u5b9e\u7b54\u6848\uff0c\u5c31\u662fV\uff08Value\uff09[11]\u3002\u4f60\u62ff\u95ee\u9898\u53bb\u6bd4\u5bf9\u6bcf\u4e00\u4e2a\u6807\u7b7e\uff0c\u54ea\u4e2a\u6807\u7b7e\u548c\u95ee\u9898\u6700\u5bf9\u5f97\u4e0a\uff0c\u4f60\u5c31\u628a\u90a3\u53e5\u8bdd\u91cc\u7684\u7b54\u6848\u62ff\u56de\u6765\u7528\u2014\u2014\u8fd9\u5c31\u662f\u6ce8\u610f\u529b\u5728\u505a\u7684\u4e8b\u3002<\/p>\n\n\n\n

\u56fe\u4e66\u9986\u67e5\u4e66\u7684\u6bd4\u55bb<\/strong>\u3002\u8d70\u8fdb\u56fe\u4e66\u9986\u627e\u4e66\uff0c\u4f60\u9012\u7ed9\u7ba1\u7406\u5458\u4e00\u5f20\u5b57\u6761\uff1a”\u6211\u8981\u627e\u8bb2\u6050\u9f99\u7684\u7ed8\u672c”\u3002\u8fd9\u5f20\u5b57\u6761\u5c31\u662fQ\u3002\u4e66\u67b6\u4e0a\u6bcf\u672c\u4e66\u7684\u4e66\u810a\u4e0a\u8d34\u7740\u7684\u5206\u7c7b\u6807\u7b7e\u2014\u2014”\u5929\u6587””\u6050\u9f99””\u7ae5\u8bdd”\u2014\u2014\u5c31\u662fK\u3002\u800c\u4e66\u91cc\u9762\u771f\u6b63\u5199\u7684\u5185\u5bb9\u3001\u56fe\u753b\u548c\u77e5\u8bc6\uff0c\u5c31\u662fV\u3002\u7ba1\u7406\u5458\u62ff\u7740\u4f60\u7684\u5b57\u6761\u53bb\u5bf9\u6bcf\u672c\u4e66\u7684\u6807\u7b7e\uff0c\u628a\u6700\u5339\u914d\u7684\u90a3\u51e0\u672c\u62bd\u51fa\u6765\u9012\u7ed9\u4f60\uff0c\u4f60\u7ffb\u5f00\u4e66\u5c31\u8bfb\u5230\u4e86\u771f\u6b63\u7684\u5185\u5bb9[2][7]\u3002\u6574\u4e2a\u8fc7\u7a0b\u4e2d\uff0cQ\u662f”\u6211\u8981\u627e\u4ec0\u4e48”\uff0cK\u662f”\u6211\u8fd9\u91cc\u6709\u4ec0\u4e48\u53ef\u4f9b\u67e5\u627e”\uff0cV\u662f”\u6211\u771f\u6b63\u80fd\u7ed9\u4f60\u7684\u4e1c\u897f”\u3002<\/p>\n\n\n\n

\u4e00\u4e2a\u5173\u952e\u6f84\u6e05<\/strong>\u3002\u5f88\u591a\u521d\u5b66\u8005\u4ee5\u4e3aQ\u3001K\u3001V\u662f\u8f93\u5165\u8bcd\u81ea\u5e26\u7684\u4e09\u79cd\u5c5e\u6027\u2014\u2014\u6bd4\u5982”\u82f9\u679c”\u8fd9\u4e2a\u8bcd\u5929\u7136\u5c31\u5e26\u7740\u4e00\u4e2aQ\u3001\u4e00\u4e2aK\u3001\u4e00\u4e2aV\u3002\u8fd9\u5176\u5b9e\u662f\u4e2a\u5e38\u89c1\u8bef\u89e3\u3002\u771f\u5b9e\u60c5\u51b5\u662f\uff1a\u53e5\u5b50\u91cc\u6bcf\u4e2a\u8bcd\u8fdb\u6765\u65f6\uff0c\u53ea\u662f\u4e00\u4e2a\u8868\u793a\u5b83\u8eab\u4efd\u7684\u5411\u91cf\uff08\u4e5f\u5c31\u662f\u4e0a\u4e00\u7ae0\u8bb2\u8fc7\u7684\u8bcd\u5d4c\u5165\uff09\uff0cQ\u3001K\u3001V\u662f\u8fd9\u4e2a\u5411\u91cf\u5206\u522b\u4e58\u4ee5\u4e09\u5957\u4e0d\u540c\u7684”\u6295\u5f71\u77e9\u9635”\u4e4b\u540e\u53d8\u51fa\u6765\u7684\u4e09\u4e2a\u5206\u8eab[10]\u3002\u53ef\u4ee5\u628a\u8fd9\u4e09\u5957\u77e9\u9635\u60f3\u8c61\u6210\u4e09\u526f\u4e0d\u540c\u7684\u773c\u955c\uff1a\u6234\u4e0a”\u63d0\u95ee\u773c\u955c”\uff0c\u8fd9\u4e2a\u8bcd\u5c31\u53d8\u6210\u4e86Q\uff0c\u8d1f\u8d23\u53bb\u95ee\u522b\u4eba\uff1b\u6234\u4e0a”\u88ab\u95ee\u773c\u955c”\uff0c\u5b83\u5c31\u53d8\u6210\u4e86K\uff0c\u7b49\u7740\u522b\u4eba\u6765\u5339\u914d\uff1b\u6234\u4e0a”\u5185\u5bb9\u773c\u955c”\uff0c\u5b83\u5c31\u53d8\u6210\u4e86V\uff0c\u51c6\u5907\u628a\u81ea\u5df1\u7684\u771f\u5b9e\u4fe1\u606f\u8d21\u732e\u51fa\u53bb\u3002\u8fd9\u4e09\u526f\u773c\u955c\u7684\u53c2\u6570\u4e0d\u662f\u4eba\u5de5\u5199\u6b7b\u7684\uff0c\u662f\u6a21\u578b\u5728\u8bad\u7ec3\u8fc7\u7a0b\u4e2d\u81ea\u5df1\u5b66\u51fa\u6765\u7684\u2014\u2014\u4e5f\u5c31\u662f\u8bf4\uff0c\u6a21\u578b\u4f1a\u6162\u6162\u6478\u7d22\u51fa”\u4ec0\u4e48\u6837\u7684\u63d0\u95ee\u65b9\u5f0f\u6700\u6709\u6548””\u4ec0\u4e48\u6837\u7684\u6807\u7b7e\u6700\u597d\u88ab\u627e\u5230”[5][8]\u3002<\/p>\n\n\n\n

\u6295\u5f71\u77e9\u9635\u5982\u4f55\u5de5\u4f5c\uff1f<\/p>\u770b\u53e6\u4e00\u7bc7\u6587\u7ae0<\/a><\/cite><\/blockquote><\/figure>\n\n\n\n

\u4e00\u53e5\u8bdd\u603b\u7ed3\u4e09\u5144\u5f1f<\/strong>\uff1aQ\u8d1f\u8d23”\u6211\u8981\u67e5\u4ec0\u4e48”\uff0cK\u8d1f\u8d23”\u6211\u7684\u6807\u7b7e\u662f\u4ec0\u4e48”\uff0cV\u8d1f\u8d23”\u6211\u7684\u5185\u5bb9\u662f\u4ec0\u4e48”\u3002\u6bcf\u4e2a\u8bcd\u540c\u65f6\u626e\u6f14\u4e09\u4e2a\u89d2\u8272\u2014\u2014\u5b83\u65e2\u662f\u63d0\u95ee\u8005\uff08\u62ff\u81ea\u5df1\u7684Q\u53bb\u8ddf\u6240\u6709\u4eba\u7684K\u6bd4\u5bf9\uff09\uff0c\u4e5f\u662f\u88ab\u95ee\u8005\uff08\u62ff\u51fa\u81ea\u5df1\u7684K\u7b49\u7740\u522b\u4eba\u6765\u5339\u914d\uff09\uff0c\u8fd8\u662f\u4fe1\u606f\u63d0\u4f9b\u8005\uff08\u628a\u81ea\u5df1\u7684V\u4ea4\u7ed9\u5339\u914d\u5ea6\u9ad8\u7684\u8bcd\u4f7f\u7528\uff09\u3002\u7406\u89e3\u4e86\u8fd9\u4e00\u70b9\uff0c\u4e0b\u4e00\u8282\u7684\u56db\u6b65\u8ba1\u7b97\u5c31\u6c34\u5230\u6e20\u6210\u4e86\u3002<\/p>\n\n\n\n

3.2 \u7b2c\u4e00\u6b65\u5230\u7b2c\u4e09\u6b65\uff1a\u70b9\u79ef\u3001\u7f29\u653e\u3001Softmax\u2014\u2014\u4ece\u6bd4\u5bf9\u5230\u6253\u5206<\/strong><\/strong><\/h3>\n\n\n\n

\u6709\u4e86Q\u3001K\u3001V\u4e4b\u540e\uff0c\u81ea\u6ce8\u610f\u529b\u5230\u5e95\u600e\u4e48\u8ba1\u7b97\uff1f\u6574\u4e2a\u6d41\u7a0b\u53ef\u4ee5\u62c6\u6210\u56db\u6b65\uff0c\u524d\u4e09\u6b65\u662f\u5728\u7b97”\u6211\u8be5\u5173\u6ce8\u8c01\u3001\u5173\u6ce8\u591a\u5c11”\uff0c\u7b2c\u56db\u6b65\u624d\u662f\u628a\u4fe1\u606f\u771f\u6b63\u53d6\u56de\u6765\u3002<\/p>\n\n\n\n

\u7b2c\u4e00\u6b65\uff1a\u70b9\u79ef\u7b97\u76f8\u4f3c\u5ea6\u2014\u2014”\u95ee\u9898\u5bf9\u6807\u7b7e”\u3002<\/strong> \u5f53\u8bcdA\u60f3\u77e5\u9053\u81ea\u5df1\u8be5\u5173\u6ce8\u53e5\u5b50\u91cc\u54ea\u4e9b\u8bcd\u65f6\uff0c\u5b83\u5c31\u62ff\u51fa\u81ea\u5df1\u7684Q\uff0c\u53bb\u548c\u6bcf\u4e2a\u8bcd\uff08\u5305\u62ec\u5b83\u81ea\u5df1\uff09\u7684K\u505a\u4e00\u6b21”\u70b9\u79ef”\u8fd0\u7b97\u3002\u70b9\u79ef\u662f\u4ec0\u4e48\uff1f\u4f60\u53ef\u4ee5\u628a\u5b83\u7406\u89e3\u6210\u4e00\u79cd”\u6253\u5206”\uff1a\u4e24\u4e2a\u5411\u91cf\u65b9\u5411\u8d8a\u4e00\u81f4\uff0c\u70b9\u79ef\u8d8a\u5927\uff0c\u8bf4\u660e”\u8fd9\u4e2a\u6807\u7b7e\u548c\u6211\u7684\u95ee\u9898\u8d8a\u5bf9\u5473”\uff1b\u65b9\u5411\u8d8a\u76f8\u53cd\uff0c\u70b9\u79ef\u8d8a\u5c0f\u751a\u81f3\u4e3a\u8d1f\uff0c\u8bf4\u660e”\u8ddf\u6211\u6ca1\u5173\u7cfb”[8][12]\u3002\u56de\u5230\u9605\u8bfb\u7406\u89e3\u7684\u6bd4\u55bb\uff1a\u4f60\u5e26\u7740\u95ee\u9898”\u5b83\u6307\u4ec0\u4e48”\u53bb\u626b\u6bcf\u4e2a\u53e5\u5b50\u7684\u6807\u7b7e\uff0c\u6807\u7b7e”\u8bb2\u4e00\u53ea\u52a8\u7269”\u548c\u4f60\u7684\u95ee\u9898\u9ad8\u5ea6\u5951\u5408\uff0c\u5f97\u5206\u5c31\u9ad8\uff1b\u6807\u7b7e”\u8bb2\u8857\u9053\u5f88\u5bbd”\u548c\u95ee\u9898\u5173\u7cfb\u4e0d\u5927\uff0c\u5f97\u5206\u5c31\u4f4e\u3002\u505a\u5b8c\u8fd9\u4e00\u6b65\uff0c\u8bcdA\u5c31\u5f97\u5230\u4e86\u4e00\u6392\u5206\u6570\uff0c\u5206\u6570\u9ad8\u4f4e\u4ee3\u8868\u5b83\u5bf9\u6bcf\u4e2a\u8bcd\u7684”\u611f\u5174\u8da3\u7a0b\u5ea6”\u3002<\/p>\n\n\n\n

\u70b9\u79ef\u662f\u4ec0\u4e48\uff1f\u548c\u4f59\u5f26\u76f8\u4f3c\u5ea6\u533a\u522b\uff1f<\/strong><\/p>\u4e00\u3001\u70b9\u79ef\uff08Dot Product\uff09\uff1a\u4e24\u4e2a\u5411\u91cf\u7684”\u6295\u7f18\u7a0b\u5ea6”
\u751f\u6d3b\u6bd4\u55bb\uff1a\u5c04\u7bad\u6253\u9776 \ud83c\udfaf
\u60f3\u8c61\u4f60\u662f\u4e00\u4e2a\u5f13\u7bad\u624b\uff1a
\u5411\u91cfA = \u4f60\u5c04\u7bad\u7684\u65b9\u5411\u548c\u529b\u6c14
\u5411\u91cfB = \u9776\u5b50\u7684\u671d\u5411\u548c\u5927\u5c0f
\u200c\u70b9\u79ef = \u7bad\u5c04\u4e2d\u9776\u5b50\u7684\u6709\u6548\u80fd\u91cf<\/strong>\u200c
\u65b9\u5411\u8d8a\u4e00\u81f4 \u2192 \u5c04\u4e2d\u8d8a\u6b63 \u2192 \u70b9\u79ef\u8d8a\u5927
\u65b9\u5411\u5782\u76f4 \u2192 \u64e6\u8fb9\u800c\u8fc7 \u2192 \u70b9\u79ef = 0
\u65b9\u5411\u76f8\u53cd \u2192 \u5c04\u53cd\u4e86 \u2192 \u70b9\u79ef\u4e3a\u8d1f\u6570

\u6570\u5b57\u4f8b\u5b50
\u5411\u91cfa = [3, 4] \u5411\u91cfb = [1, 2] \u70b9\u79ef = 3\u00d71 + 4\u00d72 = 3 + 8 = 11<\/code>
\u200c\u89c4\u5219\uff1a\u5bf9\u5e94\u4f4d\u7f6e\u76f8\u4e58\uff0c\u518d\u5168\u90e8\u52a0\u8d77\u6765\u3002<\/strong>\u200c
\u5728Transformer\u91cc\u5e72\u561b\u7528\uff1f
Q\u548cK\u505a\u70b9\u79ef\uff0c\u5c31\u662f\u5728\u7b97\uff1a\u200c“\u6211\u8fd9\u4e2a\u67e5\u8be2”\u548c”\u4f60\u8fd9\u4e2a\u94a5\u5319”\u5230\u5e95\u6709\u591a\u5339\u914d\uff1f<\/strong>\u200c
\u70b9\u79ef\u5927 = \u5f88\u5339\u914d = \u5e94\u8be5\u591a\u5173\u6ce8\u4f60
\u70b9\u79ef\u5c0f = \u4e0d\u592a\u5339\u914d = \u5c11\u5173\u6ce8\u4f60

\u4e8c\u3001\u4f59\u5f26\u76f8\u4f3c\u5ea6\uff1a\u4e24\u4e2a\u5411\u91cf”\u65b9\u5411\u50cf\u4e0d\u50cf”
\u548c\u70b9\u79ef\u7684\u533a\u522b
\u70b9\u79ef
\u65e2\u770b\u65b9\u5411\uff0c\u53c8\u770b\u957f\u5ea6\uff08\u529b\u6c14\u5927\u5c0f\uff09
\u5411\u91cf\u8d8a\u957f\uff0c\u70b9\u79ef\u53ef\u80fd\u8d8a\u5927
\u4f59\u5f26\u76f8\u4f3c\u5ea6
\u53ea\u770b\u65b9\u5411\uff0c\u4e0d\u770b\u957f\u5ea6
\u628a\u957f\u5ea6\u90fd\u5f52\u4e00\u5316\u4e3a1\uff0c\u53ea\u6bd4\u65b9\u5411
\u7528\u627e\u5bf9\u8c61\u6253\u6bd4\u65b9\uff1a


\u200c\u770b\u56fe\u7406\u89e3\uff1a<\/strong>\u200c
\u4f59\u5f26\u76f8\u4f3c\u5ea6\u53ea\u5173\u5fc3\u200c\u5939\u89d2<\/strong>\u200c\uff0c\u4e0d\u5173\u5fc3\u5411\u91cf\u957f\u77ed
\u5939\u89d2\u8d8a\u5c0f \u2192 \u8d8a\u76f8\u4f3c \u2192 \u503c\u8d8a\u63a5\u8fd11
\u5939\u89d290\u5ea6 \u2192 \u4e0d\u76f8\u5173 \u2192 \u503c\u4e3a0
\u5939\u89d2180\u5ea6 \u2192 \u5b8c\u5168\u76f8\u53cd \u2192 \u503c\u4e3a-1
\u6570\u5b57\u4f8b\u5b50
\u5411\u91cfa = [3, 4]\uff0c\u957f\u5ea6 = 5 <\/code>
\u5411\u91cfb = [1, 2]\uff0c\u957f\u5ea6 \u2248 2.236 <\/code>
\u70b9\u79ef = 3\u00d71 + 4\u00d72 = 11 <\/code>
\u4f59\u5f26\u76f8\u4f3c\u5ea6 = \u70b9\u79ef \u00f7 (a\u7684\u957f\u5ea6 \u00d7 b\u7684\u957f\u5ea6) = 11 \u00f7 (5 \u00d7 2.236) = 11 \u00f7 11.18 \u2248 0.984<\/code>
\u503c\u63a5\u8fd11\uff0c\u8bf4\u660e\u8fd9\u4e24\u4e2a\u5411\u91cf\u65b9\u5411\u975e\u5e38\u4e00\u81f4\uff01<\/cite><\/blockquote><\/figure>\n\n\n\n

\u7b2c\u4e8c\u6b65\uff1a\u9664\u4ee5\u221ad_k\u2014\u2014\u522b\u8ba9\u5206\u6570\u592a\u6781\u7aef\u3002<\/strong> \u8fd9\u4e00\u6b65\u662f\u5f88\u591a\u6559\u7a0b\u4e00\u7b14\u5e26\u8fc7\u3001\u4f46\u5176\u5b9e\u975e\u5e38\u5173\u952e\u7684\u4e00\u6b65\uff1a\u628a\u521a\u624d\u5f97\u5230\u7684\u6bcf\u4e2a\u5206\u6570\u90fd\u9664\u4ee5\u221ad_k\uff0c\u5176\u4e2dd_k\u662fK\u5411\u91cf\u7684\u7ef4\u5ea6\u6570\u3002\u4e3a\u4ec0\u4e48\u8981\u505a\u8fd9\u4e2a\u9664\u6cd5\uff1f\u4e0d\u662f\u4ec0\u4e48\u7b3c\u7edf\u7684”\u6570\u503c\u7a33\u5b9a”\uff0c\u800c\u662f\u6709\u660e\u786e\u539f\u56e0\u7684\uff1a\u5f53d_k\u6bd4\u8f83\u5927\u65f6\uff0cQ\u548cK\u505a\u70b9\u79ef\u5f97\u5230\u7684\u6570\u503c\u4f1a\u5929\u7136\u53d8\u5f97\u5f88\u5927\uff0c\u800c\u6570\u503c\u4e00\u5927\uff0c\u9001\u8fdb\u4e0b\u4e00\u6b65\u7684Softmax\u4e4b\u540e\uff0c\u7ed3\u679c\u5c31\u4f1a\u53d8\u5f97\u975e\u5e38\u6781\u7aef\u2014\u2014\u67d0\u4e00\u4e2a\u4f4d\u7f6e\u63a5\u8fd11\uff0c\u5176\u4ed6\u4f4d\u7f6e\u5168\u63a5\u8fd10\uff0c\u5c31\u50cf\u628a\u6240\u6709\u9e21\u86cb\u90fd\u653e\u8fdb\u4e00\u4e2a\u7bee\u5b50\u91cc[12]\u3002\u4e00\u65e6\u51fa\u73b0\u8fd9\u79cd\u60c5\u51b5\uff0cSoftmax\u6240\u5728\u533a\u57df\u7684\u68af\u5ea6\u4f1a\u53d8\u5f97\u975e\u5e38\u5c0f\uff0c\u6a21\u578b\u5b66\u4e60\u5c31\u4f1a\u53d8\u5f97\u5f88\u6162\u3001\u751a\u81f3\u5b66\u4e0d\u52a8\u3002\u9664\u4ee5\u221ad_k\u76f8\u5f53\u4e8e\u628a\u5206\u6570”\u538b\u56de\u4e00\u4e2a\u5408\u9002\u7684\u8303\u56f4”\uff0c\u8ba9Softmax\u80fd\u8f93\u51fa\u4e00\u4e2a\u6bd4\u8f83”\u6e29\u548c”\u7684\u6ce8\u610f\u529b\u5206\u5e03\uff0c\u8bad\u7ec3\u624d\u80fd\u987a\u7545\u8fdb\u884c[8][12]\u3002\u8fd9\u6709\u70b9\u50cf\u8001\u5e08\u6539\u5377\u5b50\u65f6\u53d1\u73b0\u5168\u73ed\u5206\u6570\u8981\u4e480\u8981\u4e48100\uff0c\u533a\u5206\u5ea6\u53cd\u800c\u6ca1\u4e86\uff0c\u4e8e\u662f\u628a\u5206\u6570\u6309\u6bd4\u4f8b\u7f29\u653e\u4e00\u4e0b\uff0c\u8ba9\u5927\u5bb6\u7684\u6210\u7ee9\u62c9\u5f00\u5408\u7406\u5dee\u8ddd\u3002<\/p>\n\n\n\n

Softmax<\/strong><\/p>Softmax \u7684\u6838\u5fc3\u5b9a\u4e49
Softmax\u662f\u4e00\u79cd\u7ecf\u5178\u7684\u6570\u503c\u8f6c\u6362\u5de5\u5177\uff0c\u6838\u5fc3\u4f5c\u7528\u662f\u5c06\u4efb\u610f\u53d6\u503c\u8303\u56f4\u7684\u201c\u968f\u4fbf\u4ec0\u4e48\u6570\u5b57\u201d\u8f6c\u6362\u4e3a\u7b26\u5408\u6982\u7387\u7279\u5f81\u7684\u7ed3\u679c\uff1a\u6240\u6709\u8f93\u51fa\u503c\u7684\u53d6\u503c\u90fd\u57280\u52301\u4e4b\u95f4\uff0c\u4e14\u6240\u6709\u8f93\u51fa\u503c\u7684\u603b\u548c\u6070\u597d\u4e3a1\uff0c\u6700\u7ec8\u5f97\u5230\u7684\u7ed3\u679c\u53ef\u4ee5\u76f4\u63a5\u4f5c\u4e3a\u4e0d\u540c\u7c7b\u522b\u7684\u9884\u6d4b\u6982\u7387\u4f7f\u7528\u3002
Softmax\uff1a\u628a”\u968f\u4fbf\u4ec0\u4e48\u6570\u5b57”\u53d8\u6210”\u6982\u7387”
\u751f\u6d3b\u6bd4\u55bb\uff1a\u5206\u86cb\u7cd5 \ud83c\udf70
\u60f3\u8c61\u4f60\u6709\u4e09\u4e2a\u5c0f\u4f19\u4f34\uff0c\u5404\u81ea\u8bf4\u4e86\u4e00\u4e2a\u6570\u5b57\uff08\u53ef\u4ee5\u662f\u4efb\u610f\u6570\uff0c\u6b63\u6570\u8d1f\u6570\u90fd\u884c\uff09\uff1a
\u5c0f\u660e\u8bf4\uff1a5
\u5c0f\u7ea2\u8bf4\uff1a2
\u5c0f\u521a\u8bf4\uff1a0
\u200cSoftmax = \u6309\u7167\u6570\u5b57\u5927\u5c0f\u5206\u86cb\u7cd5\uff0c\u5927\u7684\u591a\u5206\uff0c\u5c0f\u7684\u5c11\u5206\uff0c\u86cb\u7cd5\u521a\u597d\u5206\u5b8c\u3002<\/strong>\u200c
\u5b83\u6709\u4e09\u4e2a\u795e\u5947\u7684\u7279\u70b9\uff1a
\u200c\u6240\u6709\u8f93\u51fa\u90fd\u5728 0~1 \u4e4b\u95f4<\/strong>\u200c\uff08\u5c31\u50cf\u6982\u7387\uff09
\u200c\u6240\u6709\u8f93\u51fa\u52a0\u8d77\u6765 = 1<\/strong>\u200c\uff08\u86cb\u7cd5\u521a\u597d\u5403\u5b8c\uff09
\u200c\u5927\u7684\u6570\u4f1a\u88ab\u653e\u5927\u4f18\u52bf<\/strong>\u200c\uff08\u8d62\u5bb6\u901a\u5403\u7684\u611f\u89c9\uff09
\u6570\u5b57\u4f8b\u5b50
\u8fd8\u662f\u90a3\u4e09\u4e2a\u6570\uff1a[5, 2, 0]
\u7b2c\u4e00\u6b65\uff1a\u6bcf\u4e2a\u6570\u90fd\u53d6 e \u7684\u6b21\u65b9\uff08e \u2248 2.718\uff0c\u4e00\u79cd\u7279\u6b8a\u7684\u6570\uff09 <\/code>
e^5 \u2248 148.4 <\/code>
e^2 \u2248 7.39 <\/code>
e^0 = 1 <\/code>
\u7b2c\u4e8c\u6b65\uff1a\u7b97\u603b\u548c \u603b\u548c \u2248 148.4 + 7.39 + 1 = 156.79 <\/code>
\u7b2c\u4e09\u6b65\uff1a\u6bcf\u4e2a\u6570\u9664\u4ee5\u603b\u548c\uff08\u5f52\u4e00\u5316\uff09 <\/code>
5 \u2192 148.4 \/ 156.79 \u2248 0.947\uff0894.7%\uff09 <\/code>
2 \u2192 7.39 \/ 156.79 \u2248 0.047\uff084.7%\uff09 <\/code>
0 \u2192 1 \/ 156.79 \u2248 0.006\uff080.6%\uff09<\/code>
\u200c\u7ed3\u679c\uff1a[0.947, 0.047, 0.006]<\/strong>\u200c
\u770b\u5230\u4e86\u5417\uff1f5\u6bd42\u53ea\u5927\u4e863\uff0c\u4f46\u7ecf\u8fc7Softmax\u540e\uff0c5\u7684\u5360\u6bd4\u662f2\u7684\u200c20\u500d<\/strong>\u200c\uff01\u8fd9\u5c31\u662fSoftmax\u7684”\u653e\u5927\u6548\u5e94”\u2014\u2014\u8ba9\u5927\u7684\u66f4\u5927\uff0c\u5c0f\u7684\u66f4\u5c0f\u3002

\u5728Transformer\u91cc\u5e72\u561b\u7528\uff1f
Q\u548cK\u7b97\u5b8c\u70b9\u79ef\u5f97\u5230\u4e00\u5806”\u5339\u914d\u5206”\u540e\uff0c\u5c31\u8981\u7528Softmax\u53d8\u6210\u6743\u91cd\uff1a
“\u6211”\u548c\u6bcf\u4e2a\u8bcd\u7684\u5339\u914d\u5206 \u2192 Softmax \u2192 \u6bcf\u4e2a\u8bcd\u5e94\u8be5\u88ab\u5206\u914d\u591a\u5c11\u6ce8\u610f\u529b\uff08\u767e\u5206\u6bd4\uff09
\u8fd9\u6837\u5c31\u4fdd\u8bc1\u4e86\uff1a
\u6ce8\u610f\u529b\u6743\u91cd\u90fd\u662f\u6b63\u6570
\u6240\u6709\u6ce8\u610f\u529b\u52a0\u8d77\u6765 = 100%
\u6700\u76f8\u5173\u7684\u8bcd\u5206\u5230\u7edd\u5927\u90e8\u5206\u6ce8\u610f\u529b

Q\u548cK\u7528\u200c\u70b9\u79ef<\/strong>\u200c\u7b97\u5339\u914d\u5ea6 \u2192 \u7528\u200cSoftmax<\/strong>\u200c\u53d8\u6210\u767e\u5206\u6bd4\u6743\u91cd \u2192 \u7528\u6743\u91cd\u53bb\u52a0\u6743V \u2192 \u5f97\u5230\u6ce8\u610f\u529b\u7ed3\u679c
\u5728\u6ce8\u610f\u529b\u673a\u5236\u4e2d\uff0c\u9664\u4ee5\u221ad_k\uff08\u7f29\u653e\u70b9\u79ef\uff09\u7684\u4f5c\u7528\u4e4b\u4e00\uff0c\u5c31\u662f\u8ba9\u70b9\u79ef\u7684\u6570\u503c\u8303\u56f4\u66f4\u7a33\u5b9a\uff0c\u6709\u70b9\u50cf”\u90e8\u5206\u5f52\u4e00\u5316”\u7684\u610f\u601d\u3002<\/cite><\/blockquote><\/figure>\n\n\n\n

<\/p>\n\n\n\n

\u7b2c\u4e09\u6b65\uff1aSoftmax\u5f52\u4e00\u5316\u2014\u2014\u628a\u5206\u6570\u53d8\u6210”\u6ce8\u610f\u529b\u767e\u5206\u6bd4”\u3002<\/strong> \u7ecf\u8fc7\u7f29\u653e\u7684\u5206\u6570\u8fd8\u662f\u4e00\u5806\u6742\u4e71\u7684\u6570\u5b57\uff0cSoftmax\u8981\u505a\u4e24\u4ef6\u4e8b\uff1a\u7b2c\u4e00\uff0c\u628a\u6240\u6709\u5206\u6570\u53d8\u62100\u52301\u4e4b\u95f4\u7684\u6b63\u6570\uff08\u56e0\u4e3a”\u5173\u6ce8\u5ea6”\u4e0d\u80fd\u662f\u8d1f\u6570\uff09\uff1b\u7b2c\u4e8c\uff0c\u8ba9\u6240\u6709\u5206\u6570\u52a0\u8d77\u6765\u6b63\u597d\u7b49\u4e8e1\u3002\u4e8e\u662f\u6bcf\u4e2a\u8bcd\u62ff\u5230\u7684\u5c31\u4e0d\u518d\u662f\u4e00\u4e2a\u62bd\u8c61\u5206\u6570\uff0c\u800c\u662f\u4e00\u4e2a\u5b9e\u6253\u5b9e\u7684”\u6743\u91cd\u767e\u5206\u6bd4”\u2014\u2014\u6bd4\u5982\u5206\u7ed9\u8bcdB 0.6\u3001\u8bcdC 0.3\u3001\u8bcd\u81ea\u5df10.1\uff0c\u610f\u601d\u5c31\u662f”\u621160%\u7684\u6ce8\u610f\u529b\u653e\u5728B\u4e0a\uff0c30%\u5728C\u4e0a\uff0c10%\u770b\u770b\u81ea\u5df1”[8][6]\u3002\u8fd9\u4e2a\u6743\u91cd\u5206\u5e03\u6709\u4e00\u4e2a\u975e\u5e38\u76f4\u89c2\u7684\u6027\u8d28\uff1a\u8c01\u548c\u5f53\u524d\u8bcdQ\u6700\u5339\u914d\uff0c\u8c01\u5206\u5230\u7684\u6743\u91cd\u5c31\u6700\u5927\uff1b\u5b8c\u5168\u4e0d\u76f8\u5173\u7684\u8bcd\u5206\u5230\u7684\u6743\u91cd\u5c31\u8fd1\u4e4e\u4e3a0\u3002\u5728\u53ef\u89c6\u5316\u52a8\u753b\u4e2d\uff0c\u4f60\u4f1a\u770b\u5230\u8fd9\u4e00\u6b65\u7ed3\u675f\u540e\uff0c\u6bcf\u4e2a\u8bcd\u5934\u9876\u90fd\u4eae\u8d77\u4e86\u4e00\u6392\u4eae\u6697\u4e0d\u4e00\u7684\u8fde\u7ebf\uff0c\u8d8a\u4eae\u4ee3\u8868\u6743\u91cd\u8d8a\u9ad8[6][7]\u3002<\/p>\n\n\n\n

\u5230\u6b64\u4e3a\u6b62\uff0c\u524d\u4e09\u6b65\u505a\u5b8c\uff0c\u6a21\u578b\u5b8c\u6210\u4e86”\u8be5\u770b\u8c01\u3001\u770b\u591a\u5c11”\u7684\u8ba1\u7b97\uff0c\u53ea\u5269\u4e0b\u6700\u540e\u4e00\u6b65\u2014\u2014\u771f\u6b63\u628a\u4fe1\u606f”\u53d6\u56de\u6765”\u3002<\/p>\n\n\n\n

3.3 \u7b2c\u56db\u6b65\uff1a\u52a0\u6743\u6c42\u548c\u2014\u2014\u628aV\u63c9\u6210\u65b0\u7684\u81ea\u5df1<\/strong><\/strong><\/h3>\n\n\n\n

\u7b2c\u56db\u6b65\u662f\u6574\u4e2a\u6d41\u7a0b\u7684\u6536\u5c3e\uff1a\u7528\u7b2c\u4e09\u6b65\u5f97\u5230\u7684\u6743\u91cd\uff0c\u628a\u6bcf\u4e2a\u8bcd\u7684V\u52a0\u6743\u6c42\u548c\uff0c\u5f97\u5230\u5f53\u524d\u8bcd\u7ecf\u8fc7\u81ea\u6ce8\u610f\u529b\u66f4\u65b0\u540e\u7684\u65b0\u5411\u91cf\u3002<\/p>\n\n\n\n

\u52a0\u6743\u6c42\u548c\u5230\u5e95\u5728\u505a\u4ec0\u4e48\u3002<\/strong> \u56de\u5230\u56fe\u4e66\u9986\u7684\u6bd4\u55bb\uff1a\u4f60\u62ff\u7740\u95ee\u9898\u5b57\u6761\uff08Q\uff09\uff0c\u6bd4\u5bf9\u5b8c\u6240\u6709\u4e66\u810a\u6807\u7b7e\uff08K\uff09\uff0c\u7ba1\u7406\u5458\u7ed9\u4e86\u4f60\u4e00\u5f20\u6e05\u5355\u2014\u2014\u6050\u9f99\u767e\u79d160%\u3001\u5929\u4f53\u56fe\u927430%\u3001\u7ae5\u8bdd\u4e6610%\u3002\u4f60\u4e0d\u662f\u53ea\u62b1\u56de\u6050\u9f99\u767e\u79d1\u90a3\u4e00\u672c\u4e66\uff0c\u800c\u662f\u6309\u7167\u6bd4\u4f8b\u628a\u6bcf\u672c\u4e66\u91cc\u7684\u5185\u5bb9\u5404\u53d6\u4e00\u90e8\u5206\uff1a\u6050\u9f99\u767e\u79d1\u91cc\u62bd60%\u7684\u6bb5\u843d\uff0c\u5929\u4f53\u56fe\u9274\u91cc\u62bd30%\uff0c\u7ae5\u8bdd\u4e66\u91cc\u62bd10%\uff0c\u7136\u540e\u628a\u8fd9\u4e9b\u5185\u5bb9\u63c9\u5728\u4e00\u8d77\uff0c\u88c5\u8fdb\u81ea\u5df1\u8111\u5b50\u91cc\u2014\u2014\u8fd9\u5c31\u662f\u52a0\u6743\u6c42\u548c[2][11]\u3002\u63c9\u5b8c\u4e4b\u540e\uff0c\u4f60\u7684”\u65b0\u77e5\u8bc6”\u91cc\u65e2\u6709\u4e3b\u8981\u7684\u6050\u9f99\u4fe1\u606f\uff08\u4f60\u6700\u5173\u5fc3\u7684\uff09\uff0c\u4e5f\u5e26\u4e00\u70b9\u5929\u6587\u548c\u7ae5\u8bdd\u7684\u80cc\u666f\uff08\u4e0a\u4e0b\u6587\u8bed\u5883\uff09\uff0c\u4e8e\u662f\u4f60\u5bf9\u773c\u524d\u95ee\u9898\u7684\u7406\u89e3\u5c31\u66f4\u65b0\u4e86\u3002<\/p>\n\n\n\n

\u4e00\u4e2a\u5177\u4f53\u7684\u5c0f\u4f8b\u5b50\u3002<\/strong> \u4ee5\u7ecf\u5178\u53e5\u5b50”The animal didn’t cross the street because it was too tired”\u4e2d\u7684”it”\u4e3a\u4f8b\u3002\u7ecf\u8fc7\u524d\u9762\u4e09\u6b65\uff0c”it”\u4f5c\u4e3a\u5f53\u524d\u8bcd\uff0c\u7ed9”animal”\u5206\u914d\u4e86\u5f88\u9ad8\u7684\u6743\u91cd\uff08\u56e0\u4e3a”it”\u95ee\u7684\u662f”\u8c01\u7d2f\u4e86”\uff0c\u800c”animal”\u7684\u6807\u7b7e\u6700\u5951\u5408\uff09\uff0c\u7ed9”street””tired””cross”\u5206\u914d\u4e86\u4e2d\u7b49\u6743\u91cd\uff0c\u7ed9”the””because”\u7b49\u865a\u8bcd\u5206\u914d\u4e86\u5f88\u4f4e\u7684\u6743\u91cd[14][15]\u3002\u5230\u4e86\u7b2c\u56db\u6b65\uff0c”it”\u5c31\u628a”animal”\u7684V\u5927\u90e8\u5206\u62ff\u8fc7\u6765\uff0c\u518d\u63ba\u5165\u4e00\u70b9”tired””street”\u7684V\uff0c\u6405\u548c\u5728\u4e00\u8d77\u3002\u4e8e\u662f”it”\u66f4\u65b0\u540e\u7684\u5411\u91cf\u91cc\uff0c\u6df7\u5165\u4e86\u5927\u91cf”\u52a8\u7269”\u7684\u8bed\u4e49\u4fe1\u606f\u2014\u2014\u6a21\u578b\u56e0\u6b64”\u61c2”\u4e86\u8fd9\u91cc\u7684it\u6307\u7684\u662fanimal\u800c\u4e0d\u662fstreet[2][14]\u3002\u8fd9\u5c31\u662f\u81ea\u6ce8\u610f\u529b\u5b8c\u6210\u6307\u4ee3\u6d88\u89e3\u7684\u5e95\u5c42\u8fc7\u7a0b\u3002<\/p>\n\n\n\n

\u4e3a\u4ec0\u4e48\u8981\u7528V\u800c\u4e0d\u662f\u76f4\u63a5\u7528K\u6216\u8005\u539f\u59cb\u8bcd\u5411\u91cf\u3002<\/strong> \u8fd9\u91cc\u5bb9\u6613\u6709\u4e00\u4e2a\u7591\u60d1\uff1a\u65e2\u7136K\u5df2\u7ecf\u662f\u6bcf\u4e2a\u8bcd\u7684”\u6807\u7b7e”\u4e86\uff0c\u4e3a\u4ec0\u4e48\u8fd8\u8981\u5355\u72ec\u5f04\u4e00\u4e2aV\uff1f\u539f\u56e0\u5728\u4e8e”\u88ab\u67e5\u627e\u7684\u6807\u7b7e”\u548c”\u771f\u6b63\u8981\u4f20\u9012\u7684\u4fe1\u606f”\u4e0d\u9700\u8981\u662f\u540c\u4e00\u4e2a\u4e1c\u897f\u3002K\u8d1f\u8d23”\u597d\u4e0d\u597d\u627e”\uff0c\u8981\u7684\u662f\u8fa8\u8bc6\u5ea6\uff1bV\u8d1f\u8d23”\u63d0\u4f9b\u4ec0\u4e48\u5185\u5bb9”\uff0c\u8981\u7684\u662f\u4fe1\u606f\u91cf[10][8]\u3002\u8fd9\u5c31\u50cf\u56fe\u4e66\u9986\u91cc\uff0c\u4e66\u810a\u6807\u7b7e\uff08K\uff09\u53ea\u9700\u8981\u51e0\u4e2a\u5173\u952e\u8bcd\u5c31\u80fd\u8ba9\u4f60\u5feb\u901f\u5b9a\u4f4d\uff0c\u4f46\u4e66\u91cc\u7684\u5185\u5bb9\uff08V\uff09\u53ef\u4ee5\u662f\u4e30\u5bcc\u7684\u56fe\u6587\u548c\u8be6\u7ec6\u89e3\u91ca\u3002\u8ba9K\u548cV\u5206\u5f00\uff0c\u6a21\u578b\u5c31\u80fd\u5206\u522b\u5b66\u4f1a”\u600e\u4e48\u6253\u6807\u7b7e\u6700\u597d\u641c”\u548c”\u600e\u4e48\u5199\u5185\u5bb9\u6700\u597d\u7528”\u4e24\u4ef6\u4e8b\uff0c\u8fd9\u6bd4\u7528\u540c\u4e00\u4efd\u5411\u91cf\u8eab\u517c\u4e24\u804c\u8981\u7075\u6d3b\u5f97\u591a\u3002<\/p>\n\n\n\n

\u56db\u6b65\u5408\u5728\u4e00\u8d77\u7684\u6d41\u6c34\u7ebf\u901f\u8bb0\u3002<\/strong> \u628a\u6574\u7ae0\u4e32\u8d77\u6765\u5c31\u662f\uff1a\u7b2c\u4e00\u6b65\uff0c\u6bcf\u4e2a\u8bcd\u751f\u6210Q\u3001K\u3001V\u4e09\u4e2a\u5206\u8eab\uff08\u4e58\u4e09\u5957\u6295\u5f71\u77e9\u9635\uff09\uff1b\u7b2c\u4e8c\u6b65\uff0c\u62ffQ\u548c\u6240\u6709K\u505a\u70b9\u79ef\uff0c\u5f97\u5230\u539f\u59cb\u5339\u914d\u5206\uff1b\u7b2c\u4e09\u6b65\uff0c\u9664\u4ee5\u221ad_k\u628a\u5206\u6570\u538b\u56de\u5408\u7406\u533a\u95f4\uff0c\u518d\u8fc7Softmax\u53d8\u6210\u52a0\u8d77\u6765\u7b49\u4e8e1\u7684\u6ce8\u610f\u529b\u6743\u91cd\uff1b\u7b2c\u56db\u6b65\uff0c\u7528\u6743\u91cd\u5bf9\u6240\u6709V\u52a0\u6743\u6c42\u548c\uff0c\u5f97\u5230\u8fd9\u4e2a\u8bcd\u878d\u5408\u4e86\u5168\u53e5\u4e0a\u4e0b\u6587\u7684\u65b0\u8868\u793a[8][12]\u3002\u6574\u53e5\u8bdd\u91cc\u6bcf\u4e2a\u8bcd\u90fd\u5e76\u884c\u5730\u8d70\u5b8c\u8fd9\u56db\u6b65\uff0c\u4e00\u53e5\u8bdd\u7684\u81ea\u6ce8\u610f\u529b\u8ba1\u7b97\u5c31\u5b8c\u6210\u4e86\u3002\u6574\u4e2a\u8fc7\u7a0b\u6ca1\u6709\u4efb\u4f55”\u4f20\u8bdd”\u548c”\u7b49\u5f85”\uff0c\u6240\u6709\u8bcd\u540c\u65f6\u7b97\u3001\u540c\u65f6\u66f4\u65b0\u2014\u2014\u8fd9\u4e5f\u662f\u4e3a\u4ec0\u4e48Transformer\u6bd4\u6211\u4eec\u4e0b\u4e00\u7ae0\u4e4b\u524d\u5148\u63d0\u5230\u7684RNN\u5feb\u5f97\u591a\u7684\u5e95\u5c42\u539f\u56e0\u3002\u638c\u63e1\u4e86\u8fd9\u6761\u56db\u6b65\u6d41\u6c34\u7ebf\uff0c\u518d\u53bb\u770b\u52a8\u753b[6]\u548c\u56fe\u89e3[7]\uff0c\u4f60\u4f1a\u53d1\u73b0\u6bcf\u4e00\u4e2a\u7bad\u5934\u3001\u6bcf\u4e00\u4e2a\u8272\u5757\u90fd\u80fd\u5728\u8fd9\u56db\u6b65\u91cc\u627e\u5230\u5bf9\u5e94\u7684\u4f4d\u7f6e\uff0c\u62bd\u8c61\u516c\u5f0f\u4ece\u6b64\u53d8\u6210\u4e86\u4e00\u5e45\u770b\u5f97\u89c1\u7684\u6d41\u6c34\u7ebf\u753b\u3002<\/p>\n\n\n\n

\u7b2c4\u7ae0 \u591a\u5934\u6ce8\u610f\u529b\u4e0e\u4f4d\u7f6e\u7f16\u7801\u2014\u2014Transformer\u7684\u4e24\u53ea”\u773c\u775b”<\/strong><\/strong><\/h2>\n\n\n\n

\u5982\u679c\u628a\u81ea\u6ce8\u610f\u529b\u6bd4\u4f5cTransformer\u7684”\u5927\u8111”\uff0c\u90a3\u591a\u5934\u6ce8\u610f\u529b\u548c\u4f4d\u7f6e\u7f16\u7801\u5c31\u662f\u5b83\u7684\u4e24\u53ea”\u773c\u775b”\uff1a\u4e00\u53ea\u8ba9\u5b83\u80fd\u540c\u65f6\u4ece\u591a\u4e2a\u89d2\u5ea6\u89c2\u5bdf\u8bcd\u4e0e\u8bcd\u4e4b\u95f4\u7684\u5173\u7cfb\uff0c\u53e6\u4e00\u53ea\u5219\u8ba9\u5b83\u770b\u6e05\u8bcd\u7684\u5148\u540e\u987a\u5e8f\u3002\u4e24\u8005\u7f3a\u4e00\u4e0d\u53ef\u2014\u2014\u5c11\u4e86\u591a\u5934\uff0c\u6a21\u578b\u5c31\u50cf\u53ea\u7528\u4e00\u79cd\u6ee4\u955c\u770b\u4e16\u754c\uff0c\u9519\u8fc7\u8bed\u6cd5\u3001\u8bed\u4e49\u3001\u6307\u4ee3\u7b49\u591a\u79cd\u5173\u7cfb\uff1b\u5c11\u4e86\u4f4d\u7f6e\u7f16\u7801\uff0c”\u732b\u8ffd\u72d7”\u548c”\u72d7\u8ffd\u732b”\u5728\u6a21\u578b\u773c\u91cc\u5c31\u6210\u4e86\u540c\u4e00\u56de\u4e8b\u3002\u8fd9\u4e00\u7ae0\u6211\u4eec\u5c31\u6765\u5206\u522b\u8ba4\u8bc6\u8fd9\u4e24\u53ea”\u773c\u775b”\u3002<\/p>\n\n\n\n

4.1 \u591a\u5934\u6ce8\u610f\u529b\uff1a\u4e0d\u662f\u91cd\u590d\u8ba1\u7b97\uff0c\u662f\u591a\u526f\u773c\u955c\u770b\u4e16\u754c<\/strong><\/strong><\/h3>\n\n\n\n

\u8bb8\u591a\u521d\u5b66\u8005\u7b2c\u4e00\u6b21\u542c\u5230”\u591a\u5934\u6ce8\u610f\u529b”\u65f6\uff0c\u4f1a\u4e0b\u610f\u8bc6\u5730\u4ee5\u4e3a\u8fd9\u662f\u628a\u540c\u4e00\u4efd\u81ea\u6ce8\u610f\u529b\u8ba1\u7b97\u91cd\u590d\u505a\u4e86\u597d\u51e0\u904d\uff0c\u5c31\u50cf\u8ba9\u540c\u4e00\u4e2a\u4eba\u628a\u540c\u4e00\u9053\u9898\u505a\u516b\u6b21\u3002\u8fd9\u662f\u4e00\u4e2a\u5e38\u89c1\u7684\u8bef\u89e3\u3002\u5b9e\u9645\u4e0a\uff0c\u591a\u5934\u6ce8\u610f\u529b\u7684\u771f\u6b63\u542b\u4e49\uff0c\u662f\u628a\u6a21\u578b\u7684”\u6ce8\u610f\u529b”\u62c6\u6210\u591a\u4e2a\u72ec\u7acb\u7684”\u5934”\uff0c\u8ba9\u6bcf\u4e2a\u5934\u5728\u4e0d\u540c\u7684”\u5b50\u7a7a\u95f4”\u91cc\u89c2\u5bdf\u8bcd\u4e0e\u8bcd\u4e4b\u95f4\u4e0d\u540c\u7c7b\u578b\u7684\u5173\u7cfb[10]\u3002<\/p>\n\n\n\n

\u6211\u4eec\u53ef\u4ee5\u7528\u4e00\u4e2a\u751f\u6d3b\u5316\u7684\u6bd4\u55bb\u6765\u7406\u89e3\u3002\u60f3\u8c61\u4f60\u5728\u770b\u4e00\u573a\u8db3\u7403\u6bd4\u8d5b\uff1a\u5982\u679c\u53ea\u6234\u4e00\u526f\u770b\u6574\u4f53\u6218\u672f\u7684\u773c\u955c\uff0c\u4f60\u80fd\u770b\u5230\u7403\u961f\u7684\u9635\u578b\u548c\u8dd1\u4f4d\uff1b\u5982\u679c\u6234\u4e0a\u53e6\u4e00\u526f\u770b\u4e2a\u4eba\u6280\u672f\u7684\u773c\u955c\uff0c\u4f60\u4f1a\u6ce8\u610f\u5230\u7403\u5458\u7684\u76d8\u5e26\u548c\u5c04\u95e8\uff1b\u518d\u6234\u4e0a\u4e00\u526f\u76ef\u4eba\u9632\u5b88\u7684\u773c\u955c\uff0c\u4f60\u53c8\u4f1a\u770b\u5230\u8c01\u5728\u9632\u8c01\u3002\u591a\u5934\u6ce8\u610f\u529b\u5c31\u50cf\u662f\u540c\u65f6\u6234\u4e0a\u591a\u526f\u8fd9\u6837\u7684”\u773c\u955c”\uff0c\u6bcf\u4e00\u4e2a\u5934\u4e13\u6ce8\u4e8e\u6355\u6349\u4e00\u79cd\u7279\u5b9a\u7684\u5173\u7cfb\u6a21\u5f0f\uff0c\u6700\u540e\u628a\u6240\u6709\u5934\u770b\u5230\u7684\u4fe1\u606f\u62fc\u5728\u4e00\u8d77\uff0c\u5f62\u6210\u5bf9\u6574\u53e5\u8bdd\u66f4\u5168\u9762\u7684\u7406\u89e3[10][16]\u3002<\/p>\n\n\n\n

\u8fd9\u79cd\u8bbe\u8ba1\u7075\u611f\u5176\u5b9e\u6765\u6e90\u4e8e\u5377\u79ef\u795e\u7ecf\u7f51\u7edc\uff08CNN\uff09\u4e2d\u7684”\u591a\u8f93\u51fa\u901a\u9053”\u601d\u60f3\u3002\u5728CNN\u91cc\uff0c\u4e0d\u540c\u7684\u5377\u79ef\u6838\u53ef\u4ee5\u5206\u522b\u8bc6\u522b\u8fb9\u7f18\u3001\u7eb9\u7406\u3001\u989c\u8272\u7b49\u4e0d\u540c\u7279\u5f81\uff1b\u5728Transformer\u91cc\uff0c\u4e0d\u540c\u7684\u6ce8\u610f\u529b\u5934\u5219\u53ef\u4ee5\u5206\u522b\u8bc6\u522b\u4e0d\u540c\u7c7b\u578b\u7684\u8bed\u8a00\u5173\u7cfb[16]\u3002\u6bd4\u5982\u5728\u7ecf\u5178\u4f8b\u53e5”The animal didn’t cross the street because it was too tired”\u4e2d\uff0c\u4e00\u4e2a\u6ce8\u610f\u529b\u5934\u53ef\u80fd\u4e13\u95e8\u8d1f\u8d23”\u6307\u4ee3\u6d88\u89e3”\uff0c\u628a”it”\u548c”animal”\u8fde\u8d77\u6765\uff1b\u53e6\u4e00\u4e2a\u5934\u53ef\u80fd\u5173\u6ce8\u8bed\u6cd5\u4f9d\u8d56\uff0c\u628a”cross”\u548c”street”\u8054\u7cfb\u8d77\u6765\uff1b\u8fd8\u6709\u7684\u5934\u53ef\u80fd\u6355\u6349\u8bed\u4e49\u76f8\u4f3c\u6027\uff0c\u53d1\u73b0”animal”\u548c”tired”\u4e4b\u95f4\u7684\u72b6\u6001\u5173\u8054[14]\u3002\u5982\u679c\u53ea\u6709\u4e00\u4e2a\u5934\uff0c\u6a21\u578b\u5c31\u5f88\u96be\u540c\u65f6\u628a\u8fd9\u4e9b\u5173\u7cfb\u90fd\u5904\u7406\u597d\u3002<\/p>\n\n\n\n

\u4ece\u8ba1\u7b97\u6d41\u7a0b\u4e0a\u770b\uff0c\u591a\u5934\u6ce8\u610f\u529b\u5e76\u4e0d\u590d\u6742\u3002\u6a21\u578b\u5148\u628aQ\u3001K\u3001V\u5206\u522b\u901a\u8fc7\u4e0d\u540c\u7684\u7ebf\u6027\u53d8\u6362\u77e9\u9635\uff08\u5c31\u662f\u6211\u4eec\u524d\u9762\u8bf4\u7684\u6295\u5f71\u77e9\u9635<\/strong>\uff09”\u5207”\u6210h\u4efd\uff08\u539f\u8bba\u6587\u4e2dh=8\uff09\uff0c\u6bcf\u4efd\u9001\u5165\u4e00\u4e2a\u72ec\u7acb\u7684\u6ce8\u610f\u529b\u5934\u5e76\u884c\u8ba1\u7b97\uff0c\u5f97\u5230h\u7ec4\u52a0\u6743\u540e\u7684\u7ed3\u679c\uff0c\u6700\u540e\u518d\u628a\u8fd9h\u7ec4\u7ed3\u679c\u62fc\u63a5\u8d77\u6765\uff0c\u7ecf\u8fc7\u4e00\u6b21\u7ebf\u6027\u53d8\u6362\u878d\u5408\u6210\u6700\u7ec8\u8f93\u51fa[10][16]\u3002\u5173\u952e\u5728\u4e8e\uff1a\u6bcf\u4e2a\u5934\u4f7f\u7528\u7684\u53d8\u6362\u77e9\u9635\u53c2\u6570\u662f\u72ec\u7acb\u5b66\u4e60\u7684\uff0c\u56e0\u6b64\u5b83\u4eec\u4f1a\u81ea\u52a8\u5b66\u4f1a\u5173\u6ce8\u4e0d\u540c\u79cd\u7c7b\u7684\u5173\u7cfb\u2014\u2014\u8fd9\u4e0d\u662f\u4eba\u4e3a\u6307\u5b9a\u7684\uff0c\u800c\u662f\u6a21\u578b\u5728\u8bad\u7ec3\u8fc7\u7a0b\u4e2d\u81ea\u5df1”\u53d1\u73b0”\u7684\u3002<\/p>\n\n\n\n

\u591a\u5934\u6ce8\u610f\u529b\u5982\u4f55\u5de5\u4f5c\u7684\uff1f<\/p>\u5f88\u591a\u4eba\u4ee5\u4e3a”\u5207”\u662f\u628a\u5411\u91cf\u50cf\u5207\u86cb\u7cd5\u4e00\u6837\u5207\u6210\u51e0\u6bb5\uff0c\u5176\u5b9e\u66f4\u51c6\u786e\u7684\u8bf4\u6cd5\u662f\u200c“\u7528h\u5957\u4e0d\u540c\u7684\u6ee4\u955c\u5404\u770b\u4e00\u904d”<\/strong>\u200c\u3002
\u6211\u7528\u6570\u5b57\u7ed9\u4f60\u8d70\u4e00\u904d\u5b8c\u6574\u6d41\u7a0b\uff1a
\u5047\u8bbe\u6761\u4ef6\uff1a
\u8f93\u5165\u8bcd\u5411\u91cf\u7ef4\u5ea6 = 512\uff08\u5c31\u8bf4\u8fd9\u4e2a\u8bcd\u6709512\u4e2a\u7279\u5f81\uff09
\u5934\u6570 h = 8
\u6bcf\u4e2a\u5934\u5206\u5230\u7684\u7ef4\u5ea6 = 512 \u00f7 8 = 64
\u6b65\u9aa41\uff1a\u4e09\u5957\u5927\u77e9\u9635\u5206\u522b\u751f\u6210Q\u3001K\u3001V

\u8f93\u5165 x\uff08512\u7ef4\uff09 \u00d7 W_Q\uff08512\u00d7512\u7684\u5927\u77e9\u9635\uff09 \u5f97\u5230 Q\u603b\uff08512\u7ef4\uff09 <\/code>
\u8f93\u5165 x\uff08512\u7ef4\uff09 \u2193 \u00d7 W_K\uff08512\u00d7512\u7684\u5927\u77e9\u9635\uff09 \u5f97\u5230 <\/code>K\u603b\uff08512\u7ef4\uff09 <\/code>
\u8f93\u5165 x\uff08512\u7ef4\uff09 \u2193 \u00d7 W_V\uff08512\u00d7512\u7684\u5927\u77e9\u9635\uff09 \u5f97\u5230 <\/code>V\u603b\uff08512\u7ef4\uff09<\/code>
\u6b65\u9aa42\uff1a”\u5207”\u62108\u4efd
Q\u603b[512] \u2192 Q1[64], Q2[64], Q3[64], ..., Q8[64] \uff08\u524d64\u7ef4\u7ed9\u59341\uff0c\u63a5\u4e0b\u676564\u7ef4\u7ed9\u59342\u2026\u2026\uff09 <\/code>
K\u603b[512] \u2192 K1[64], K2[64], K3[64], ..., K8[64] <\/code>
V\u603b[512] \u2192 V1[64], V2[64], V3[64], ..., V8[64]<\/code>
\u200c\u5c31\u50cf\u4e00\u6839512\u8282\u7684\u957f\u9999\u80a0\uff0c\u5207\u62108\u6bb5\uff0c\u6bcf\u6bb564\u8282\u3002<\/strong>\u200c
\u6b65\u9aa43\uff1a8\u4e2a\u5934\u5e76\u884c\u8ba1\u7b97
\u59341\uff1aQ1 \u00d7 K1\u1d40 \u2192 softmax \u2192 \u52a0\u6743V1 \u2192 \u8f93\u51fa1[64] <\/code>
\u59342\uff1aQ2 \u00d7 K2\u1d40 \u2192 softmax \u2192 \u52a0\u6743V2 \u2192 \u8f93\u51fa2[64] <\/code>
\u59343\uff1aQ3 \u00d7 K3\u1d40 \u2192 softmax \u2192 \u52a0\u6743V3 \u2192 \u8f93\u51fa3[64] ... <\/code>
\u59348\uff1aQ8 \u00d7 K8\u1d40 \u2192 softmax \u2192 \u52a0\u6743V8 \u2192 \u8f93\u51fa8[64]<\/code>
\u6b65\u9aa44\uff1a\u62fc\u63a5 + \u6700\u540e\u4e00\u6b21\u7ebf\u6027\u53d8\u6362
\u62fc\u63a5\uff1a\u8f93\u51fa1 + \u8f93\u51fa2 + ... + \u8f93\u51fa8 = \u603b\u8f93\u51fa[512] \u2193 \u00d7 W_O\uff08512\u00d7512\u7684\u77e9\u9635\uff09 \u6700\u7ec8\u8f93\u51fa[512]<\/code>
\u6700\u540e\u8fd9\u6b65W_O\u5c31\u662f”\u7ec4\u957f”\uff0c\u628a8\u4e2a\u4fa6\u63a2\u7684\u7ed3\u8bba\u7efc\u5408\u6210\u4e00\u4efd\u5b8c\u6574\u62a5\u544a\u3002

\u56db\u3001\u51e0\u4f55\u7406\u89e3\uff1a\u591a\u5934 = \u4ece\u591a\u4e2a\u89d2\u5ea6\u770b\u540c\u4e00\u4e2a\u4e1c\u897f
\u200c\u770b\u56fe\u7406\u89e3\uff1a<\/strong>\u200c
\u4e00\u4e2a\u8bcd\u5411\u91cf\u5c31\u50cf\u4e00\u4e2a3D\u7269\u4f53
\u6bcf\u4e2a\u6ce8\u610f\u529b\u5934 = \u4ece\u4e0d\u540c\u89d2\u5ea6\u62cd\u4e00\u5f20\u7167\uff08\u6295\u5f71\u5230\u4e0d\u540c\u7684\u5b50\u7a7a\u95f4\uff09
\u59341\u4ece\u6b63\u9762\u62cd\uff0c\u770b\u5230”\u8bcd\u4e49\u5173\u7cfb”
\u59342\u4ece\u4fa7\u9762\u62cd\uff0c\u770b\u5230”\u8bed\u6cd5\u5173\u7cfb”
\u59343\u4ece\u4e0a\u9762\u62cd\uff0c\u770b\u5230”\u6307\u4ee3\u5173\u7cfb”
\u6700\u540e\u628a\u6240\u6709\u7167\u7247\u62fc\u8d77\u6765\uff0c\u5c31\u5f97\u5230\u4e86\u6bd4\u5355\u89d2\u5ea6\u4e30\u5bcc\u5f97\u591a\u7684\u4fe1\u606f

\u4e94\u3001\u6bcf\u4e2a\u5934\u4e3a\u4ec0\u4e48\u4f1a\u81ea\u52a8\u5206\u5de5<\/strong>\uff1f
\u8fd9\u662f\u6700\u795e\u5947\u7684\u5730\u65b9\u2014\u2014\u200c\u6ca1\u4eba\u544a\u8bc9\u6bcf\u4e2a\u5934\u8be5\u5173\u6ce8\u4ec0\u4e48\uff0c\u5b83\u4eec\u662f”\u81ea\u7136\u9009\u62e9”\u51fa\u6765\u7684\u3002<\/strong>\u200c
\u7528”\u516b\u4e2a\u4eba\u76f2\u62e7\u9b54\u65b9”\u6253\u6bd4\u65b9 \ud83c\udfb2
\u60f3\u8c61\u67098\u4e2a\u4eba\u4e00\u8d77\u76f2\u62e7\u4e00\u4e2a\u9b54\u65b9\uff0c\u76ee\u6807\u662f\u628a\u9b54\u65b9\u8fd8\u539f\uff1a
\u4e00\u5f00\u59cb\uff0c8\u4e2a\u4eba\u90fd\u662f\u4e71\u62e7\u7684\uff0c\u6bcf\u4e2a\u4eba\u7684\u52a8\u4f5c\u90fd\u5dee\u4e0d\u591a
\u6bcf\u6b21\u62e7\u5b8c\u540e\uff0c\u6709\u4e2a\u88c1\u5224\u8bf4\uff1a”\u79bb\u8fd8\u539f\u8fd8\u5dee\u591a\u5c11”\uff08\u635f\u5931\u51fd\u6570\uff09
\u79bb\u76ee\u6807\u8fd1\u7684\u90a3\u51e0\u4e2a\u4eba\u7684\u52a8\u4f5c\u88ab\u4fdd\u7559\u5e76\u53d1\u626c\u5149\u5927
\u6162\u6162\u5730\uff0c\u6709\u4eba\u4e13\u95e8\u8d1f\u8d23”\u5bf9\u9f50\u767d\u8272\u9762”\uff0c\u6709\u4eba\u4e13\u95e8\u8d1f\u8d23”\u62fc\u597d\u7b2c\u4e00\u5c42”\uff0c\u6709\u4eba\u4e13\u95e8\u8d1f\u8d23”\u9876\u5c42\u5341\u5b57”\u2026\u2026
\u200c\u5206\u5de5\u4e0d\u662f\u5b89\u6392\u7684\uff0c\u662f”\u8fd9\u6837\u505a\u6210\u529f\u7387\u6700\u9ad8”\u81ea\u7136\u6f14\u5316\u51fa\u6765\u7684<\/strong>\u200c
\u56de\u5230\u591a\u5934\u6ce8\u610f\u529b\uff1a
\u200c\u8bad\u7ec3\u521d\u671f<\/strong>\u200c\uff1a8\u4e2a\u5934\u7684\u77e9\u9635\u90fd\u662f\u968f\u673a\u7684\uff0c\u5927\u5bb6\u5e72\u7684\u6d3b\u5dee\u4e0d\u591a
\u200c\u8bad\u7ec3\u4e2d<\/strong>\u200c\uff1a\u5982\u679c\u59341\u5076\u7136\u53d1\u73b0”\u5173\u6ce8\u4e3b\u8bed\u548c\u5bbe\u8bed\u7684\u5173\u7cfb”\u80fd\u8ba9\u9884\u6d4b\u66f4\u51c6\uff0c\u5b83\u5c31\u4f1a\u88ab\u671d\u8fd9\u4e2a\u65b9\u5411\u5fae\u8c03
\u200c\u8bad\u7ec3\u540e\u671f<\/strong>\u200c\uff1a8\u4e2a\u5934\u5404\u6709\u6240\u957f\u2014\u2014\u6709\u7684\u5173\u6ce8\u610f\u4e49\u76f8\u4f3c\u6027\u3001\u6709\u7684\u5173\u6ce8\u8bed\u6cd5\u4f9d\u8d56\u3001\u6709\u7684\u5173\u6ce8\u6307\u4ee3\u5173\u7cfb\u2026\u2026
\u200c\u4e3a\u4ec0\u4e48\u4e0d\u4f1a\u5e72\u91cd\u590d\u7684\u6d3b\uff1f<\/strong>\u200c
\u5982\u679c\u4e24\u4e2a\u5934\u5e72\u4e00\u6a21\u4e00\u6837\u7684\u4e8b\uff0c\u90a3\u6a21\u578b\u5c31\u6d6a\u8d39\u4e86\u5bb9\u91cf\u3002\u8bad\u7ec3\u8fc7\u7a0b\u4e2d\uff0c\u6a21\u578b\u4f1a\u81ea\u52a8\u8ba9\u5b83\u4eec”\u5206\u5316”\uff0c\u56e0\u4e3a\u5404\u6709\u6240\u957f\u6bd4\u90fd\u4e00\u6837\u6548\u679c\u66f4\u597d\u2014\u2014\u8fd9\u53eb\u200c\u5197\u4f59\u6700\u5c11\u5316<\/strong>\u200c\u3002

\u6a21\u578b\u662f\u5982\u4f55\u8ba9\u4ed6\u4eec\u5206\u5316\u7684\u5462\uff1f
\u8fd9\u4e2a\u95ee\u9898\u95ee\u5230\u4e86\u6df1\u5ea6\u5b66\u4e60\u7684\u4e00\u4e2a\u6838\u5fc3\u5965\u79d8\u2014\u2014\u200c\u6a21\u578b\u600e\u4e48”\u81ea\u53d1”\u957f\u51fa\u5206\u5de5\u7684\uff1f<\/strong>\u200c \u7b54\u6848\u85cf\u5728\u200c\u968f\u673a\u521d\u59cb\u5316 + \u68af\u5ea6\u4e0b\u964d + \u5bf9\u79f0\u6027\u7834\u7f3a<\/strong>\u200c\u8fd9\u4e09\u4ef6\u4e8b\u91cc\u3002\u6211\u7528\u6700\u901a\u4fd7\u7684\u65b9\u5f0f\u7ed9\u4f60\u8bb2\u3002

\u4e00\u3001\u751f\u6d3b\u6bd4\u55bb\uff1a\u516b\u4e2a\u4eba\u627e\u5b9d\u85cf
\u60f3\u8c61\u67098\u4e2a\u4eba\u5728\u4e00\u7247\u5927\u5c71\u4e0a\u627e\u5b9d\u85cf\uff08\u5b9d\u85cf = \u6700\u5c0f\u635f\u5931\uff09\uff1a
\u8868\u683c
\u9636\u6bb5
\u53d1\u751f\u4e86\u4ec0\u4e48
\u5bf9\u5e94\u6a21\u578b
\u200c\u7b2c1\u5929<\/strong>\u200c
8\u4e2a\u4eba\u88ab\u8499\u7740\u773c\u968f\u673a\u6254\u5230\u5c71\u7684\u4e0d\u540c\u4f4d\u7f6e
\u968f\u673a\u521d\u59cb\u5316\uff0c8\u4e2a\u5934\u7684\u77e9\u9635\u4e0d\u4e00\u6837
\u200c\u7b2c2\u5929<\/strong>\u200c
\u6bcf\u4e2a\u4eba\u90fd\u671d”\u5f80\u4e0b\u8d70”\u7684\u65b9\u5411\u8d70\uff08\u54ea\u8fb9\u4f4e\u5f80\u54ea\u8d70\uff09
\u68af\u5ea6\u4e0b\u964d\uff0c\u6bcf\u4e2a\u5934\u5404\u81ea\u4f18\u5316
\u200c\u7b2c3\u5929<\/strong>\u200c
\u6709\u4eba\u8d70\u5230\u4e86A\u5c71\u8c37\uff0c\u6709\u4eba\u8d70\u5230\u4e86B\u5c71\u8c37\uff0c\u6709\u4eba\u8d70\u5230\u4e86C\u5c71\u6d1e
\u5206\u5316\u6210\u4e0d\u540c\u7684\u5c40\u90e8\u6700\u4f18
\u200c\u6700\u540e<\/strong>\u200c
8\u4e2a\u4eba\u5360\u636e\u4e868\u4e2a\u4e0d\u540c\u7684”\u597d\u4f4d\u7f6e”\uff0c\u4ece\u4e0d\u540c\u89d2\u5ea6\u6316\u5b9d\u85cf
8\u4e2a\u5934\u5404\u6709\u6240\u957f\uff0c\u4e92\u8865
\u200c\u4e3a\u4ec0\u4e48\u4e0d\u4f1a\u8d70\u5230\u540c\u4e00\u4e2a\u5730\u65b9\uff1f<\/strong>\u200c
\u56e0\u4e3a\u4ed6\u4eec\u200c\u8d77\u70b9\u5c31\u4e0d\u4e00\u6837<\/strong>\u200c\uff0c\u800c\u4e14\u5c71\u7684\u5730\u5f62\u51f9\u51f8\u4e0d\u5e73\u2014\u2014\u8d77\u70b9\u7a0d\u5fae\u4e0d\u540c\uff0c\u8d70\u7684\u8def\u5c31\u5b8c\u5168\u4e0d\u540c\uff0c\u6700\u540e\u843d\u5728\u4e0d\u540c\u7684\u5c71\u8c37\u91cc\u3002
\u8fd9\u5728\u7269\u7406\u5b66\u91cc\u53eb\u200c\u5bf9\u79f0\u6027\u7834\u7f3a<\/strong>\u200c\u2014\u2014\u672c\u6765\u5927\u5bb6\u662f\u5bf9\u79f0\u7684\uff08\u90fd\u4e00\u6837\uff09\uff0c\u4f46\u56e0\u4e3a\u4e00\u70b9\u70b9\u968f\u673a\u5dee\u5f02\uff0c\u5bf9\u79f0\u6027\u88ab\u6253\u7834\u4e86\uff0c\u7136\u540e\u8d8a\u8d70\u8d8a\u4e0d\u4e00\u6837\u3002

\u4e8c\u3001\u51e0\u4f55\u56fe\u89e3\uff1a\u5206\u5316\u8fc7\u7a0b\u53ef\u89c6\u5316


\u200c\u770b\u56fe\u7406\u89e3\uff1a<\/strong>\u200c
\u200c\u8bad\u7ec3\u5f00\u59cb<\/strong>\u200c\uff1a8\u4e2a\u5934\u968f\u673a\u6563\u5e03\u5728”\u53c2\u6570\u5c71”\u7684\u4e0d\u540c\u4f4d\u7f6e\uff08\u989c\u8272\u4e0d\u4e00\uff0c\u4f4d\u7f6e\u4e0d\u4e00\uff09
\u200c\u8bad\u7ec3\u4e2d\u671f<\/strong>\u200c\uff1a\u6bcf\u4e2a\u5934\u90fd\u5f80\u4f4e\u5904\u6eda\uff0c\u4f46\u56e0\u4e3a\u8d77\u70b9\u4e0d\u540c\uff0c\u6eda\u5411\u4e86\u4e0d\u540c\u7684\u5c71\u8c37
\u200c\u8bad\u7ec3\u540e\u671f<\/strong>\u200c\uff1a8\u4e2a\u5934\u5206\u522b\u843d\u57288\u4e2a\u4e0d\u540c\u7684”\u597d\u4f4d\u7f6e”\u2014\u2014\u6bcf\u4e2a\u4f4d\u7f6e\u5bf9\u5e94\u4e00\u79cd”\u64c5\u957f\u7684\u5173\u6ce8\u6a21\u5f0f”
\u8fd9\u91cc\u7684”\u5c71”\u4e0d\u662f3D\u7684\uff0c\u800c\u662f\u51e0\u767e\u4e07\u7ef4\u7684\u8d85\u66f2\u9762\uff0c\u5c71\u8c37\u591a\u5230\u6570\u4e0d\u6e05\uff0c\u6240\u4ee58\u4e2a\u5934\u51e0\u4e4e\u4e0d\u53ef\u80fd\u6070\u597d\u6eda\u8fdb\u540c\u4e00\u4e2a\u5c71\u8c37\u3002

\u4e09\u3001\u5173\u952e\u673a\u5236\u4e00\uff1a\u968f\u673a\u521d\u59cb\u5316\u2014\u2014”\u8d77\u8dd1\u7ebf\u5c31\u4e0d\u4e00\u6837”
\u8fd9\u662f\u6700\u6839\u672c\u7684\u539f\u56e0\u3002\u6a21\u578b\u5728\u8bad\u7ec3\u5f00\u59cb\u524d\uff0c\u6bcf\u4e2a\u5934\u7684\u6295\u5f71\u77e9\u9635\u90fd\u662f\u200c\u968f\u673a\u751f\u6210<\/strong>\u200c\u7684\uff1a

\u59341\u7684Q\u77e9\u9635 = \u968f\u673a\u6570\uff08\u6bd4\u5982 [0.12, -0.34, 0.56, ...]\uff09 <\/code>
\u59342\u7684Q\u77e9\u9635 = \u968f\u673a\u6570\uff08\u6bd4\u5982 [-0.21, 0.43, 0.08, ...]\uff09 <\/code>
\u59343\u7684Q\u77e9\u9635 = \u968f\u673a\u6570\uff08\u6bd4\u5982 [0.67, -0.02, -0.41, ...]\uff09 ... <\/code>
8\u4e2a\u5934\u7684\u77e9\u9635\uff0c\u4ece\u7b2c\u4e00\u4e2a\u6570\u5b57\u5f00\u59cb\u5c31\u4e0d\u4e00\u6837\uff01<\/code>
\u200c\u6253\u4e2a\u6bd4\u65b9\uff1a<\/strong>\u200c
\u4f60\u67098\u4e2a\u4e00\u6a21\u4e00\u6837\u7684\u9e21\u86cb\uff0c\u770b\u8d77\u6765\u5b8c\u5168\u5bf9\u79f0\u3002\u4f46\u5982\u679c\u4f60\u628a\u5b83\u4eec\u90fd\u653e\u5728\u659c\u5761\u4e0a\uff1a
\u54ea\u6015\u6bcf\u4e2a\u9e21\u86cb\u521d\u59cb\u4f4d\u7f6e\u53ea\u5dee\u4e860.001\u6beb\u7c73
\u6eda\u4e0b\u53bb\u4e4b\u540e\uff0c\u5b83\u4eec\u7684\u8def\u5f84\u4f1a\u5dee\u5f97\u8d8a\u6765\u8d8a\u8fdc
\u6700\u540e\u505c\u5728\u5b8c\u5168\u4e0d\u540c\u7684\u5730\u65b9
\u200c\u521d\u59cb\u7684\u5fae\u5c0f\u968f\u673a\u5dee\u5f02 \u2192 \u8bad\u7ec3\u4e2d\u88ab\u653e\u5927 \u2192 \u6700\u7ec8\u5206\u5de5\u5b8c\u5168\u4e0d\u540c<\/strong>\u200c
\u8fd9\u5c31\u662f”\u5dee\u4e4b\u6beb\u5398\uff0c\u8c2c\u4ee5\u5343\u91cc”\u7684\u6570\u5b66\u7248\u672c\u3002

\u56db\u3001\u5173\u952e\u673a\u5236\u4e8c\uff1a\u68af\u5ea6\u4e0b\u964d\u2014\u2014”\u5404\u8d70\u5404\u7684\u4f18\u5316\u8def”
\u68af\u5ea6\u4e0b\u964d\u7684\u89c4\u5219\u5f88\u7b80\u5355\uff1a\u200c\u6bcf\u4e2a\u53c2\u6570\u90fd\u5f80”\u8ba9\u635f\u5931\u53d8\u5c0f”\u7684\u65b9\u5411\u8c03\u6574\u4e00\u70b9\u70b9\u3002<\/strong>\u200c
\u4f46\u56e0\u4e3a8\u4e2a\u5934\u7684\u521d\u59cb\u77e9\u9635\u4e0d\u4e00\u6837\uff0c\u5b83\u4eec\u7684”\u68af\u5ea6\u65b9\u5411”\u4e5f\u4e0d\u4e00\u6837\uff1a

\u59341\u5f53\u524d\u72b6\u6001 \u2192 \u7b97\u635f\u5931 \u2192 \u68af\u5ea6\u8bf4\uff1a\"\u4f60\u5e94\u8be5\u591a\u5173\u6ce8\u8bed\u6cd5\u5173\u7cfb\" \u2192 \u59341\u671d\u8fd9\u4e2a\u65b9\u5411\u5fae\u8c03 <\/code>
\u59342\u5f53\u524d\u72b6\u6001 \u2192 \u7b97\u635f\u5931 \u2192 \u68af\u5ea6\u8bf4\uff1a\"\u4f60\u5e94\u8be5\u591a\u5173\u6ce8\u8bcd\u4e49\u76f8\u4f3c\" \u2192 \u59342\u671d\u8fd9\u4e2a\u65b9\u5411\u5fae\u8c03 <\/code>
\u59343\u5f53\u524d\u72b6\u6001 \u2192 \u7b97\u635f\u5931 \u2192 \u68af\u5ea6\u8bf4\uff1a\"\u4f60\u5e94\u8be5\u591a\u5173\u6ce8\u6307\u4ee3\u5173\u7cfb\" \u2192 \u59343\u671d\u8fd9\u4e2a\u65b9\u5411\u5fae\u8c03 ...<\/code>
\u200c\u4e3a\u4ec0\u4e48\u68af\u5ea6\u7ed9\u6bcf\u4e2a\u5934\u7684\u5efa\u8bae\u4e0d\u4e00\u6837\uff1f<\/strong>\u200c
\u56e0\u4e3a\u5b83\u4eec\u7684”\u73b0\u72b6”\u4e0d\u4e00\u6837\u3002\u4e3e\u4e2a\u751f\u6d3b\u5316\u7684\u4f8b\u5b50\uff1a
\u4f60\u548c\u670b\u53cb\u4e00\u8d77\u505a”\u5f39\u5f13\u6253\u9776”\u6bd4\u8d5b\uff0c\u76ee\u6807\u662f\u6253\u4e2d\u9776\u5fc3\uff1a
\u4f60\u73b0\u5728\u504f\u5de6\u4e86 \u2192 \u6559\u7ec3\u8ba9\u4f60”\u5f80\u53f3\u8c03\u4e00\u70b9”
\u4f60\u670b\u53cb\u73b0\u5728\u504f\u4e0a\u4e86 \u2192 \u6559\u7ec3\u8ba9\u4ed6”\u5f80\u4e0b\u538b\u4e00\u70b9”
\u540c\u6837\u7684\u76ee\u6807\uff08\u6700\u5c0f\u5316\u635f\u5931\uff09\uff0c\u4f46\u56e0\u4e3a\u5f53\u524d\u72b6\u6001\u4e0d\u540c\uff0c\u8c03\u6574\u65b9\u5411\u5b8c\u5168\u4e0d\u540c\u3002
\u200c\u68af\u5ea6\u5c31\u662f\u90a3\u4e2a\u6559\u7ec3\u2014\u2014\u5b83\u544a\u8bc9\u6bcf\u4e2a\u5934”\u4f60\u8be5\u5f80\u54ea\u8fb9\u8d70”\uff0c\u4f46\u56e0\u4e3a\u6bcf\u4e2a\u5934\u7ad9\u7684\u4f4d\u7f6e\u4e0d\u4e00\u6837\uff0c\u8d70\u7684\u8def\u81ea\u7136\u5c31\u4e0d\u4e00\u6837\u4e86\u3002<\/strong>\u200c

\u4e94\u3001\u5173\u952e\u673a\u5236\u4e09\uff1a”\u91cd\u590d\u4e86\u4e5f\u6ca1\u7528”\u2014\u2014\u81ea\u7136\u9009\u62e9\u538b\u529b
\u5047\u8bbe\u51fa\u73b0\u4e86\u6781\u7aef\u60c5\u51b5\uff1a\u4e24\u4e2a\u5934\u78b0\u5de7\u771f\u7684\u53d8\u5f97\u5f88\u50cf\u4e86\uff0c\u4f1a\u53d1\u751f\u4ec0\u4e48\uff1f
\u7b54\u6848\u662f\uff1a\u200c\u5b83\u4eec\u4f1a\u88ab”\u81ea\u7136\u9009\u62e9”\u63a8\u5f00\u3002<\/strong>\u200c
\u7528”\u751f\u6001\u4f4d”\u6253\u6bd4\u65b9 \ud83c\udf3f
\u60f3\u8c61\u4e00\u4e2a\u5c9b\u4e0a\u53ea\u6709\u4e24\u79cd\u98df\u7269\uff1a\u575a\u679c\u548c\u6606\u866b\u3002
\u5982\u679c\u67098\u53ea\u9e1f\uff0c\u90fd\u53ea\u5403\u575a\u679c \u2192 \u575a\u679c\u4e0d\u591f\u5403\uff0c\u5927\u5bb6\u90fd\u997f\u809a\u5b50\uff08\u6548\u679c\u5dee\uff09
\u5982\u679c\u5176\u4e2d3\u53ea\u9e1f\u6539\u5403\u6606\u866b \u2192 \u575a\u679c\u591f\u5403\uff0c\u6606\u866b\u4e5f\u591f\u5403\uff0c\u5927\u5bb6\u90fd\u6d3b\u5f97\u597d\uff08\u6548\u679c\u597d\uff09
\u200c\u591a\u5934\u6ce8\u610f\u529b\u4e5f\u662f\u4e00\u6837\u7684\u9053\u7406\uff1a<\/strong>\u200c
\u5982\u679c8\u4e2a\u5934\u90fd\u5173\u6ce8”\u8bed\u6cd5\u5173\u7cfb” \u2192 \u5197\u4f59\u4e86\uff0c\u7b49\u4e8e\u91cd\u590d\u52b3\u52a8\uff0c\u5bf9\u6700\u7ec8\u9884\u6d4b\u7684\u5e2e\u52a9\u4e0d\u5927
\u5982\u679c\u6709\u7684\u5173\u6ce8\u8bed\u6cd5\u3001\u6709\u7684\u5173\u6ce8\u610f\u4e49\u3001\u6709\u7684\u5173\u6ce8\u6307\u4ee3\u3001\u6709\u7684\u5173\u6ce8\u4f4d\u7f6e \u2192 \u4fe1\u606f\u4e92\u8865\uff0c\u6700\u7ec8\u9884\u6d4b\u66f4\u51c6
\u635f\u5931\u51fd\u6570\u4f1a\u81ea\u52a8”\u5956\u52b1”\u90a3\u4e9b\u80fd\u5e26\u6765\u200c\u65b0\u4fe1\u606f<\/strong>\u200c\u7684\u5934\uff0c”\u60e9\u7f5a”\u90a3\u4e9b\u8ddf\u522b\u4eba\u5e72\u4e00\u6837\u6d3b\u7684\u5934\u2014\u2014\u56e0\u4e3a\u91cd\u590d\u7684\u4fe1\u606f\u4e0d\u4f1a\u8ba9\u635f\u5931\u8fdb\u4e00\u6b65\u4e0b\u964d\u3002
\u8fd9\u5c31\u50cf\u521b\u4e1a\uff1a\u5982\u679c\u4e00\u6761\u8857\u5df2\u7ecf\u670910\u5bb6\u5976\u8336\u5e97\u4e86\uff0c\u4f60\u518d\u5f00\u7b2c11\u5bb6\u4e5f\u8d5a\u4e0d\u5230\u591a\u5c11\u94b1\uff1b\u4f46\u5982\u679c\u4f60\u5f00\u4e00\u5bb6\u5496\u5561\u5e97\uff0c\u53cd\u800c\u751f\u610f\u5f88\u597d\u2014\u2014\u200c\u586b\u8865\u7a7a\u767d\u7684\u6536\u76ca\u6700\u5927\u3002<\/strong>\u200c

\u516d\u3001\u4e00\u4e2a\u8d85\u7b80\u5355\u7684\u6570\u5b66\u6f14\u793a
\u6211\u4eec\u505a\u4e2a\u601d\u60f3\u5b9e\u9a8c\uff0c\u611f\u53d7\u4e00\u4e0b”\u5bf9\u79f0\u6027\u7834\u7f3a”\uff1a
\u5047\u8bbe\uff1a
\u6709\u4e24\u4e2a\u5934\uff0c\u53c2\u6570\u5206\u522b\u662f a \u548c b
\u635f\u5931\u51fd\u6570 = (a – 3)\u00b2 + (b – 5)\u00b2 + 0.5\u00d7(a – b)\u00b2\u524d\u4e24\u9879\uff1a\u5404\u81ea\u8981\u9760\u8fd1\u76ee\u6807\u503c
\u7b2c\u4e09\u9879\uff1a\u5982\u679c a \u548c b \u592a\u50cf\u4e86\uff0c\u5c31\u52a0\u4e00\u70b9\u60e9\u7f5a\uff08\u6a21\u62df”\u5197\u4f59\u6d6a\u8d39”\uff09
\u5982\u679c\u521d\u59cb\u5b8c\u5168\u4e00\u6837\uff1aa = b = 0
\u68af\u5ea6\u544a\u8bc9a\uff1a\u5f803\u8d70 \u68af\u5ea6\u544a\u8bc9b\uff1a\u5f805\u8d70 \u2192 \u56e0\u4e3a\u76ee\u6807\u503c\u4e0d\u540c\uff0ca\u548cb\u81ea\u7136\u5c31\u5206\u5f00\u4e86<\/code>
\u771f\u5b9e\u7684Transformer\u4e2d\uff1a
\u867d\u7136\u6ca1\u6709\u663e\u5f0f\u7684”\u76f8\u4f3c\u60e9\u7f5a\u9879”\uff0c\u4f46\u200c\u635f\u5931\u51fd\u6570\u672c\u8eab\u5c31\u9690\u542b\u4e86\u8fd9\u79cd\u538b\u529b<\/strong>\u200c\u2014\u2014\u56e0\u4e3a\u91cd\u590d\u7684\u5934\u5bf9\u964d\u4f4e\u635f\u5931\u6ca1\u6709\u989d\u5916\u8d21\u732e\uff0c\u68af\u5ea6\u4f1a\u81ea\u7136\u5730\u628a\u6bcf\u4e2a\u5934\u63a8\u5411\u4e0d\u540c\u7684”\u6709\u7528\u65b9\u5411”\u3002


\u200c\u591a\u5934\u81ea\u52a8\u5206\u5de5 = \u968f\u673a\u8d77\u70b9\u4e0d\u540c + \u5404\u81ea\u671d\u635f\u5931\u51cf\u5c11\u7684\u65b9\u5411\u8d70 + \u53c2\u6570\u7a7a\u95f4\u592a\u5927\u592a\u590d\u6742 + “\u586b\u8865\u7a7a\u767d”\u6536\u76ca\u6700\u5927 \u2192 \u81ea\u7136\u5c31\u5206\u5316\u51fa\u4e86\u4e0d\u540c\u7684\u4e13\u957f\u3002<\/strong>\u200c
\u5c31\u50cf\u4e00\u7247\u539f\u59cb\u68ee\u6797\uff0c\u6ca1\u4eba\u89c4\u5212\u8c01\u957f\u5728\u54ea\u91cc\uff0c\u4f46\u6700\u540e\uff1a
\u9ad8\u7684\u6811\u5360\u636e\u4e86\u9876\u5c42\u9633\u5149
\u77ee\u7684\u704c\u6728\u5360\u636e\u4e86\u6797\u4e0b\u7a7a\u95f4
\u82d4\u85d3\u957f\u5728\u9634\u6697\u6f6e\u6e7f\u7684\u5730\u65b9
\u8611\u83c7\u957f\u5728\u8150\u6728\u4e0a
\u200c\u6bcf\u4e00\u79cd\u751f\u7269\u90fd\u627e\u5230\u4e86\u81ea\u5df1\u7684”\u751f\u6001\u4f4d”\u2014\u2014\u591a\u5934\u6ce8\u610f\u529b\u7684\u6bcf\u4e2a\u5934\uff0c\u4e5f\u662f\u5982\u6b64\u3002<\/strong><\/cite><\/blockquote><\/figure>\n\n\n\n

\u6b63\u56e0\u4e3a\u6bcf\u4e2a\u5934\u4e13\u6ce8\u4e0d\u540c\u5b50\u7a7a\u95f4\uff0c\u591a\u5934\u6ce8\u610f\u529b\u624d\u4e0d\u662f”\u91cd\u590d\u52b3\u52a8”\uff0c\u800c\u662f”\u5206\u5de5\u534f\u4f5c”\u3002\u8fd9\u5c31\u50cf\u4e00\u4e2a\u7f16\u8f91\u90e8\uff1a\u6709\u8d1f\u8d23\u4e8b\u5b9e\u6838\u67e5\u7684\u7f16\u8f91\uff0c\u6709\u8d1f\u8d23\u8bed\u8a00\u6da6\u8272\u7684\u7f16\u8f91\uff0c\u8fd8\u6709\u8d1f\u8d23\u903b\u8f91\u68b3\u7406\u7684\u7f16\u8f91\uff0c\u6bcf\u4e2a\u4eba\u770b\u540c\u4e00\u7bc7\u7a3f\u5b50\u5173\u6ce8\u7684\u70b9\u4e0d\u540c\uff0c\u5408\u5728\u4e00\u8d77\u624d\u80fd\u4ea7\u51fa\u9ad8\u8d28\u91cf\u7684\u7a3f\u4ef6\u3002Transformer\u7684\u5f3a\u5927\u8868\u8fbe\u80fd\u529b\uff0c\u5f88\u5927\u7a0b\u5ea6\u4e0a\u6b63\u6765\u81ea\u4e8e\u8fd9\u79cd\u591a\u89c6\u89d2\u5e76\u884c\u5efa\u6a21\u7684\u673a\u5236\u3002<\/p>\n\n\n\n

4.2 \u4f4d\u7f6e\u7f16\u7801\uff1a\u7ed9\u6bcf\u4e2a\u8bcd\u8d34\u4e0a”\u5ea7\u4f4d\u53f7”<\/strong><\/strong><\/h3>\n\n\n\n

\u7406\u89e3\u4e86\u591a\u5934\u6ce8\u610f\u529b\u4e4b\u540e\uff0c\u4f60\u53ef\u80fd\u4f1a\u53d1\u73b0\u4e00\u4e2a\u9690\u85cf\u7684”bug”\uff1a\u81ea\u6ce8\u610f\u529b\u5728\u8ba1\u7b97\u65f6\uff0c\u662f\u628a\u53e5\u5b50\u91cc\u6240\u6709\u8bcd\u540c\u65f6\u62ff\u8fc7\u6765\u4e24\u4e24\u6bd4\u5bf9\u7684\uff0c\u5b83\u6839\u672c\u4e0d\u7ba1\u8c01\u5728\u524d\u3001\u8c01\u5728\u540e\u3002\u5bf9\u81ea\u6ce8\u610f\u529b\u6765\u8bf4\uff0c”\u6211\u6253\u4f60”\u548c”\u4f60\u6253\u6211”\u8fd9\u4e24\u53e5\u8bdd\uff0c\u5982\u679c\u8bcd\u5b8c\u5168\u4e00\u6837\u53ea\u662f\u987a\u5e8f\u4e0d\u540c\uff0c\u7b97\u51fa\u6765\u7684\u7ed3\u679c\u5c45\u7136\u662f\u4e00\u6837\u7684\u2014\u2014\u56e0\u4e3a\u5b83\u770b\u5230\u7684\u8bcd\u96c6\u5408\u5b8c\u5168\u76f8\u540c\uff0cQKV\u70b9\u79ef\u7684\u7ed3\u679c\u4e0d\u4f1a\u56e0\u4e3a\u8bcd\u5e8f\u8c03\u6362\u800c\u6539\u53d8[12][10]\u3002<\/p>\n\n\n\n

\u8fd9\u663e\u7136\u662f\u4e2a\u5927\u95ee\u9898\u3002\u8bed\u8a00\u91cc\u987a\u5e8f\u5c31\u662f\u610f\u4e49\u672c\u8eab\uff1a”\u732b\u8ffd\u72d7”\u548c”\u72d7\u8ffd\u732b”\u63cf\u8ff0\u7684\u662f\u622a\u7136\u76f8\u53cd\u7684\u573a\u666f\uff1b”\u867d\u7136\u4ed6\u5f88\u52aa\u529b\uff0c\u4f46\u662f\u6ca1\u8003\u597d”\u548c”\u867d\u7136\u4ed6\u6ca1\u8003\u597d\uff0c\u4f46\u662f\u5f88\u52aa\u529b”\u4f20\u8fbe\u7684\u6001\u5ea6\u4e5f\u5b8c\u5168\u4e0d\u540c\u3002\u81ea\u6ce8\u610f\u529b\u8fd9\u79cd”\u5168\u5458\u540c\u65f6\u5728\u7ebf”\u7684\u673a\u5236\u867d\u7136\u5e26\u6765\u4e86\u5e76\u884c\u8ba1\u7b97\u7684\u5de8\u5927\u4f18\u52bf\uff0c\u5374\u5929\u751f\u662f\u4e2a”\u987a\u5e8f\u76f2”\u3002\u4e3a\u4e86\u5f25\u8865\u8fd9\u4e2a\u7f3a\u9677\uff0cTransformer\u5f15\u5165\u4e86\u4f4d\u7f6e\u7f16\u7801\uff08Positional Encoding\uff09\u2014\u2014\u7b80\u5355\u8bf4\uff0c\u5c31\u662f\u7ed9\u53e5\u5b50\u91cc\u6bcf\u4e2a\u8bcd\u8d34\u4e0a\u4e00\u5f20”\u5ea7\u4f4d\u53f7”\u6807\u7b7e\uff0c\u8ba9\u6a21\u578b\u77e5\u9053\u8c01\u5728\u7b2c\u51e0\u4e2a\u4f4d\u7f6e\u4e0a[12][10]\u3002<\/p>\n\n\n\n

\u4f4d\u7f6e\u7f16\u7801\u7684\u601d\u8def\u5176\u5b9e\u5f88\u6734\u7d20\uff1a\u65e2\u7136\u6a21\u578b\u81ea\u5df1\u5206\u4e0d\u6e05\u987a\u5e8f\uff0c\u90a3\u6211\u4eec\u5c31\u5728\u6bcf\u4e2a\u8bcd\u7684\u8bcd\u5411\u91cf\u4e0a”\u52a0”\u4e00\u4e2a\u4ee3\u8868\u4f4d\u7f6e\u7684\u5411\u91cf\uff0c\u8ba9\u6a21\u578b\u4e00\u770b\u8fd9\u4e2a\u5411\u91cf\u5c31\u77e5\u9053”\u54e6\uff0c\u4f60\u662f\u7b2c3\u4e2a\u8bcd”\u3002\u539f\u8bba\u6587\u91c7\u7528\u7684\u662f\u4e00\u79cd\u57fa\u4e8e\u6b63\u5f26\u548c\u4f59\u5f26\u51fd\u6570\u7684\u4f4d\u7f6e\u7f16\u7801\u65b9\u6848\uff1a\u7528\u4e0d\u540c\u9891\u7387\u7684sin\u548ccos\u51fd\u6570\u751f\u6210\u4e00\u7ec4\u56fa\u5b9a\u7684\u5411\u91cf\uff0c\u6bcf\u4e2a\u4f4d\u7f6e\u5bf9\u5e94\u552f\u4e00\u7684\u4e00\u4e32\u6570\u5b57\uff0c\u4f4d\u7f6e\u76f8\u8fd1\u7684\u5411\u91cf\u5728\u6570\u503c\u4e0a\u4e5f\u76f8\u8fd1\uff0c\u4f4d\u7f6e\u8fdc\u7684\u5219\u5dee\u5f02\u660e\u663e[12]\u3002\u8fd9\u5c31\u50cf\u7535\u5f71\u9662\u91cc\u7684\u5ea7\u4f4d\u53f7\u2014\u2014\u6392\u6570\u548c\u53f7\u6570\u7ec4\u5408\u8d77\u6765\u552f\u4e00\u786e\u5b9a\u4e00\u4e2a\u4f4d\u7f6e\uff0c\u76f8\u90bb\u5ea7\u4f4d\u7684\u53f7\u6570\u662f\u8fde\u7eed\u7684\u3002<\/p>\n\n\n\n

\u4f60\u53ef\u80fd\u4f1a\u95ee\uff1a\u4e3a\u4ec0\u4e48\u4e0d\u76f4\u63a5\u7528”1\u30012\u30013\u2026\u2026”\u8fd9\u6837\u7684\u6570\u5b57\u5e8f\u53f7\u5462\uff1f\u56e0\u4e3a\u5f53\u53e5\u5b50\u5f88\u957f\u65f6\uff0c\u6570\u5b57\u4f1a\u8d8a\u6765\u8d8a\u5927\uff0c\u53ef\u80fd\u5f71\u54cd\u8bad\u7ec3\u7a33\u5b9a\u6027\uff1b\u66f4\u91cd\u8981\u7684\u662f\uff0c\u6b63\u5f26\u4f59\u5f26\u7f16\u7801\u6709\u4e00\u4e2a\u5de7\u5999\u7684\u6027\u8d28\uff1a\u4efb\u610f\u4e24\u4e2a\u4f4d\u7f6e\u4e4b\u95f4\u7684\u76f8\u5bf9\u4f4d\u7f6e\u5173\u7cfb\uff0c\u53ef\u4ee5\u901a\u8fc7\u7ebf\u6027\u53d8\u6362\u8868\u793a\u51fa\u6765\uff0c\u8fd9\u8ba9\u6a21\u578b\u4e0d\u4ec5\u77e5\u9053”\u7edd\u5bf9\u4f4d\u7f6e”\uff0c\u8fd8\u80fd\u611f\u77e5”\u76f8\u5bf9\u8ddd\u79bb”[12]\u3002\u5f53\u7136\uff0c\u540e\u6765\u7684\u7814\u7a76\u8005\u4e5f\u63d0\u51fa\u4e86\u53ef\u5b66\u4e60\u7684\u4f4d\u7f6e\u7f16\u7801\u3001\u76f8\u5bf9\u4f4d\u7f6e\u7f16\u7801\u3001\u65cb\u8f6c\u4f4d\u7f6e\u7f16\u7801\uff08RoPE\uff09\u7b49\u591a\u79cd\u65b9\u6848\uff0c\u4f46\u6838\u5fc3\u601d\u60f3\u59cb\u7ec8\u4e00\u81f4\u2014\u2014\u7ed9\u65e0\u5e8f\u7684\u81ea\u6ce8\u610f\u529b\u6ce8\u5165\u987a\u5e8f\u4fe1\u606f\u3002<\/p>\n\n\n\n

\u6709\u4e86\u4f4d\u7f6e\u7f16\u7801\u4e4b\u540e\uff0c\u6bcf\u4e2a\u8f93\u5165\u6a21\u578b\u7684\u8bcd\u5411\u91cf\u5176\u5b9e\u662f”\u8bcd\u7684\u8bed\u4e49\u5411\u91cf+\u4f4d\u7f6e\u5411\u91cf”\u7684\u7ec4\u5408\u3002\u8fd9\u6837\u5f53\u81ea\u6ce8\u610f\u529b\u8ba1\u7b97\u8bcd\u4e0e\u8bcd\u7684\u5173\u7cfb\u65f6\uff0c\u4e0d\u4ec5\u80fd\u770b\u5230\u4e24\u4e2a\u8bcd\u7684\u8bed\u4e49\u5185\u5bb9\uff0c\u8fd8\u80fd\u770b\u5230\u5b83\u4eec\u76f8\u9694\u591a\u8fdc\u3001\u8c01\u5148\u8c01\u540e\u3002\u5c31\u50cf\u5728\u5168\u5458\u7535\u8bdd\u4f1a\u8bae\u4e0a\uff0c\u6bcf\u4e2a\u4eba\u4e0d\u4ec5\u8981\u53d1\u8a00\uff0c\u8fd8\u8981\u81ea\u62a5\u5ea7\u4f4d\u53f7\uff0c\u8fd9\u6837\u5927\u5bb6\u867d\u7136\u540c\u65f6\u5728\u7ebf\uff0c\u4e5f\u80fd\u6e05\u695a\u77e5\u9053\u662f\u8c01\u5728\u4ec0\u4e48\u4f4d\u7f6e\u8bf4\u4e86\u4ec0\u4e48\u3002\u4f4d\u7f6e\u7f16\u7801\u5c31\u50cf\u8fd9\u573a\u4f1a\u8bae\u7684”\u5ea7\u4f4d\u8868”\uff0c\u6ca1\u6709\u5b83\uff0c\u518d\u806a\u660e\u7684\u53c2\u4f1a\u8005\u4e5f\u4f1a\u88ab\u987a\u5e8f\u641e\u7cca\u6d82\u3002<\/p>\n\n\n\n

4.3 \u4e24\u53ea”\u773c\u775b”\u5982\u4f55\u534f\u540c\u5de5\u4f5c<\/strong><\/strong><\/h3>\n\n\n\n

\u591a\u5934\u6ce8\u610f\u529b\u8d1f\u8d23”\u591a\u89d2\u5ea6\u770b\u5173\u7cfb”\uff0c\u4f4d\u7f6e\u7f16\u7801\u8d1f\u8d23”\u770b\u6e05\u987a\u5e8f”\uff0c\u8fd9\u4e24\u53ea”\u773c\u775b”\u5e76\u4e0d\u662f\u5404\u81ea\u4e3a\u6218\uff0c\u800c\u662f\u5728Transformer\u7684\u6bcf\u4e00\u5c42\u91cc\u7d27\u5bc6\u914d\u5408\uff0c\u5171\u540c\u5b8c\u6210\u5bf9\u53e5\u5b50\u7684\u7406\u89e3\u3002\u6211\u4eec\u53ef\u4ee5\u7528\u4e00\u4e2a\u65e5\u5e38\u573a\u666f\u6765\u4e32\u8d77\u5b83\u4eec\u7684\u534f\u4f5c\u8fc7\u7a0b\uff1a\u60f3\u8c61\u4f60\u5728\u9605\u8bfb\u4e00\u6bb5\u60ac\u7591\u5c0f\u8bf4\uff0c\u9700\u8981\u540c\u65f6\u5f04\u6e05\u695a”\u8c01\u505a\u4e86\u4ec0\u4e48”\uff08\u591a\u79cd\u8bed\u4e49\u5173\u7cfb\uff09\u548c”\u4e8b\u60c5\u53d1\u751f\u7684\u5148\u540e\u987a\u5e8f”\uff08\u4f4d\u7f6e\u4fe1\u606f\uff09\u3002<\/p>\n\n\n\n

\u5177\u4f53\u6765\u8bf4\uff0c\u8f93\u5165\u7684\u8bcd\u5411\u91cf\u5728\u8fdb\u5165\u6ce8\u610f\u529b\u5c42\u4e4b\u524d\uff0c\u5c31\u5df2\u7ecf\u53e0\u52a0\u4e86\u4f4d\u7f6e\u7f16\u7801\uff0c\u56e0\u6b64\u6bcf\u4e2a\u5934\u62ff\u5230\u7684Q\u3001K\u3001V\u91cc\u5929\u7136\u5305\u542b\u4f4d\u7f6e\u4fe1\u606f\u3002\u5f53\u67d0\u4e2a\u6ce8\u610f\u529b\u5934\u5728\u5224\u65ad\u4ee3\u8bcd”\u4ed6”\u6307\u4ee3\u7684\u662f\u8c01\u65f6\uff0c\u5b83\u4e0d\u4ec5\u4f1a\u6bd4\u8f83”\u4ed6”\u548c\u5176\u4ed6\u8bcd\u8bed\u4e49\u4e0a\u7684\u76f8\u4f3c\u5ea6\uff0c\u8fd8\u4f1a\u53c2\u8003\u4f4d\u7f6e\u4e0a\u7684\u8ddd\u79bb\u2014\u2014\u901a\u5e38\u4ee3\u8bcd\u4e0d\u4f1a\u79bb\u5b83\u6240\u6307\u4ee3\u7684\u540d\u8bcd\u592a\u8fdc[14][10]\u3002\u800c\u53e6\u4e00\u4e2a\u8d1f\u8d23\u6355\u6349\u8bed\u6cd5\u5173\u7cfb\u7684\u5934\uff0c\u53ef\u80fd\u4f1a\u5229\u7528\u4f4d\u7f6e\u4fe1\u606f\u8bc6\u522b\u51fa”\u52a8\u8bcd\u5f80\u5f80\u51fa\u73b0\u5728\u540d\u8bcd\u540e\u9762”\u8fd9\u6837\u7684\u987a\u5e8f\u6a21\u5f0f\u3002\u518d\u6bd4\u5982\uff0c\u5728\u5904\u7406”\u5c0f\u660e\u628a\u5c0f\u7ea2\u7684\u4e66\u5305\u9012\u7ed9\u4e86\u8001\u5e08”\u8fd9\u53e5\u8bdd\u65f6\uff0c\u6709\u7684\u5934\u4f1a\u5173\u6ce8”\u5c0f\u660e”\u548c”\u9012\u7ed9”\u4e4b\u95f4\u7684\u65bd\u4e8b\u5173\u7cfb\uff0c\u6709\u7684\u5934\u4f1a\u5173\u6ce8”\u5c0f\u7ea2”\u548c”\u4e66\u5305”\u4e4b\u95f4\u7684\u6240\u5c5e\u5173\u7cfb\uff0c\u8fd8\u6709\u7684\u5934\u4f1a\u501f\u52a9\u4f4d\u7f6e\u4fe1\u606f\u7406\u6e05”\u9012\u7ed9”\u8fd9\u4e2a\u52a8\u4f5c\u7684\u5148\u540e\u987a\u5e8f\uff1a\u5148\u62ff\u4e66\u5305\u3001\u518d\u9012\u7ed9\u8001\u5e08\u3002<\/p>\n\n\n\n

\u8fd9\u79cd\u534f\u540c\u5e26\u6765\u4e86Transformer\u76f8\u5bf9\u4e8eRNN\u7684\u4e00\u4e2a\u672c\u8d28\u533a\u522b\u3002\u5728RNN\u7684”\u4f20\u8bdd\u6e38\u620f”\u91cc\uff0c\u987a\u5e8f\u4fe1\u606f\u662f\u5929\u7136\u5185\u5d4c\u5728\u5904\u7406\u6d41\u7a0b\u4e2d\u7684\u2014\u2014\u56e0\u4e3a\u5b83\u5fc5\u987b\u4ece\u5de6\u5230\u53f3\u4e00\u4e2a\u8bcd\u4e00\u4e2a\u8bcd\u5730\u8bfb\uff0c\u4f4d\u7f6e\u4fe1\u606f\u9760”\u7b2c\u51e0\u4e2a\u65f6\u95f4\u6b65\u5904\u7406”\u6765\u9690\u542b\u4f20\u9012\uff0c\u4f46\u4ee3\u4ef7\u662f\u4fe1\u606f\u5728\u4e00\u6b21\u6b21\u4f20\u9012\u4e2d\u4f1a\u8870\u51cf\uff0c\u4e14\u65e0\u6cd5\u5e76\u884c[3][4]\u3002\u800cTransformer\u7684\u505a\u6cd5\u66f4\u50cf\u662f”\u7ed9\u6bcf\u4e2a\u4eba\u53d1\u4e00\u4efd\u5e26\u5ea7\u4f4d\u53f7\u7684\u5b8c\u6574\u4f1a\u8bae\u8d44\u6599”\uff1a\u6240\u6709\u4eba\u540c\u65f6\u770b\u5230\u5168\u90e8\u4fe1\u606f\uff08\u81ea\u6ce8\u610f\u529b+\u591a\u5934\u5e76\u884c\uff09\uff0c\u540c\u65f6\u6bcf\u4e2a\u4eba\u90fd\u77e5\u9053\u81ea\u5df1\u548c\u4ed6\u4eba\u5750\u5728\u54ea\u91cc\uff08\u4f4d\u7f6e\u7f16\u7801\uff09\uff0c\u65e2\u4fdd\u7559\u4e86\u5168\u5458\u5e76\u884c\u6c9f\u901a\u7684\u6548\u7387\uff0c\u53c8\u6ca1\u6709\u4e22\u5931\u987a\u5e8f\u8fd9\u4e00\u5173\u952e\u4fe1\u606f\u3002<\/p>\n\n\n\n

\u503c\u5f97\u4e00\u63d0\u7684\u662f\uff0c\u591a\u5934\u6ce8\u610f\u529b\u548c\u4f4d\u7f6e\u7f16\u7801\u7684\u7ec4\u5408\u5e76\u975e\u4e00\u6210\u4e0d\u53d8\u3002\u5728\u539f\u8bba\u6587\u7684\u8bbe\u8ba1\u4e2d\uff0c\u4f4d\u7f6e\u7f16\u7801\u662f\u56fa\u5b9a\u7684\u6b63\u5f26\u51fd\u6570\uff0c\u591a\u5934\u662f8\u5934\u5e76\u884c\uff1b\u800c\u5728\u540e\u6765\u7684BERT\u3001GPT\u7b49\u6a21\u578b\u4e2d\uff0c\u7814\u7a76\u8005\u4eec\u5c1d\u8bd5\u4e86\u66f4\u591a\u53d8\u5316\uff1a\u6709\u7684\u628a\u4f4d\u7f6e\u7f16\u7801\u6362\u6210\u53ef\u5b66\u4e60\u7684\u53c2\u6570\uff0c\u6709\u7684\u7528\u76f8\u5bf9\u4f4d\u7f6e\u4ee3\u66ff\u7edd\u5bf9\u4f4d\u7f6e\uff0c\u6709\u7684\u628a\u5934\u6570\u589e\u52a0\u5230\u51e0\u5341\u4e0a\u767e\u3002\u4f46\u65e0\u8bba\u600e\u4e48\u53d8\uff0c\u6838\u5fc3\u903b\u8f91\u59cb\u7ec8\u6ca1\u53d8\u2014\u2014\u7528\u591a\u5934\u673a\u5236\u8986\u76d6\u591a\u6837\u7684\u5173\u7cfb\u7c7b\u578b\uff0c\u7528\u4f4d\u7f6e\u7f16\u7801\u8865\u8db3\u987a\u5e8f\u611f\u77e5\uff0c\u4e24\u8005\u4e00\u8d77\u8ba9Transformer\u65e2\u770b\u5f97\u5168\u3001\u53c8\u770b\u5f97\u51c6\u3002\u7406\u89e3\u4e86\u8fd9\u4e24\u53ea”\u773c\u775b”\u5982\u4f55\u5206\u5de5\u914d\u5408\uff0c\u4f60\u5c31\u638c\u63e1\u4e86Transformer\u9664QKV\u4e4b\u5916\u6700\u6838\u5fc3\u7684\u4e24\u5927\u8bbe\u8ba1\u5de7\u601d\uff0c\u4e3a\u4e0b\u4e00\u7ae0\u5bf9\u6bd4\u5b83\u4e3a\u4f55\u80fd\u5168\u9762\u8d85\u8d8aRNN\u6253\u4e0b\u4e86\u57fa\u7840\u3002<\/p>\n\n\n\n

\u7b2c5\u7ae0 \u4e3a\u4ec0\u4e48\u6bd4RNN\u5f3a\u2014\u2014\u5e76\u884c\u8ba1\u7b97\u4e0e\u957f\u8ddd\u79bb\u4f9d\u8d56\u7684\u901a\u4fd7\u5bf9\u6bd4<\/strong><\/strong><\/h2>\n\n\n\n

\u524d\u51e0\u7ae0\u6211\u4eec\u5df2\u7ecf\u628aTransformer\u7684”\u773c\u775b”\u2014\u2014\u81ea\u6ce8\u610f\u529b\u3001QKV\u3001\u591a\u5934\u6ce8\u610f\u529b\u3001\u4f4d\u7f6e\u7f16\u7801\u2014\u2014\u4e00\u4e2a\u4e00\u4e2a\u62c6\u5f00\u770b\u8fc7\u4e86\u3002\u4f46\u8981\u771f\u6b63\u7406\u89e3Transformer\u4e3a\u4ec0\u4e48\u80fd\u6380\u8d77\u5927\u6a21\u578b\u9769\u547d\uff0c\u8fd8\u5f97\u628a\u5b83\u653e\u56de\u5386\u53f2\u7684\u5750\u6807\u7cfb\u91cc\uff0c\u548c\u5b83\u7684”\u524d\u4efb”RNN\u597d\u597d\u6bd4\u4e00\u6bd4\u3002\u8fd9\u4e00\u7ae0\u6211\u4eec\u5c31\u7528\u6700\u751f\u6d3b\u5316\u7684\u6bd4\u65b9\uff0c\u8bb2\u6e05\u695aTransformer\u78be\u538bRNN\u7684\u4e09\u5927\u672c\u8d28\u4f18\u52bf\u3002<\/p>\n\n\n\n

5.1 \u9010\u884c\u8bfb\u4e66vs\u6574\u9875\u626b\u8bfb\uff1a\u5e76\u884c\u80fd\u529b\u7684\u5929\u58e4\u4e4b\u522b<\/strong><\/strong><\/h3>\n\n\n\n

RNN\u7684\u5de5\u4f5c\u65b9\u5f0f\uff0c\u6253\u4e2a\u6700\u5f62\u8c61\u7684\u6bd4\u65b9\uff0c\u5c31\u50cf\u4e00\u4e2a\u4eba\u9010\u884c\u8bfb\u4e66<\/strong>\u3002\u4f60\u5fc5\u987b\u5148\u770b\u5b8c\u7b2c\u4e00\u884c\uff0c\u624d\u80fd\u770b\u7b2c\u4e8c\u884c\uff1b\u5fc5\u987b\u770b\u5b8c\u7b2c\u4e8c\u884c\uff0c\u624d\u80fd\u770b\u7b2c\u4e09\u884c\u3002\u4e3a\u4ec0\u4e48\uff1f\u56e0\u4e3aRNN\u5904\u7406\u4e00\u53e5\u8bdd\u65f6\uff0c\u662f\u4ece\u5de6\u5230\u53f3\u4e00\u4e2a\u8bcd\u4e00\u4e2a\u8bcd\u5730\u5543<\/strong>\uff1a\u62ff\u5230\u7b2c\u4e00\u4e2a\u8bcd\uff0c\u628a\u5b83\u8bb0\u5728\u8111\u5b50\u91cc\uff1b\u518d\u8bfb\u7b2c\u4e8c\u4e2a\u8bcd\uff0c\u7ed3\u5408\u8111\u5b50\u91cc\u7684\u8bb0\u5fc6\u66f4\u65b0\u7406\u89e3\uff1b\u518d\u8bfb\u7b2c\u4e09\u4e2a\u8bcd\uff0c\u518d\u66f4\u65b0\u2026\u2026\u76f4\u5230\u6700\u540e\u4e00\u4e2a\u8bcd\u3002<\/p>\n\n\n\n

\u8fd9\u79cd”\u4e32\u884c”\u5904\u7406\u7684\u81f4\u547d\u95ee\u9898\u5728\u4e8e\uff1a\u540e\u9762\u7684\u8bcd\u5fc5\u987b\u7b49\u524d\u9762\u7684\u8bcd\u5904\u7406\u5b8c\u624d\u80fd\u5f00\u59cb\u7b97<\/strong>\u3002\u4e00\u53e5\u8bdd\u670910\u4e2a\u8bcd\uff0c\u5c31\u5f97\u639210\u6b65\u961f\uff1b\u67091000\u4e2a\u8bcd\uff0c\u5c31\u5f97\u63921000\u6b65\u961f\u3002\u66f4\u7cdf\u7cd5\u7684\u662f\uff0cGPU\u8fd9\u79cd\u5929\u751f\u64c5\u957f”\u540c\u65f6\u7b97\u4e00\u5927\u5806\u4e1c\u897f”\u7684\u8d85\u7ea7\u8ba1\u7b97\u5361\uff0c\u5728RNN\u9762\u524d\u82f1\u96c4\u65e0\u7528\u6b66\u4e4b\u5730\u2014\u2014\u4f60\u8ba9\u5b83\u540c\u65f6\u7b97\u7b2c1\u4e2a\u8bcd\u548c\u7b2c5\u4e2a\u8bcd\uff1f\u4e0d\u884c\uff0c\u7b2c5\u4e2a\u8bcd\u7684\u8ba1\u7b97\u5fc5\u987b\u7b49\u7b2c4\u4e2a\u8bcd\u7684\u7ed3\u679c\uff0c\u7b2c4\u4e2a\u8bcd\u7684\u7ed3\u679c\u53c8\u4f9d\u8d56\u7b2c3\u4e2a\u8bcd\u2026\u2026GPU\u7684\u4e0a\u5343\u4e2a\u8ba1\u7b97\u6838\u5fc3\u53ea\u80fd\u773c\u5df4\u5df4\u5730\u6392\u961f\uff0c\u5927\u90e8\u5206\u65f6\u95f4\u5728\u95f2\u7f6e[9][13]\u3002<\/p>\n\n\n\n

\u90a3Transformer\u5462\uff1f\u5b83\u7684\u5de5\u4f5c\u65b9\u5f0f\u5c31\u50cf\u628a\u6574\u9875\u4e66\u644a\u5f00\u5728\u4f60\u9762\u524d\uff0c\u4f60\u4e00\u773c\u626b\u8fc7\u53bb\uff0c\u6240\u6709\u5b57\u540c\u65f6\u6620\u5165\u773c\u5e18<\/strong>\u3002\u8fd8\u8bb0\u5f97\u81ea\u6ce8\u610f\u529b\u662f\u600e\u4e48\u5de5\u4f5c\u7684\u5417\uff1f\u6bcf\u4e2a\u8bcd\u90fd\u53ef\u4ee5\u540c\u65f6\u53bb”\u770b”\u53e5\u5b50\u91cc\u7684\u5176\u4ed6\u6240\u6709\u8bcd\uff0cQKV\u77e9\u9635\u7684\u8ba1\u7b97\u5168\u90e8\u53ef\u4ee5\u540c\u65f6\u8fdb\u884c\u2014\u2014\u56e0\u4e3a\u6bcf\u4e2a\u8bcd\u7684Query\u3001Key\u3001Value\u90fd\u53ea\u4f9d\u8d56\u5b83\u81ea\u5df1\uff0c\u4e0d\u9700\u8981\u7b49\u522b\u7684\u8bcd\u7b97\u5b8c\u3002\u8fd9\u5c31\u610f\u5473\u7740\uff0c\u4e0d\u7ba1\u4e00\u53e5\u8bdd\u662f10\u4e2a\u8bcd\u8fd8\u662f1000\u4e2a\u8bcd\uff0c\u4ece\u8ba1\u7b97\u6d41\u7a0b\u4e0a\u770b\uff0cTransformer\u51e0\u4e4e\u53ea\u9700\u8981”\u4e00\u6b65”\u5c31\u80fd\u628a\u6240\u6709\u8bcd\u4e4b\u95f4\u7684\u5173\u7cfb\u5168\u90e8\u7b97\u5b8c<\/strong>[3][4]\u3002<\/p>\n\n\n\n

\u8fd9\u79cd\u5dee\u5f02\uff0c\u5728\u89c6\u9891\u91cc\u88ab\u5f62\u8c61\u5730\u6bd4\u55bb\u4e3a”\u4f20\u8bdd\u6e38\u620f”\u548c”\u5168\u5458\u7535\u8bdd\u4f1a\u8bae”\u7684\u533a\u522b[3][4]\u3002RNN\u50cf\u662f10\u4e2a\u4eba\u6392\u6210\u4e00\u6392\u73a9\u4f20\u8bdd\uff0c\u7b2c1\u4e2a\u4eba\u544a\u8bc9\u7b2c2\u4e2a\u4eba\uff0c\u7b2c2\u4e2a\u4eba\u518d\u544a\u8bc9\u7b2c3\u4e2a\u4eba\u2026\u2026\u4fe1\u606f\u5fc5\u987b\u4e00\u4e2a\u63a5\u4e00\u4e2a\u5730\u4f20\u9012\uff0c\u8c01\u4e5f\u4e0d\u80fd\u63d0\u524d\u8bf4\u8bdd\uff1bTransformer\u5219\u50cf\u662f\u8fd910\u4e2a\u4eba\u540c\u65f6\u63a5\u5165\u4e00\u4e2a\u7535\u8bdd\u4f1a\u8bae\uff0c\u6240\u6709\u4eba\u540c\u65f6\u5728\u7ebf\uff0c\u8c01\u60f3\u8ddf\u8c01\u8bf4\u8bdd\u76f4\u63a5\u5f00\u53e3\uff0c\u6839\u672c\u4e0d\u9700\u8981\u7ecf\u8fc7\u4e2d\u95f4\u4eba\u4f20\u8bdd\u3002GPU\u7684\u5e76\u884c\u7b97\u529b\u5728Transformer\u67b6\u6784\u4e0b\u88ab\u5f7b\u5e95”\u5582\u9971”\u2014\u2014\u6bcf\u4e2a\u8bcd\u7684QKV\u8ba1\u7b97\u3001\u6bcf\u4e2a\u6ce8\u610f\u529b\u5934\u7684\u8fd0\u7b97\u3001\u6bcf\u4e2a\u4f4d\u7f6e\u7684\u52a0\u6743\u6c42\u548c\uff0c\u90fd\u53ef\u4ee5\u6254\u5230GPU\u7684\u4e0d\u540c\u6838\u5fc3\u4e0a\u540c\u65f6\u8dd1[13][16]\u3002\u8fd9\u5c31\u662f\u4e3a\u4ec0\u4e48\u5728\u540c\u6837\u7684\u786c\u4ef6\u4e0a\uff0cTransformer\u7684\u8bad\u7ec3\u901f\u5ea6\u53ef\u4ee5\u6bd4RNN\u5feb\u4e0a\u51e0\u5341\u500d\u751a\u81f3\u4e0a\u767e\u500d\uff0c\u4e5f\u6b63\u662f\u4e3a\u4ec0\u4e48\u5b83\u80fd\u5403\u4e0b\u4e92\u8054\u7f51\u4e0a\u6d77\u91cf\u7684\u6587\u672c\u6570\u636e\uff0c\u8bad\u7ec3\u51fa\u53c2\u6570\u91cf\u8fbe\u767e\u4ebf\u3001\u5343\u4ebf\u7ea7\u522b\u7684\u5927\u6a21\u578b\u3002<\/p>\n\n\n\n

5.2 \u4f20\u8bdd\u6e38\u620f\u7684\u8870\u51cf\u9b54\u5492\uff1a\u957f\u8ddd\u79bb\u4f9d\u8d56\u7684\u672c\u8d28\u7a81\u7834<\/strong><\/strong><\/h3>\n\n\n\n

\u5982\u679c\u8bf4\u5e76\u884c\u8ba1\u7b97\u89e3\u51b3\u7684\u662f”\u5feb\u4e0d\u5feb”\u7684\u95ee\u9898\uff0c\u90a3\u957f\u8ddd\u79bb\u4f9d\u8d56\u89e3\u51b3\u7684\u5c31\u662f”\u61c2\u4e0d\u61c2”\u7684\u95ee\u9898\u3002\u4ec0\u4e48\u53eb\u957f\u8ddd\u79bb\u4f9d\u8d56\uff1f\u4e3e\u4e2a\u4f8b\u5b50\uff1a”\u6211\u51fa\u751f\u5728\u6cd5\u56fd\u2026\u2026\uff08\u4e2d\u95f4\u7701\u756530\u53e5\u8bdd\uff09\u2026\u2026\u6240\u4ee5\u6211\u80fd\u8bf4\u4e00\u53e3\u6d41\u5229\u7684____\u3002”\u4f60\u51e0\u4e4e\u80fd\u7acb\u523b\u586b\u4e0a”\u6cd5\u8bed”\uff0c\u56e0\u4e3a\u5f00\u5934\u7684”\u6cd5\u56fd”\u548c\u7ed3\u5c3e\u7684”\u6cd5\u8bed”\u9694\u4e8630\u591a\u53e5\u8bdd\uff0c\u4f46\u4f60\u4f9d\u7136\u80fd\u628a\u5b83\u4eec\u8054\u7cfb\u8d77\u6765\u3002\u8fd9\u5c31\u53eb”\u957f\u8ddd\u79bb\u4f9d\u8d56”\u2014\u2014\u53e5\u5b50\u4e2d\u76f8\u9694\u5f88\u8fdc\u7684\u4e24\u4e2a\u8bcd\uff0c\u5728\u8bed\u4e49\u4e0a\u5176\u5b9e\u662f\u7d27\u5bc6\u76f8\u5173\u7684\u3002<\/p>\n\n\n\n

RNN\u5904\u7406\u8fd9\u79cd\u95ee\u9898\u5c31\u5f88\u5403\u529b\u4e86\u3002\u8fd8\u8bb0\u5f97”\u4f20\u8bdd\u6e38\u620f”\u7684\u6bd4\u55bb\u5417\uff1f10\u4e2a\u4eba\u6392\u6210\u4e00\u6392\u4f20\u8bdd\uff0c\u7b2c1\u4e2a\u4eba\u8bf4”\u6cd5\u56fd”\uff0c\u4f20\u5230\u7b2c2\u4e2a\u4eba\u53ef\u80fd\u53d8\u6210”\u4e00\u4e2a\u6b27\u6d32\u56fd\u5bb6”\uff0c\u4f20\u5230\u7b2c5\u4e2a\u4eba\u53ef\u80fd\u53ea\u5269\u4e0b”\u5916\u56fd”\uff0c\u4f20\u5230\u7b2c10\u4e2a\u4eba\u8033\u6735\u91cc\uff0c\u53ef\u80fd\u65e9\u5c31\u9762\u76ee\u5168\u975e\u4e86\u3002RNN\u7684\u8bb0\u5fc6\u5c31\u662f\u8fd9\u6837\uff1a\u6bcf\u5904\u7406\u4e00\u4e2a\u65b0\u8bcd\uff0c\u5b83\u5c31\u5f97\u628a\u4e4b\u524d\u7684\u8bb0\u5fc6”\u538b\u7f29”\u4e00\u904d\uff0c\u585e\u8fdb\u4e00\u4e2a\u56fa\u5b9a\u5927\u5c0f\u7684\u9690\u85cf\u72b6\u6001\u91cc\u3002\u4fe1\u606f\u4ece\u7b2c1\u4e2a\u8bcd\u4f20\u5230\u7b2c100\u4e2a\u8bcd\uff0c\u8981\u7ecf\u8fc799\u6b21\u538b\u7f29\u548c\u4f20\u9012\uff0c\u6bcf\u4e00\u6b21\u538b\u7f29\u90fd\u4f1a\u4e22\u6389\u4e00\u70b9\u7ec6\u8282\uff0c\u6bcf\u4e00\u6b21\u4f20\u9012\u90fd\u4f1a\u8ba9\u6700\u65e9\u7684\u4fe1\u606f\u8870\u51cf\u4e00\u70b9<\/strong>[3][13]\u3002<\/p>\n\n\n\n

\u8fd9\u5728\u6280\u672f\u4e0a\u53eb”\u68af\u5ea6\u6d88\u5931”\uff1a\u56e0\u4e3aRNN\u7684\u8bad\u7ec3\u4f9d\u8d56\u53cd\u5411\u4f20\u64ad\uff0c\u9519\u8bef\u4fe1\u53f7\u8981\u4ece\u6700\u540e\u4e00\u4e2a\u8bcd\u4e00\u8def\u4f20\u56de\u5230\u7b2c\u4e00\u4e2a\u8bcd\uff0c\u6bcf\u4f20\u4e00\u6b65\u90fd\u8981\u4e58\u4e0a\u4e00\u4e2a\u6743\u91cd\u77e9\u9635\uff0c\u5982\u679c\u8fd9\u4e2a\u6743\u91cd\u7684\u6570\u503c\u5c0f\u4e8e1\uff0c\u4e58\u4e0a\u51e0\u5341\u4e0a\u767e\u6b21\u4e4b\u540e\uff0c\u68af\u5ea6\u5c31\u4f1a\u8d8b\u8fd1\u4e8e0\uff0c\u7b49\u4e8e\u544a\u8bc9\u524d\u9762\u7684\u8bcd”\u4f60\u5bf9\u7ed3\u679c\u6ca1\u4ec0\u4e48\u5f71\u54cd\uff0c\u4e0d\u7528\u6539\u4e86”[13]\u3002\u867d\u7136\u540e\u6765\u4eba\u4eec\u53d1\u660e\u4e86LSTM\u548cGRU\uff0c\u7528”\u95e8\u63a7”\u673a\u5236\u8bd5\u56fe\u7559\u4f4f\u91cd\u8981\u4fe1\u606f\u3001\u9057\u5fd8\u4e0d\u91cd\u8981\u4fe1\u606f\uff0c\u4e00\u5b9a\u7a0b\u5ea6\u4e0a\u7f13\u89e3\u4e86\u8fd9\u4e2a\u95ee\u9898\uff0c\u4f46\u672c\u8d28\u4e0a\u8fd8\u662f”\u4f20\u8bdd\u6e38\u620f”\u2014\u2014\u4fe1\u606f\u4f9d\u7136\u8981\u4e00\u6b65\u4e00\u6b65\u4f20\u9012\uff0c\u9694\u5f97\u592a\u8fdc\u8fd8\u662f\u4f1a\u6a21\u7cca\u3002\u5c31\u50cf\u4e00\u4e2a\u8bb0\u6027\u518d\u597d\u7684\u4eba\uff0c\u8ba9\u4ed6\u590d\u8ff030\u5206\u949f\u524d\u542c\u5230\u7684\u4e00\u53e5\u8bdd\u7684\u7b2c\u4e09\u4e2a\u8bcd\uff0c\u4ed6\u4e5f\u5f88\u96be\u4e00\u5b57\u4e0d\u5dee\u3002<\/p>\n\n\n\n

Transformer\u89e3\u51b3\u957f\u8ddd\u79bb\u4f9d\u8d56\u7684\u65b9\u5f0f\u975e\u5e38”\u66b4\u529b”\u800c\u76f4\u63a5\uff1a\u65e2\u7136\u4f20\u8bdd\u4f1a\u8870\u51cf\uff0c\u90a3\u5c31\u5e72\u8106\u4e0d\u4f20\u8bdd\u4e86\uff0c\u76f4\u63a5\u62c9\u4e00\u6839\u7ebf<\/strong>\u3002\u5728\u81ea\u6ce8\u610f\u529b\u673a\u5236\u4e0b\uff0c\u53e5\u5b50\u91cc\u4efb\u610f\u4e24\u4e2a\u8bcd\u4e4b\u95f4\u90fd\u6709\u4e00\u6761”\u76f4\u8fbe\u901a\u9053”\u2014\u2014\u4e0d\u7ba1\u5b83\u4eec\u9694\u4e8610\u4e2a\u8bcd\u8fd8\u662f1000\u4e2a\u8bcd\uff0c\u7b2c1\u4e2a\u8bcd\u7684Q\u53ef\u4ee5\u76f4\u63a5\u53bb\u548c\u7b2c1000\u4e2a\u8bcd\u7684K\u505a\u70b9\u79ef\uff0c\u8ba1\u7b97\u5b83\u4eec\u4e4b\u95f4\u7684\u5173\u8054\u5ea6\uff0c\u7136\u540e\u76f4\u63a5\u4ece\u7b2c1000\u4e2a\u8bcd\u7684V\u91cc\u53d6\u4fe1\u606f[3][4][13]\u3002\u8fd9\u5c31\u50cf\u5728”\u5168\u5458\u7535\u8bdd\u4f1a\u8bae”\u91cc\uff0c\u5750\u5728\u6700\u8fb9\u4e0a\u7684\u4eba\u60f3\u8ddf\u5750\u5728\u53e6\u4e00\u5934\u7684\u4eba\u8bf4\u8bdd\uff0c\u76f4\u63a5\u558a\u4e00\u58f0\u5c31\u884c\uff0c\u4e0d\u9700\u8981\u7ecf\u8fc7\u4e2d\u95f48\u4e2a\u4eba\u7684\u53e3\u53e3\u76f8\u4f20\uff0c\u4fe1\u606f\u4e00\u6b65\u5230\u4f4d\uff0c\u96f6\u635f\u8017\u3002<\/p>\n\n\n\n

\u90a3\u4e2a\u7ecf\u5178\u7684\u4f8b\u5b50\u2014\u2014”The animal didn’t cross the street because it was too tired”\u4e2d\u7684”it”\u5230\u5e95\u6307\u8c01\u2014\u2014\u5bf9RNN\u6765\u8bf4\uff0c”it”\u53ea\u80fd\u4ece\u7d27\u6328\u7740\u5b83\u524d\u9762\u7684\u51e0\u4e2a\u8bcd\u91cc\u731c\uff0c\u53ef\u80fd\u4f1a\u8bef\u4ee5\u4e3a\u662f”street”\uff1b\u4f46\u5bf9Transformer\u6765\u8bf4\uff0c”it”\u53ef\u4ee5\u76f4\u63a5”\u56de\u5934”\u53bb\u770b\u53e5\u9996\u7684”animal”\uff0c\u76f4\u63a5\u5efa\u7acb\u8054\u7cfb\uff0c\u51c6\u786e\u5224\u65ad”it”\u6307\u4ee3\u7684\u662f\u52a8\u7269[14]\u3002\u8fd9\u79cd”\u4efb\u610f\u4e24\u8bcd\u76f4\u8fde”\u7684\u80fd\u529b\uff0c\u8ba9Transformer\u7406\u89e3\u957f\u6587\u672c\u7684\u80fd\u529b\u8fdc\u8fdc\u8d85\u8fc7RNN\u2014\u2014\u8fd9\u4e5f\u662f\u4e3a\u4ec0\u4e48GPT\u80fd\u8bfb\u5b8c\u51e0\u5343\u5b57\u7684\u6587\u7ae0\u540e\u56de\u7b54\u5173\u4e8e\u5f00\u5934\u7ec6\u8282\u7684\u95ee\u9898\uff0c\u800c\u57fa\u4e8eRNN\u7684\u65e7\u6a21\u578b\u8bfb\u5230\u540e\u9762\u65e9\u5c31\u5fd8\u4e86\u524d\u9762\u8bf4\u4ec0\u4e48\u3002<\/p>\n\n\n\n

5.3 \u6ce8\u610f\u529b\u77e9\u9635vs\u9690\u85cf\u72b6\u6001\uff1a\u4fe1\u606f\u6d41\u52a8\u65b9\u5f0f\u7684\u6839\u672c\u5dee\u5f02<\/strong><\/strong><\/h3>\n\n\n\n

\u8bb2\u5b8c\u4e86\u901f\u5ea6\u548c\u8ddd\u79bb\uff0c\u6211\u4eec\u518d\u5f80\u6df1\u4e00\u5c42\u770b\uff0c\u770b\u770bRNN\u548cTransformer\u5728”\u5982\u4f55\u5b58\u50a8\u548c\u4f20\u9012\u4fe1\u606f”\u8fd9\u4ef6\u4e8b\u4e0a\u7684\u6839\u672c\u67b6\u6784\u5dee\u5f02\u3002<\/p>\n\n\n\n

RNN\u7684\u4fe1\u606f\u662f\u88c5\u5728\u4e00\u4e2a\u53eb”\u9690\u85cf\u72b6\u6001”\uff08hidden state\uff09\u7684\u5c0f\u74f6\u5b50\u91cc\u7684\u3002\u4e0d\u7ba1\u4f60\u8f93\u5165\u7684\u53e5\u5b50\u6709\u591a\u957f\u3001\u4fe1\u606f\u91cf\u6709\u591a\u5927\uff0cRNN\u90fd\u8981\u628a\u6240\u6709\u5185\u5bb9\u538b\u7f29\u8fdb\u8fd9\u4e00\u4e2a\u56fa\u5b9a\u7ef4\u5ea6\u7684\u5411\u91cf\u91cc\u3002\u4f60\u53ef\u4ee5\u628a\u5b83\u60f3\u8c61\u6210\u4e00\u4e2a\u53ea\u80fd\u5199100\u4e2a\u5b57\u7684\u7b14\u8bb0\u672c<\/strong>\uff1a\u8bfb\u7b2c\u4e00\u53e5\u8bdd\u65f6\u4f60\u5199\u4e86\u6ee1\u6ee1\u4e00\u9875\uff0c\u8bfb\u7b2c\u4e8c\u53e5\u8bdd\u65f6\u4f60\u5fc5\u987b\u64e6\u6389\u4e00\u4e9b\u65e7\u5185\u5bb9\u624d\u80fd\u5199\u65b0\u5185\u5bb9\uff0c\u8bfb\u7b2c\u4e09\u53e5\u8bdd\u65f6\u53c8\u5f97\u518d\u64e6\u2026\u2026\u8bfb\u5230\u6700\u540e\uff0c\u7b14\u8bb0\u672c\u4e0a\u6700\u65b0\u7684\u5185\u5bb9\u662f\u6e05\u6670\u7684\uff0c\u4f46\u6700\u65e9\u5199\u7684\u4e1c\u897f\u65e9\u5c31\u88ab\u64e6\u5f97\u5dee\u4e0d\u591a\u4e86\u3002\u8fd9\u4e2a”\u56fa\u5b9a\u5927\u5c0f\u7684\u74f6\u9888”\uff0c\u662fRNN\u957f\u8ddd\u79bb\u80fd\u529b\u5f31\u7684\u6839\u6e90\u2014\u2014\u5b83\u7684\u8bb0\u5fc6\u5bb9\u91cf\u5929\u751f\u5c31\u88ab\u90a3\u4e2a\u9690\u85cf\u72b6\u6001\u7684\u7ef4\u5ea6\u5361\u6b7b\u4e86[13]\u3002<\/p>\n\n\n\n

Transformer\u5219\u5f7b\u5e95\u629b\u5f03\u4e86\u8fd9\u4e2a”\u5c0f\u74f6\u5b50”\uff0c\u53d6\u800c\u4ee3\u4e4b\u7684\u662f\u4e00\u5f20\u5de8\u5927\u7684\u6ce8\u610f\u529b\u77e9\u9635<\/strong>\u3002\u5982\u679cRNN\u7684\u4fe1\u606f\u4f20\u9012\u662f”\u4e00\u6839\u7ba1\u5b50\u6d41\u6c34”\uff0c\u90a3Transformer\u7684\u4fe1\u606f\u4f20\u9012\u5c31\u662f”\u4e00\u5f20\u8718\u86db\u7f51”\u2014\u2014\u6bcf\u4e2a\u8bcd\u548c\u6bcf\u4e2a\u8bcd\u4e4b\u95f4\u90fd\u6709\u4e00\u6839\u4e1d\u7ebf\uff0c\u4e1d\u7ebf\u4e0a\u7684\u6743\u91cd\uff08\u4e5f\u5c31\u662f\u6ce8\u610f\u529b\u5206\u6570\uff09\u4ee3\u8868\u5b83\u4eec\u4e4b\u95f4\u5173\u7cfb\u7684\u5f3a\u5f31[3][9][16]\u3002\u8fd9\u5f20\u7f51\u6709\u591a\u5927\uff1f\u4e00\u4e2a\u957f\u5ea6\u4e3aN\u7684\u53e5\u5b50\uff0c\u6ce8\u610f\u529b\u77e9\u9635\u5c31\u662fN\u00d7N\u7684\u2014\u2014\u4e5f\u5c31\u662f\u8bf4\uff0c\u6bcf\u4e2a\u8bcd\u90fd\u4e13\u95e8\u4e3a\u81ea\u5df1\u548c\u5176\u4ed6\u6240\u6709\u8bcd\u5206\u522b”\u8bb0\u4e86\u4e00\u672c\u8d26”\uff0c\u800c\u4e0d\u662f\u628a\u6240\u6709\u4fe1\u606f\u90fd\u585e\u8fdb\u4e00\u4e2a\u7edf\u4e00\u7684\u5c0f\u74f6\u5b50\u91cc\u3002\u4f60\u53e5\u5b50\u957f\u4e86\uff1f\u6ca1\u5173\u7cfb\uff0c\u77e9\u9635\u8ddf\u7740\u53d8\u5927\uff0c\u6bcf\u65b0\u589e\u4e00\u4e2a\u8bcd\u5c31\u65b0\u589e\u4e00\u884c\u4e00\u5217\uff0c\u4e13\u95e8\u8bb0\u5f55\u5b83\u548c\u6240\u6709\u5df2\u6709\u8bcd\u4e4b\u95f4\u7684\u5173\u7cfb\uff0c\u8001\u8bcd\u7684\u4fe1\u606f\u4e0d\u4f1a\u88ab\u65b0\u8bcd\u6324\u6389\u3002<\/p>\n\n\n\n

\u8fd9\u79cd\u8bbe\u8ba1\u5e26\u6765\u4e86\u4e00\u4e2a\u989d\u5916\u7684\u597d\u5904\uff1a\u53ef\u89e3\u91ca\u6027<\/strong>\u3002RNN\u7684\u9690\u85cf\u72b6\u6001\u662f\u4e00\u4e2a\u9ed1\u76d2\u5b50\u2014\u2014\u4f60\u5f88\u96be\u77e5\u9053\u90a3\u4e2a\u5411\u91cf\u91cc\u7684\u67d0\u4e2a\u6570\u5b57\u5230\u5e95\u4ee3\u8868\u4ec0\u4e48\u542b\u4e49\u3002\u4f46Transformer\u7684\u6ce8\u610f\u529b\u6743\u91cd\u662f\u53ef\u4ee5\u76f4\u63a5\u53ef\u89c6\u5316\u7684\uff1a\u4f60\u628a\u90a3\u4e2aN\u00d7N\u7684\u77e9\u9635\u753b\u6210\u70ed\u529b\u56fe\uff0c\u5c31\u80fd\u6e05\u6e05\u695a\u695a\u5730\u770b\u5230”it”\u8fd9\u4e00\u884c\u5728”animal”\u90a3\u4e00\u5217\u6709\u4e00\u4e2a\u4eae\u6591\uff0c\u8bf4\u660e\u6a21\u578b\u5728\u5904\u7406”it”\u65f6\u786e\u5b9e\u628a\u6ce8\u610f\u529b\u653e\u5230\u4e86”animal”\u4e0a[14]\u3002\u8fd9\u8ba9\u7814\u7a76\u8005\u548c\u5de5\u7a0b\u5e08\u80fd\u591f\u76f4\u89c2\u5730”\u770b\u89c1”\u6a21\u578b\u5728\u60f3\u4ec0\u4e48\uff0c\u8c03\u8bd5\u548c\u6539\u8fdb\u4e5f\u66f4\u6709\u65b9\u5411\u3002<\/p>\n\n\n\n

\u591a\u5934\u6ce8\u610f\u529b\u5219\u8ba9\u8fd9\u5f20”\u8718\u86db\u7f51”\u66f4\u52a0\u7acb\u4f53\u3002RNN\u53ea\u6709\u4e00\u4e2a\u9690\u85cf\u72b6\u6001\uff0c\u76f8\u5f53\u4e8e\u53ea\u6709\u4e00\u79cd”\u770b\u5f85\u53e5\u5b50\u7684\u65b9\u5f0f”\uff1b\u4f46Transformer\u53ef\u4ee5\u67098\u4e2a\u300116\u4e2a\u751a\u81f3\u4e0a\u767e\u4e2a\u6ce8\u610f\u529b\u5934\uff0c\u6bcf\u4e2a\u5934\u90fd\u5728\u8fd9\u5f20\u7f51\u91cc\u7ec7\u81ea\u5df1\u7684\u4e1d\u7ebf\u2014\u2014\u6709\u7684\u5934\u4e13\u95e8\u7ec7”\u6307\u4ee3\u5173\u7cfb”\u7684\u4e1d\u7ebf\uff08\u8c01\u6307\u4ee3\u8c01\uff09\uff0c\u6709\u7684\u5934\u4e13\u95e8\u7ec7”\u8bed\u6cd5\u5173\u7cfb”\u7684\u4e1d\u7ebf\uff08\u8c01\u662f\u4e3b\u8bed\u8c01\u662f\u52a8\u8bcd\uff09\uff0c\u6709\u7684\u5934\u4e13\u95e8\u7ec7”\u8bed\u4e49\u5173\u7cfb”\u7684\u4e1d\u7ebf\uff08\u54ea\u4e9b\u8bcd\u610f\u601d\u76f8\u8fd1\uff09[10][16]\u3002\u8fd9\u5c31\u50cf\u4e00\u4e2a\u4fa6\u63a2\u56e2\u961f\uff0c\u6709\u4eba\u67e5\u52a8\u673a\u3001\u6709\u4eba\u67e5\u65f6\u95f4\u7ebf\u3001\u6709\u4eba\u67e5\u7269\u8bc1\uff0c\u6700\u540e\u628a\u6240\u6709\u4eba\u7684\u7ebf\u7d22\u6c47\u603b\u5728\u4e00\u8d77\uff0c\u5f97\u51fa\u7684\u7ed3\u8bba\u81ea\u7136\u6bd4\u4e00\u4e2a\u4eba\u5355\u5e72\u8981\u51c6\u786e\u5f97\u591a\u3002<\/p>\n\n\n\n

\u5f53\u7136\uff0cTransformer\u4e5f\u4e0d\u662f\u6ca1\u6709\u4ee3\u4ef7\uff1a\u56e0\u4e3a\u8981\u8ba1\u7b97N\u00d7N\u7684\u6ce8\u610f\u529b\u77e9\u9635\uff0c\u5f53\u53e5\u5b50\u7279\u522b\u957f\u7684\u65f6\u5019\uff08\u6bd4\u5982\u4e00\u672c\u4e66\u90a3\u4e48\u957f\uff09\uff0c\u8fd9\u4e2a\u77e9\u9635\u4f1a\u53d8\u5f97\u975e\u5e38\u5de8\u5927\uff0c\u663e\u5b58\u548c\u8ba1\u7b97\u91cf\u4f1a\u6309\u5e73\u65b9\u7ea7\u522b\u589e\u957f\u2014\u2014\u8fd9\u662f\u5b83\u7684”\u963f\u5580\u7409\u65af\u4e4b\u8e35”\uff0c\u4e5f\u662f\u540e\u6765\u5404\u79cd”\u7ebf\u6027\u6ce8\u610f\u529b””\u7a00\u758f\u6ce8\u610f\u529b”\u7b49\u6539\u8fdb\u65b9\u6848\u60f3\u8981\u89e3\u51b3\u7684\u95ee\u9898\u3002\u4f46\u7455\u4e0d\u63a9\u745c\uff0c\u76f8\u6bd4RNN\u5728\u5e76\u884c\u5ea6\u548c\u957f\u8ddd\u79bb\u4f9d\u8d56\u4e0a\u7684\u5148\u5929\u7f3a\u9677\uff0cTransformer\u7528\u6ce8\u610f\u529b\u77e9\u9635\u4ee3\u66ff\u9690\u85cf\u72b6\u6001\u7684\u67b6\u6784\u9769\u547d\uff0c\u7528”\u5168\u5458\u76f4\u8fde”\u4ee3\u66ff”\u4e32\u884c\u4f20\u8bdd”\u7684\u4fe1\u606f\u6d41\u52a8\u65b9\u5f0f\uff0c\u4ece\u6839\u672c\u4e0a\u89e3\u9501\u4e86\u5927\u6a21\u578b\u7684 scalability\u2014\u2014\u8ba9\u6a21\u578b\u8d8a\u505a\u8d8a\u5927\u3001\u6570\u636e\u8d8a\u5403\u8d8a\u591a\u3001\u80fd\u529b\u8d8a\u6765\u8d8a\u5f3a\u6210\u4e3a\u53ef\u80fd[9][16]\u3002\u8fd9\u6b63\u662f\u4e3a\u4ec0\u4e482017\u5e74\u4e4b\u540e\uff0cRNN\u8fc5\u901f\u4eceNLP\u7684\u738b\u5ea7\u4e0a\u9000\u4f4d\uff0c\u800cTransformer\u4e00\u7edf\u5929\u4e0b\uff0c\u6210\u4e3aGPT\u3001BERT\u4ee5\u53ca\u51e0\u4e4e\u6240\u6709\u73b0\u4ee3\u5927\u6a21\u578b\u7684\u6838\u5fc3\u9aa8\u67b6\u3002<\/p>\n\n\n\n

\u7b2c6\u7ae0 \u52a8\u753b\u4e0e\u4e92\u52a8\u8d44\u6e90\u5927\u96c6\u5408\u2014\u2014\u770b\u7740\u73a9\u5c31\u5b66\u4f1a\u4e86<\/strong><\/strong><\/h2>\n\n\n\n

\u524d\u9762\u4e94\u7ae0\u6211\u4eec\u7528\u5927\u91cf\u751f\u6d3b\u5316\u6bd4\u55bb\u628aTransformer\u7684\u6838\u5fc3\u6982\u5ff5\u63b0\u788e\u4e86\u8bb2\uff0c\u4f46\u5149\u9760\u6587\u5b57\u8111\u8865\uff0c\u5f88\u591a\u5c0f\u767d\u5728\u7406\u89e3”QKV\u600e\u4e48\u4ea4\u4e92””\u591a\u5934\u5230\u5e95\u600e\u4e48\u5206\u5934\u770b””\u4f4d\u7f6e\u7f16\u7801\u600e\u4e48\u8d34\u5230\u8bcd\u4e0a”\u65f6\uff0c\u8111\u5b50\u91cc\u8fd8\u662f\u7f3a\u4e00\u5757\u52a8\u6001\u753b\u9762\u3002\u8fd9\u4e00\u7ae0\u6211\u4eec\u628a\u5168\u7f51\u516c\u8ba4\u6700\u9002\u5408\u96f6\u57fa\u7840\u7684\u89c6\u89c9\u5316\u8d44\u6e90\u505a\u4e00\u6b21\u7cfb\u7edf\u7b5b\u9009\uff0c\u6309”\u52a8\u753b\u89c6\u9891\u2192\u56fe\u89e3\u535a\u5ba2\u2192\u7cfb\u7edf\u8bfe\u7a0b”\u4e09\u7c7b\u6574\u7406\uff0c\u5e76\u7ed9\u51fa\u6bcf\u7c7b\u8d44\u6e90\u7684\u6b63\u786e\u6253\u5f00\u65b9\u5f0f\uff0c\u8ba9\u4f60\u770b\u7740\u73a9\u5c31\u80fd\u5b66\u4f1a\u3002<\/p>\n\n\n\n

6.1 \u52a8\u753b\u89c6\u9891\uff1a\u7528\u52a8\u6001\u753b\u9762\u5efa\u7acb\u7b2c\u4e00\u76f4\u89c9<\/strong><\/strong><\/h3>\n\n\n\n

\u96f6\u57fa\u7840\u5b66Transformer\u6700\u5927\u7684\u95e8\u69db\uff0c\u662f\u8111\u5b50\u91cc\u6ca1\u6709”\u7b97\u6cd5\u8dd1\u8d77\u6765\u957f\u4ec0\u4e48\u6837”\u7684\u753b\u9762\u3002\u52a8\u753b\u89c6\u9891\u7684\u6838\u5fc3\u4ef7\u503c\u5c31\u662f\u66ff\u4f60\u628a\u8fd9\u5f20\u753b\u9762\u753b\u51fa\u6765\u2014\u2014\u4f60\u4e0d\u9700\u8981\u5148\u61c2\u4efb\u4f55\u516c\u5f0f\uff0c\u76ef\u7740\u52a8\u753b\u770b\u4e24\u5708\uff0c\u81ea\u6ce8\u610f\u529b\u90a3\u5957”\u5168\u5458\u540c\u65f6\u8fde\u7ebf”\u7684\u611f\u89c9\u5c31\u4f1a\u81ea\u7136\u5efa\u7acb\u3002<\/p>\n\n\n\n

\u7b2c\u4e00\u68af\u961f\u9996\u63a83Blue1Brown\uff083B1B\uff09\u7684GPT\/Transformer\u7cfb\u5217\u52a8\u753b[1]\u3002\u8fd9\u4e2a\u7cfb\u5217\u5728\u6570\u5b66\u79d1\u666e\u5708\u51e0\u4e4e\u662f”\u795e\u7ea7”\u5b58\u5728\uff0c\u4f5c\u8005Grant Sanderson\u64c5\u957f\u7528\u6700\u7b80\u6d01\u7684\u51e0\u4f55\u52a8\u753b\u628a\u62bd\u8c61\u6570\u5b66\u6982\u5ff5\u62cd\u7ed9\u4f60\u770b\u3002\u5728Transformer\u8fd9\u4e00\u96c6\u91cc\uff0c\u4ed6\u4eceTokenization\uff08\u5206\u8bcd\uff09\u4e00\u8def\u8bb2\u5230Embedding\u3001Attention\u3001MLP\uff0c\u6bcf\u4e2a\u73af\u8282\u90fd\u662f\u7cbe\u5fc3\u8bbe\u8ba1\u7684\u52a8\u6001\u53ef\u89c6\u5316\uff1a\u4f60\u4f1a\u770b\u5230\u4e00\u4e2a\u4e2a\u8bcd\u5411\u91cf\u50cf\u5c0f\u65b9\u5757\u4e00\u6837\u6392\u597d\u961f\uff0c\u6ce8\u610f\u529b\u6743\u91cd\u50cf\u5149\u5e26\u4e00\u6837\u5728\u8bcd\u4e0e\u8bcd\u4e4b\u95f4\u6d41\u52a8\uff0c\u591a\u5934\u6ce8\u610f\u529b\u5219\u50cf\u5206\u5c4f\u4e00\u6837\u540c\u65f6\u5c55\u793a\u51e0\u79cd\u4e0d\u540c\u989c\u8272\u7684\u8fde\u63a5\u7ebf\u3002B\u7ad9\u6709\u5b8c\u6574\u4e2d\u6587\u8bd1\u7248\uff0c\u5efa\u8bae\u7b2c\u4e00\u6b21\u770b\u7684\u65f6\u5019\u5173\u6389\u5f39\u5e55\u3001\u4e0d\u8981\u6682\u505c\u3001\u4ece\u5934\u5230\u5c3e\u5b8c\u6574\u770b\u4e00\u904d<\/strong>\uff0c\u76ee\u7684\u4e0d\u662f\u62a0\u7ec6\u8282\uff0c\u800c\u662f\u5148\u8ba9\u5927\u8111\u5bf9”Transformer\u5728\u5e72\u561b”\u7559\u4e0b\u6574\u4f53\u5370\u8c61\uff1b\u7b2c\u4e8c\u904d\u518d\u914d\u5408\u7b2c2-4\u7ae0\u7684\u77e5\u8bc6\u70b9\u56de\u770b\u67d0\u4e2a\u5177\u4f53\u7247\u6bb5\uff0c\u4f1a\u6709\u4e00\u79cd”\u539f\u6765\u52a8\u753b\u91cc\u90a3\u4e2a\u5c0f\u5149\u70b9\u5c31\u662fQK\u70b9\u79ef”\u7684\u987f\u609f\u611f[8]\u3002<\/p>\n\n\n\n

\u5982\u679c\u60f3\u770b\u4e2d\u6587\u672c\u571f\u8bb2\u89e3\uff0c\u6709\u4e24\u652f\u89c6\u9891\u975e\u5e38\u5bf9\u5c0f\u767d\u80c3\u53e3\u3002\u4e00\u652f\u662fBV1PSxYzJEkf”\u5927\u767d\u8bdd\u8bb2\u660e\u767dTransformer\u7684\u6ce8\u610f\u529b\u673a\u5236”[2]\uff0c\u4f5c\u8005\u7528”\u67e5\u5c0f\u7ea2\u8eab\u9ad8”\u8fd9\u79cd\u5e7c\u513f\u56ed\u7ea7\u4f8b\u5b50\u89e3\u91caQKV\uff1a\u4f60\u8111\u5b50\u91cc\u5e26\u7740”\u5c0f\u7ea2\u591a\u9ad8”\u8fd9\u4e2a\u95ee\u9898\uff08Q\uff09\u53bb\u82b1\u540d\u518c\u627e\u5199\u7740”\u5c0f\u7ea2”\u7684\u6807\u7b7e\uff08K\uff09\uff0c\u7ffb\u5230\u90a3\u4e00\u884c\u770b\u5230\u8eab\u9ad8\u6570\u5b57\uff08V\uff09\u2014\u2014\u6574\u4e2a\u81ea\u6ce8\u610f\u529b\u8fc7\u7a0b\u5c31\u662f\u8fd9\u4e2a\u52a8\u4f5c\u5bf9\u6240\u6709\u8bcd\u540c\u65f6\u505a\u4e00\u904d\u3002\u89c6\u9891\u8fd8\u7528”\u5b83\u6307\u4ee3\u82f9\u679c\u8fd8\u662f\u6c34\u679c”\u7684\u771f\u5b9e\u4f8b\u53e5\u6f14\u793a\u6307\u4ee3\u6d88\u89e3\uff0c\u5168\u7a0b\u51e0\u4e4e\u4e0d\u51fa\u73b0\u516c\u5f0f\uff0c\u975e\u5e38\u9002\u5408\u4f5c\u4e3a\u7b2c2\u30013\u7ae0\u7684\u914d\u5957\u5f71\u50cf\u8d44\u6599\u3002\u53e6\u4e00\u652f\u662fBV1TmGD64Edz\u7684\u77ed\u79d1\u666e[15]\uff0c\u4f5c\u8005\u7528”\u5b83\u574f\u4e86\uff0c\u670b\u53cb\u79d2\u56de\u6362\u65b0\u7684”\u8fd9\u79cd\u65e5\u5e38\u5bf9\u8bdd\uff0c\u52a8\u753b\u5c55\u793a\u6bcf\u4e2a\u5b57\u5982\u4f55\u77ac\u95f4”\u770b\u5230”\u6574\u53e5\u8bdd\u6240\u6709\u5b57\u5e76\u81ea\u52a8\u5bf9\u9f50\u6307\u4ee3\u5173\u7cfb\uff0c\u7247\u957f\u53ea\u6709\u51e0\u5206\u949f\uff0c\u9002\u5408\u788e\u7247\u65f6\u95f4\u5237\u3002<\/p>\n\n\n\n

\u60f3\u76f4\u63a5\u770b\u7b97\u6cd5\u672c\u8eab\u600e\u4e48\u8dd1\uff08\u800c\u4e0d\u662f\u6bd4\u55bb\uff09\uff0c\u63a8\u8350Gal Lahat\u5236\u4f5c\u7684”Transformer\u53ef\u89c6\u5316”\u89c6\u9891[6]\u3002\u8fd9\u652f\u89c6\u9891\u4e0d\u8d70\u7c7b\u6bd4\u8def\u7ebf\uff0c\u800c\u662f\u628aQKV\u77e9\u9635\u8fd0\u7b97\u76f4\u63a5\u505a\u6210\u52a8\u753b\uff1a\u4f60\u4f1a\u770b\u5230Q\u77e9\u9635\u7684\u4e00\u884c\u600e\u6837\u626b\u8fc7K\u77e9\u9635\u7684\u6240\u6709\u5217\uff0c\u70b9\u79ef\u7ed3\u679c\u600e\u6837\u53d8\u6210\u70ed\u5ea6\u56fe\uff0csoftmax\u4e4b\u540e\u600e\u6837\u628a\u6743\u91cd\u538b\u56de\u52300-1\u4e4b\u95f4\uff0c\u6700\u540e\u52a0\u6743V\u5f97\u5230\u8f93\u51fa\u3002\u7b2c\u4e00\u6b21\u770b\u53ef\u80fd\u6709\u70b9\u61f5\uff0c\u4f46\u5f53\u4f60\u5df2\u7ecf\u8bfb\u8fc7\u7b2c3\u7ae0\u7684\u56db\u6b65\u6d41\u7a0b\u518d\u56de\u5934\u770b\uff0c\u4f1a\u53d1\u73b0\u5b83\u628a”\u751f\u6210QKV\u2192\u70b9\u79ef\u2192\u7f29\u653e\u2192softmax\u2192\u52a0\u6743\u6c42\u548c”\u6bcf\u4e00\u6b65\u90fd\u62cd\u5f97\u6e05\u6e05\u695a\u695a\uff0c\u662f\u628a\u6587\u5b57\u6d41\u7a0b\u7ffb\u8bd1\u6210\u52a8\u6001\u753b\u9762\u7684\u6700\u4f73\u6865\u6881\u3002<\/p>\n\n\n\n

\u60f3\u4e00\u6b21\u6027\u5bf9\u6bd4RNN\u548cTransformer\u4e3a\u4ec0\u4e48\u5dee\u8fd9\u4e48\u591a\uff0c\u63a8\u8350BV1S3XvBjEtk”RNN VS transformer”[4]\uff0c\u89c6\u9891\u628aRNN\u753b\u6210\u4e00\u4e32\u624b\u62c9\u624b\u4f20\u7eb8\u6761\u7684\u5c0f\u4eba\uff08\u4f20\u8bdd\u6e38\u620f\u3001\u4fe1\u606f\u8d8a\u4f20\u8d8a\u7cca\uff09\uff0c\u628aTransformer\u753b\u6210\u4e00\u5f20\u6240\u6709\u4eba\u540c\u65f6\u901a\u8bdd\u7684\u5927\u5706\u684c\uff08\u7535\u8bdd\u4f1a\u8bae\u3001\u4fe1\u606f\u96f6\u635f\u8017\uff09\uff0c\u52a8\u753b\u5bf9\u6bd4\u6781\u5176\u76f4\u89c2\uff0c\u53ef\u4f5c\u4e3a\u7b2c5\u7ae0\u7684\u914d\u5957\u590d\u4e60\u6750\u6599\u3002<\/p>\n\n\n\n

6.2 \u56fe\u89e3\u535a\u5ba2\u4e0e\u4e92\u52a8\u7f51\u9875\uff1a\u81ea\u5df1\u52a8\u624b\u62d6\u52a8\u6ed1\u5757<\/strong><\/strong><\/h3>\n\n\n\n

\u89c6\u9891\u662f”\u522b\u4eba\u52a8\u7ed9\u4f60\u770b”\uff0c\u56fe\u89e3\u535a\u5ba2\u548c\u4e92\u52a8Demo\u5219\u662f”\u4f60\u81ea\u5df1\u70b9\u7740\u770b”\uff0c\u540e\u8005\u5728\u5efa\u7acb\u6df1\u5ea6\u7406\u89e3\u4e0a\u6709\u4e0d\u53ef\u66ff\u4ee3\u7684\u4ef7\u503c\u2014\u2014\u4f60\u53ef\u4ee5\u505c\u5728\u4efb\u610f\u4e00\u6b65\u3001\u653e\u5927\u4efb\u610f\u4e00\u4e2a\u77e9\u9635\u3001\u751a\u81f3\u62d6\u52a8\u6ed1\u5757\u6539\u53c2\u6570\u89c2\u5bdf\u7ed3\u679c\u53d8\u5316\u3002<\/p>\n\n\n\n

\u8fd9\u4e00\u7c7b\u8d44\u6e90\u91cc\u7ed5\u4e0d\u5f00\u7684”\u795e\u6587”\u662fJay Alammar\u7684\u300aThe Illustrated Transformer\u300b\uff08\u300a\u56fe\u89e3Transformer\u300b\uff09[7]\u3002\u8fd9\u7bc72018\u5e74\u53d1\u5e03\u7684\u535a\u5ba2\u88ab\u5168\u7403\u516c\u8ba4\u4e3aTransformer\u5165\u95e8\u7b2c\u4e00\u6587\uff0c\u51e0\u4e4e\u6240\u6709\u540e\u6765\u7684\u4e2d\u6587\u8bb2\u89e3\u90fd\u5728\u4e0d\u540c\u7a0b\u5ea6\u4e0a\u53c2\u8003\u4e86\u5b83\u3002Jay Alammar\u7684\u505a\u6cd5\u662f\u628a\u8bba\u6587\u91cc\u6bcf\u4e00\u4e2a\u5f20\u91cf\uff08tensor\uff09\u90fd\u753b\u6210\u5e26\u989c\u8272\u7684\u65b9\u5757\uff1a\u8f93\u5165\u662f\u4e00\u6392\u6d45\u7070\u8272\u65b9\u5757\uff0c\u4e58\u4ee5\u6743\u91cd\u77e9\u9635\u53d8\u6210Q\/K\/V\u4e09\u6392\u5f69\u8272\u65b9\u5757\uff0c\u6ce8\u610f\u529b\u6743\u91cd\u662f\u4e00\u5f20\u70ed\u529b\u683c\u5b50\u56fe\uff0c\u591a\u5934\u662f\u628a\u8fd9\u5f20\u56fe\u5207\u6210\u51e0\u4efd\u5e76\u884c\u5904\u7406\u2026\u2026\u6574\u7bc7\u6587\u7ae0\u50cf\u8fde\u73af\u753b\u4e00\u6837\u4ece\u8f93\u5165\u8d70\u5230\u8f93\u51fa\uff0c\u4e00\u4e2a\u516c\u5f0f\u914d\u4e00\u5f20\u56fe\u3001\u4e00\u5f20\u56fe\u914d\u4e00\u6bb5\u5927\u767d\u8bdd\u89e3\u91ca\u3002\u5f3a\u70c8\u5efa\u8bae\u8bfb\u7684\u65f6\u5019\u4e0d\u8981\u8df3\u8fc7\u56fe<\/strong>\u2014\u2014\u5f88\u591a\u4eba\u770b\u6587\u5b57\u89c9\u5f97”\u6211\u61c2\u4e86”\uff0c\u4f46\u771f\u6b63\u8ba9\u6982\u5ff5\u843d\u5730\u7684\u662f\u90a3\u4e9b\u5f69\u8272\u65b9\u5757\u5728\u4f60\u8111\u5b50\u91cc\u7559\u4e0b\u7684\u7a7a\u95f4\u8bb0\u5fc6\u3002\u7f51\u4e0a\u6709\u591a\u4e2a\u4e2d\u6587\u7ffb\u8bd1\u7248\u672c\uff0c\u53ef\u4ee5\u914d\u5408\u82f1\u6587\u539f\u7248\u5bf9\u7167\u770b\uff0c\u907f\u514d\u7ffb\u8bd1\u504f\u5dee\u3002\u8fd9\u7bc7\u535a\u5ba2\u7684\u4e2d\u6587\u8f6c\u8ff0\u7248\u5728\u535a\u5ba2\u56ed\u7b49\u5e73\u53f0\u4e5f\u5e7f\u4e3a\u6d41\u4f20\uff0c\u5e38\u7528”The animal didn’t cross the street because it was too tired”\u4e2dit\u6307\u4ee3animal\u7684\u7ecf\u5178\u6848\u4f8b\u6765\u6f14\u793a\u81ea\u6ce8\u610f\u529b\u7684\u6307\u4ee3\u6d88\u89e3\u80fd\u529b[14]\u3002<\/p>\n\n\n\n

\u5982\u679c\u4f60\u5df2\u7ecf\u770b\u5b8c\u56fe\u89e3\u535a\u5ba2\u60f3”\u4eb2\u624b\u73a9\u4e00\u73a9”\uff0c\u53ef\u4ee5\u641c\u4e00\u4e0b\u57fa\u4e8e\u8fd9\u7bc7\u535a\u5ba2\u884d\u751f\u51fa\u6765\u7684\u5404\u7c7b\u4e92\u52a8Demo\uff08\u901a\u5e38\u5728GitHub\u6216Observable\u5e73\u53f0\u4e0a\u4ee5”Transformer Visualization””Attention Visualizer”\u7b49\u5173\u952e\u8bcd\u51fa\u73b0\uff09\u3002\u8fd9\u7c7b\u4e92\u52a8\u7f51\u9875\u4e00\u822c\u4f1a\u7ed9\u4f60\u4e00\u4e2a\u8f93\u5165\u6846\u8ba9\u4f60\u81ea\u5df1\u6572\u4e00\u53e5\u8bdd\uff0c\u7136\u540e\u5b9e\u65f6\u663e\u793a\u6bcf\u4e2a\u8bcd\u5bf9\u5176\u4ed6\u8bcd\u7684\u6ce8\u610f\u529b\u6743\u91cd\u70ed\u529b\u56fe\u2014\u2014\u4f60\u6572”\u732b\u8ffd\u72d7”\uff0c\u4f1a\u770b\u5230”\u8ffd”\u5b57\u5bf9”\u732b”\u548c”\u72d7”\u90fd\u4eae\u7740\u9ad8\u6743\u91cd\uff1b\u4f60\u6362\u6210”\u72d7\u8ffd\u732b”\uff0c\u4eae\u706f\u4f4d\u7f6e\u5c31\u8c03\u6362\u4e86\u3002\u8fd9\u79cd”\u6539\u4e00\u4e2a\u5b57\u3001\u56fe\u5c31\u53d8”\u7684\u5373\u65f6\u53cd\u9988\uff0c\u662f\u4efb\u4f55\u9759\u6001\u6587\u7ae0\u548c\u89c6\u9891\u90fd\u66ff\u4ee3\u4e0d\u4e86\u7684\uff0c\u80fd\u8ba9\u4f60\u771f\u6b63\u4f53\u4f1a\u5230\u4ec0\u4e48\u53eb”\u81ea\u6ce8\u610f\u529b\u8ba9\u6bcf\u4e2a\u8bcd\u770b\u904d\u5168\u53e5”\u3002\u5efa\u8bae\u5728\u770b\u5b8c\u7b2c2-4\u7ae0\u540e\u82b130\u5206\u949f\u4e13\u95e8\u73a9\u8fd9\u7c7bDemo\uff0c\u968f\u4fbf\u6572\u5404\u79cd\u5947\u602a\u53e5\u5b50\uff08\u6bd4\u5982\u957f\u53e5\u3001\u5e26\u6b67\u4e49\u7684\u6307\u4ee3\u53e5\u3001\u8bed\u5e8f\u98a0\u5012\u53e5\uff09\u89c2\u5bdf\u6ce8\u610f\u529b\u56fe\u600e\u4e48\u53d8\uff0c\u4f1a\u6bd4\u518d\u8bfb\u5341\u7bc7\u6587\u7ae0\u90fd\u7ba1\u7528\u3002<\/p>\n\n\n\n

\u9700\u8981\u63d0\u9192\u7684\u662f\uff0c\u56fe\u89e3\u535a\u5ba2\u6709\u4e00\u4e2a\u5e38\u89c1\u5751\uff1a\u56fe\u91cc\u7684\u65b9\u5757\u5927\u5c0f\u548c\u7bad\u5934\u65b9\u5411\u90fd\u662f\u7b80\u5316\u7684<\/strong>\uff0c\u771f\u5b9e\u6a21\u578b\u7684\u7ef4\u5ea6\u8fdc\u5927\u4e8e\u56fe\u91cc\u753b\u7684\u51e0\u4e2a\u683c\u5b50\u3002\u6240\u4ee5\u8bfb\u56fe\u65f6\u91cd\u70b9\u662f\u7406\u89e3”\u8c01\u548c\u8c01\u53d1\u751f\u4e86\u4ec0\u4e48\u8fd0\u7b97”\uff0c\u4e0d\u8981\u7ea0\u7ed3”\u4e3a\u4ec0\u4e48\u56fe\u91cc\u53ea\u753b\u4e864\u4e2a\u5934\u800c\u8bba\u6587\u91cc\u662f8\u4e2a”\u8fd9\u7c7b\u7ec6\u8282\u2014\u2014\u90a3\u4e9b\u662f\u793a\u610f\uff0c\u4e0d\u662f\u5de5\u7a0b\u84dd\u56fe\u3002<\/p>\n\n\n\n

6.3 \u7cfb\u7edf\u8bfe\u7a0b\u4e0e\u4e00\u7ad9\u5f0f\u8d44\u6e90\uff1a\u60f3\u6df1\u5165\u65f6\u7684\u8fdb\u9636\u901a\u9053<\/strong><\/strong><\/h3>\n\n\n\n

\u5f53\u4f60\u9760\u52a8\u753b\u548c\u56fe\u89e3\u5efa\u7acb\u8d77\u76f4\u89c9\u4e4b\u540e\uff0c\u5982\u679c\u60f3\u628a\u96f6\u6563\u77e5\u8bc6\u70b9\u4e32\u6210\u5b8c\u6574\u4f53\u7cfb\uff0c\u5c31\u9700\u8981\u4e00\u95e8\u7cfb\u7edf\u8bfe\u7a0b\u3002\u8fd9\u4e00\u7c7b\u8d44\u6e90\u7684\u7279\u70b9\u662f\u957f\u3001\u5168\u3001\u6709\u8001\u5e08\u5e26\u7740\u8d70<\/strong>\uff0c\u9002\u5408\u5df2\u7ecf\u4e0d\u6ee1\u8db3\u4e8e”\u770b\u61c2\u6bd4\u55bb”\u3001\u60f3\u771f\u6b63\u4ece\u96f6\u642d\u51fa\u5b8c\u6574\u8ba4\u77e5\u6846\u67b6\u7684\u5b66\u4e60\u8005\u3002<\/p>\n\n\n\n

\u4e2d\u6587\u4e16\u754c\u6700\u88ab\u63a8\u5d07\u7684\u5165\u95e8\u8bfe\u662f\u53f0\u5927\u674e\u5b8f\u6bc5\u6559\u6388\u7684\u300a\u81ea\u6ce8\u610f\u529b\u673a\u5236\u4e0eTransformer\u300b[5]\u3002\u674e\u5b8f\u6bc5\u7684\u8bb2\u8bfe\u98ce\u683c\u5728\u673a\u5668\u5b66\u4e60\u5708\u51fa\u4e86\u540d\u7684”\u542c\u5f97\u61c2”\u2014\u2014\u4ed6\u4f1a\u4ece”\u4e3a\u4ec0\u4e48\u8f93\u5165\u8981\u662f\u4e00\u6392\u5411\u91cf”\u8fd9\u79cd\u6700\u57fa\u7840\u7684\u95ee\u9898\u8bb2\u8d77\uff0c\u987a\u7740BPNN\u2192CNN\u2192RNN\u2192Self-Attention\u7684\u6f14\u8fdb\u8def\u7ebf\u4e00\u8def\u63a8\u8fc7\u6765\uff0c\u8ba9\u4f60\u660e\u767dTransformer\u4e0d\u662f\u5929\u4e0a\u6389\u4e0b\u6765\u7684\u9ed1\u79d1\u6280\uff0c\u800c\u662f\u4e3a\u4e86\u89e3\u51b3\u524d\u51e0\u4ee3\u6a21\u578b”\u770b\u4e0d\u5230\u8fdc\u65b9”\u95ee\u9898\u7684\u5fc5\u7136\u7b54\u6848\u3002\u8bfe\u7a0b\u914d\u5957\u6709\u9ad8\u6e05PPT\uff0c\u6bcf\u4e2a\u516c\u5f0f\u65c1\u8fb9\u90fd\u914d\u4e86\u76f4\u89c9\u89e3\u91ca\uff0c\u8bb2\u591a\u5934\u6ce8\u610f\u529b\u65f6\u4f1a\u73b0\u573a\u753b\u56fe\u6f14\u793a”\u4e0d\u540c\u5934\u5173\u6ce8\u4e0d\u540c\u5173\u7cfb”\uff08\u6709\u7684\u5934\u770b\u8bed\u6cd5\u3001\u6709\u7684\u5934\u770b\u6307\u4ee3\uff09\uff0c\u8bb2\u4f4d\u7f6e\u7f16\u7801\u65f6\u4f1a\u7528”\u7ed9\u5ea7\u4f4d\u7f16\u53f7”\u7684\u6bd4\u55bb\u8bb2\u6e05\u4e3a\u4ec0\u4e48\u9700\u8981\u7ed9\u5929\u751f\u65e0\u5e8f\u7684\u6ce8\u610f\u529b\u673a\u5236\u52a0\u4e0a\u987a\u5e8f\u4fe1\u606f\u3002\u6574\u95e8\u8bfe\u5728B\u7ad9\u53ef\u514d\u8d39\u89c2\u770b\uff0c\u603b\u65f6\u957f\u5927\u7ea6\u51e0\u4e2a\u5c0f\u65f6\uff0c\u63a8\u8350\u7528\u6cd5\u662f\uff1a\u5148\u770b\u5b8c\u672c\u4e66\u7b2c1-5\u7ae0\u5efa\u7acb\u6982\u5ff5\uff0c\u518d\u4ee51.5\u500d\u901f\u5237\u8fd9\u95e8\u8bfe<\/strong>\uff0c\u4f60\u4f1a\u53d1\u73b0\u8001\u5e08\u8bb2\u7684\u6bcf\u4e00\u53e5\u8bdd\u4f60\u90fd\u80fd\u5bf9\u5e94\u5230\u524d\u9762\u8bfb\u8fc7\u7684\u67d0\u4e2a\u6bd4\u55bb\u6216\u56fe\u89e3\uff0c\u5de9\u56fa\u6548\u679c\u6781\u4f73\u3002<\/p>\n\n\n\n

\u5982\u679c\u4f60\u5e0c\u671b\u4e00\u95e8\u8bfe\u8fde\u524d\u7f6e\u77e5\u8bc6\u90fd\u4e00\u5e76\u8986\u76d6\uff08\u6bd4\u5982Softmax\u662f\u4ec0\u4e48\u3001Word2Vec\u600e\u4e48\u505a\u3001Embedding\u600e\u4e48\u6765\u7684\uff09\uff0c\u63a8\u8350B\u7ad9BV1p4KneqEpy”Deepseek\u5e95\u5c42\u7b97\u6cd5\u2014\u2014Transformer\u67b6\u6784\u5165\u95e8\u5230\u7cbe\u901a”[17]\u3002\u8fd9\u95e8\u8bfe\u7684\u7279\u70b9\u662f”\u4e00\u7ad9\u5f0f”\uff1a\u4e0d\u5047\u8bbe\u4f60\u6709\u6df1\u5ea6\u5b66\u4e60\u57fa\u7840\uff0c\u4ece\u5411\u91cf\u3001\u77e9\u9635\u4e58\u6cd5\u8bb2\u8d77\uff0c\u4e00\u8def\u94fa\u5230\u6ce8\u610f\u529b\u3001\u4f4d\u7f6e\u7f16\u7801\u3001\u5b8c\u6574Encoder-Decoder\u7ed3\u6784\uff0c\u9002\u5408\u5b8c\u5168\u6ca1\u63a5\u89e6\u8fc7\u673a\u5668\u5b66\u4e60\u4f46\u613f\u610f\u82b1\u6574\u5757\u65f6\u95f4\u7cfb\u7edf\u5b66\u7684\u5c0f\u767d\u3002\u6ce8\u610f\u8fd9\u7c7b\u957f\u8bfe\u65f6\u8bfe\u7a0b\u5bb9\u6613”\u770b\u5b8c\u5c31\u5fd8”\uff0c\u5efa\u8bae\u8fb9\u770b\u8fb9\u505a\u7b14\u8bb0\uff0c\u6bcf\u770b\u5b8c\u4e00\u4e2a\u77e5\u8bc6\u70b9\u5c31\u5408\u4e0a\u89c6\u9891\u81ea\u5df1\u7528\u5927\u767d\u8bdd\u590d\u8ff0\u4e00\u904d\uff0c\u80fd\u590d\u8ff0\u51fa\u6765\u624d\u7b97\u771f\u61c2\u3002<\/p>\n\n\n\n

\u5df2\u7ecf\u5de5\u4f5c\u7684\u5de5\u7a0b\u5e08\u5982\u679c\u60f3\u66f4\u8fdb\u4e00\u6b65\u770b\u5230\u4ee3\u7801\u5c42\u9762\uff0c\u674e\u6c90\u8001\u5e08\u7684”\u8ddf\u674e\u6c90\u5b66AI”\u7cfb\u5217\uff08CSDN\u548cB\u7ad9\u90fd\u6709\uff09\u662f\u7ecf\u5178\u9009\u62e9[16]\u3002\u674e\u6c90\u4f1a\u5e26\u7740\u4f60\u9010\u884c\u8bfb\u8bba\u6587\u3001\u624b\u5199PyTorch\u5b9e\u73b0\uff0c\u628a\u591a\u5934\u6ce8\u610f\u529b\u4e0eCNN\u591a\u901a\u9053\u505a\u7c7b\u6bd4\uff0c\u89e3\u91ca\u4e3a\u4ec0\u4e48\u591a\u5934\u4e0d\u662f”\u91cd\u590d\u8ba1\u7b97”\u800c\u662f”\u50cfCNN\u591a\u8f93\u51fa\u901a\u9053\u4e00\u6837\u8ba9\u6bcf\u4e2a\u5934\u8bc6\u522b\u4e0d\u540c\u6a21\u5f0f”\u3002\u8fd9\u90e8\u5206\u504f\u5de5\u7a0b\uff0c\u4e0d\u5efa\u8bae\u96f6\u57fa\u7840\u7b2c\u4e00\u5929\u5c31\u51b2<\/strong>\uff0c\u5426\u5219\u4f1a\u88ab\u4ee3\u7801\u7ec6\u8282\u6df9\u6ca1\u800c\u4e22\u5931\u6574\u4f53\u76f4\u89c9\uff1b\u66f4\u5408\u9002\u7684\u65f6\u673a\u662f\u4f60\u5df2\u7ecf\u80fd\u5bf9\u7740\u56fe\u89e3\u8bb2\u6e05Transformer\u5168\u6d41\u7a0b\u4e4b\u540e\uff0c\u518d\u7528\u4ee3\u7801\u8bfe\u628a”\u4f1a\u8bf4”\u5347\u7ea7\u6210”\u4f1a\u5199”\u3002<\/p>\n\n\n\n

\u6700\u540e\u7ed9\u4e00\u4e2a\u8d44\u6e90\u4f7f\u7528\u7684\u901a\u7528\u539f\u5219\uff1a\u52a8\u753b\u5efa\u7acb\u76f4\u89c9\u2192\u56fe\u89e3\u9a8c\u8bc1\u7406\u89e3\u2192\u4e92\u52a8Demo\u4eb2\u624b\u6298\u817e\u2192\u7cfb\u7edf\u8bfe\u7a0b\u4e32\u6210\u4f53\u7cfb<\/strong>\uff0c\u6309\u8fd9\u4e2a\u987a\u5e8f\u6765\uff0c\u6bd4\u4e00\u4e0a\u6765\u5c31\u62b1\u7740\u8bba\u6587\u786c\u5543\u6548\u7387\u9ad8\u5341\u500d\u3002\u8d44\u6e90\u4e0d\u5728\u591a\u800c\u5728\u7cbe\uff0c\u4e0a\u9762\u5217\u7684\u8fd9\u51e0\u4e2a\u90fd\u662f\u7ecf\u8fc7\u65f6\u95f4\u68c0\u9a8c\u3001\u88ab\u65e0\u6570\u5c0f\u767d\u9a8c\u8bc1\u8fc7”\u771f\u7684\u770b\u5f97\u61c2”\u7684\u5185\u5bb9\uff0c\u4f60\u4e0d\u9700\u8981\u518d\u53bb\u641c\u4e00\u767e\u7bc7\u540c\u7c7b\u6587\u7ae0\uff0c\u628a\u8fd9\u51e0\u4efd\u5403\u900f\uff0cTransformer\u5c31\u4e0d\u518d\u662f\u5929\u4e66\u3002<\/p>\n\n\n\n

\u7b2c7\u7ae0 \u4ece\u96f6\u5230\u61c2\u7684\u5b8c\u6574\u5b66\u4e60\u8def\u5f84\u2014\u2014\u6545\u4e8b\u5316\u77e5\u8bc6\u95ed\u73af<\/strong><\/strong><\/h2>\n\n\n\n

7.1 \u6982\u5ff5\u5bf9\u7167\u8868\uff1a\u4e00\u5f20\u8868\u628a\u6240\u6709\u672f\u8bed\u4e32\u8d77\u6765<\/strong><\/strong><\/h3>\n\n\n\n

\u5b66\u5230\u8fd9\u91cc\uff0c\u4f60\u5df2\u7ecf\u63a5\u89e6\u4e86\u81ea\u6ce8\u610f\u529b\u3001QKV\u3001\u591a\u5934\u3001\u4f4d\u7f6e\u7f16\u7801\u3001softmax\u3001\u221ad_k\u3001RNN\u3001\u5e76\u884c\u8ba1\u7b97\u7b49\u4e00\u8fde\u4e32\u672f\u8bed\uff0c\u5f88\u5bb9\u6613\u5728\u8111\u6d77\u4e2d”\u4e92\u76f8\u6253\u67b6”\u3002\u6700\u597d\u7684\u65b9\u5f0f\u4e0d\u662f\u53cd\u590d\u80cc\u8bf5\uff0c\u800c\u662f\u628a\u5b83\u4eec\u6446\u5230\u540c\u4e00\u5f20\u8ba4\u77e5\u5730\u56fe\u4e0a\uff0c\u7528\u4e00\u4e2a\u8d2f\u7a7f\u59cb\u7ec8\u7684\u6545\u4e8b\u4e32\u8d77\u6765\u2014\u2014\u628aTransformer\u60f3\u6210”\u4e00\u573a\u5168\u5458\u5728\u7ebf\u7684\u7535\u8bdd\u4f1a\u8bae”\u3002<\/p>\n\n\n\n

\u67b6\u6784\u6f14\u5316\u5bf9\u7167<\/strong>\uff1aRNN\u50cf”\u4f20\u8bdd\u6e38\u620f”\u2014\u2014\u4fe1\u606f\u4ece\u5de6\u5230\u53f3\u4e00\u4e2a\u4eba\u4f20\u7ed9\u4e0b\u4e00\u4e2a\u4eba\uff0c\u4f20\u5230\u6700\u540e\u5e38\u5e38\u8d70\u6837\uff0c\u800c\u4e14\u5fc5\u987b\u7b49\u524d\u4e00\u4e2a\u4eba\u8bf4\u5b8c\u4e0b\u4e00\u4e2a\u4eba\u624d\u80fd\u5f00\u53e3\uff0c\u65e2\u6162\u53c8\u8bb0\u4e0d\u4f4f\u8fdc\u5904\u7684\u8bdd[3][4][13]\uff1bTransformer\u50cf”\u5168\u5458\u7535\u8bdd\u4f1a\u8bae”\u2014\u2014\u6240\u6709\u4eba\u540c\u65f6\u5728\u7ebf\uff0c\u4efb\u610f\u4e24\u4e2a\u4eba\u53ef\u4ee5\u76f4\u63a5\u5bf9\u8bdd\uff0c\u4fe1\u606f\u96f6\u635f\u8017\uff0c\u8fd8\u80fd\u6240\u6709\u4eba\u4e00\u8d77\u8bf4\uff0c\u901f\u5ea6\u81ea\u7136\u5feb[3][4][9]\u3002\u8fd9\u4e2a\u5bf9\u7167\u662f\u7406\u89e3\u540e\u9762\u6240\u6709\u6982\u5ff5\u7684\u5730\u57fa\u3002<\/p>\n\n\n\n

\u6838\u5fc3\u89d2\u8272\u5bf9\u7167<\/strong>\uff1a\u5728\u8fd9\u573a\u4f1a\u8bae\u91cc\uff0c\u6bcf\u4e00\u4e2a\u8bcd\u90fd\u662f\u4e00\u4f4d\u53c2\u4f1a\u8005\u3002Q\uff08Query\uff09\u662f\u8fd9\u4f4d\u53c2\u4f1a\u8005”\u6b64\u523b\u60f3\u95ee\u4ec0\u4e48\u95ee\u9898”\uff0cK\uff08Key\uff09\u662f\u4ed6”\u6302\u5728\u80f8\u524d\u7684\u8eab\u4efd\u6807\u7b7e”\uff0cV\uff08Value\uff09\u662f\u4ed6”\u771f\u6b63\u8981\u5206\u4eab\u7684\u5185\u5bb9”\u3002\u60f3\u627e\u76f8\u5173\u7684\u4eba\u53d1\u8a00\uff0c\u5c31\u62ff\u81ea\u5df1\u7684Q\u53bb\u548c\u6240\u6709\u4eba\u7684K\u5bf9\u4e00\u5bf9”\u6295\u4e0d\u6295\u7f18”\uff0c\u8fd9\u5c31\u662f\u70b9\u79ef\u76f8\u4f3c\u5ea6\uff1b\u518d\u628a\u5339\u914d\u5ea6\u6362\u7b97\u6210\u767e\u5206\u6bd4\u6743\u91cd\uff08softmax\uff09\uff0c\u6309\u6743\u91cd\u628a\u6bcf\u4e2a\u4eba\u7684V\u52a0\u6743\u6c47\u603b\uff0c\u5c31\u5f97\u5230”\u542c\u5b8c\u4e00\u5708\u610f\u89c1\u4e4b\u540e\u7684\u65b0\u6211”\u2014\u2014\u8fd9\u5c31\u662f\u81ea\u6ce8\u610f\u529b\u7684\u8f93\u51fa[2][11][8]\u3002<\/p>\n\n\n\n

\u5173\u952e\u7ec6\u8282\u5bf9\u7167<\/strong>\uff1a\u9664\u4ee5\u221ad_k\u4e0d\u662f\u968f\u4fbf\u505a\u7684”\u6570\u503c\u7a33\u5b9a”\uff0c\u800c\u662f\u9632\u6b62\u70b9\u79ef\u503c\u592a\u5927\u628asoftmax\u63a8\u5230\u9971\u548c\u533a\u3001\u8ba9\u68af\u5ea6\u53d8\u5f97\u6781\u5c0f\u3001\u6a21\u578b\u5b66\u5f97\u6162[12]\u3002\u591a\u5934\u6ce8\u610f\u529b\u4e0d\u662f”\u540c\u4e00\u9053\u9898\u505a\u516b\u904d”\uff0c\u800c\u662f\u8ba9\u4e0d\u540c\u7684\u5934\u6234\u7740\u4e0d\u540c”\u773c\u955c”\u5206\u5934\u53bb\u770b\u2014\u2014\u6709\u7684\u5934\u4e13\u6293\u4ee3\u8bcd\u6307\u4ee3\uff08it\u6307animal\u8fd8\u662fstreet\uff09\uff0c\u6709\u7684\u5934\u6293\u8bed\u6cd5\u5173\u7cfb\uff0c\u6709\u7684\u5934\u6293\u8bed\u4e49\u5173\u8054\uff0c\u6700\u540e\u628a\u591a\u8def\u89c6\u89d2\u62fc\u8d77\u6765[10][14][16]\u3002\u4f4d\u7f6e\u7f16\u7801\u5219\u662f\u7ed9\u6bcf\u4e2a\u53c2\u4f1a\u8005\u8d34\u4e0a”\u5ea7\u4f4d\u53f7”\uff0c\u56e0\u4e3a\u7535\u8bdd\u4f1a\u8bae\u91cc\u8c01\u5148\u8c01\u540e\u9760\u542c\u58f0\u97f3\u5206\u4e0d\u51fa\u6765\uff0c\u4e0d\u8d34\u6807\u7b7e\u5c31\u5206\u4e0d\u6e05”\u732b\u8ffd\u72d7”\u548c”\u72d7\u8ffd\u732b”[12][10]\u3002\u6700\u540e\u8bb0\u5f97\u5206\u5de5\uff1a\u6ce8\u610f\u529b\u8d1f\u8d23”\u628a\u76f8\u5173\u4fe1\u606f\u8fde\u8d77\u6765”\uff0c\u524d\u9988\u7f51\u7edcFFN\u8d1f\u8d23”\u628a\u8fde\u597d\u7684\u4fe1\u606f\u505a\u4e00\u6b21\u601d\u8003\u53d8\u6362”\uff0c\u4e8c\u8005\u5404\u53f8\u5176\u804c[10]\u3002\u628a\u8fd9\u5f20\u8868\u5370\u5728\u8111\u5b50\u91cc\uff0c\u540e\u9762\u4e0d\u7ba1\u770b\u5230\u4ec0\u4e48BERT\u3001GPT\u3001\u5927\u6a21\u578b\uff0c\u672c\u8d28\u90fd\u662f\u8fd9\u5957\u79ef\u6728\u5728\u5806\u5c42\u6570\u3002<\/p>\n\n\n\n

7.2 \u4ea4\u4e92\u5f0f\u81ea\u6d4b\uff1a\u516b\u9053\u9898\u68c0\u9a8c\u4f60\u662f\u5426\u771f\u61c2<\/strong><\/strong><\/h3>\n\n\n\n

\u5149\u770b\u6982\u5ff5\u5bb9\u6613\u4ea7\u751f”\u6211\u61c2\u4e86”\u7684\u9519\u89c9\u3002\u4e0b\u9762\u516b\u9053\u81ea\u6d4b\u9898\u6309\u96be\u5ea6\u9012\u8fdb\uff0c\u5168\u90e8\u6765\u81ea\u524d\u6587\u8bb2\u89e3\u7684\u6838\u5fc3\u70b9\uff0c\u5efa\u8bae\u5148\u81ea\u5df1\u53e3\u8ff0\u7b54\u6848\uff0c\u518d\u5bf9\u7167\u62ec\u53f7\u91cc\u7684\u8981\u70b9\u590d\u76d8\u3002\u80fd\u987a\u7545\u7b54\u51fa\u516d\u9053\u4ee5\u4e0a\uff0c\u8bf4\u660e\u76f4\u89c9\u5df2\u7ecf\u5efa\u7acb\uff1b\u7b54\u9519\u7684\u9898\u6b63\u597d\u66b4\u9732\u76f2\u533a\uff0c\u56de\u5934\u7ffb\u5bf9\u5e94\u5c0f\u8282\u5373\u53ef\u3002<\/p>\n\n\n\n

\u7b2c\u4e00\u7ec4\uff1a\u5730\u57fa\u9898<\/strong>\u3002\u7b2c1\u9898\uff1a\u4e3a\u4ec0\u4e48RNN\u5904\u7406\u957f\u53e5\u5b50\u4f1a”\u5065\u5fd8”\uff1f\uff08\u8981\u70b9\uff1a\u9010\u8bcd\u4e32\u884c\u4f20\u9012\uff0c\u4fe1\u606f\u50cf\u4f20\u8bdd\u6e38\u620f\u5c42\u5c42\u8870\u51cf\uff0c\u68af\u5ea6\u8fde\u4e58\u6613\u6d88\u5931[3][13]\uff09\u3002\u7b2c2\u9898\uff1aTransformer\u91cc\u4efb\u610f\u4e24\u4e2a\u8bcd\u4e4b\u95f4\u6700\u591a\u9694”\u51e0\u8df3”\u80fd\u4e92\u76f8\u770b\u5230\uff1f\uff08\u8981\u70b9\uff1a\u4e00\u8df3\u2014\u2014\u81ea\u6ce8\u610f\u529b\u77e9\u9635\u76f4\u63a5\u8fde\u7ebf\uff0c\u4e0d\u5b58\u5728\u4e2d\u95f4\u4f20\u8bdd[3][9]\uff09\u3002\u7b2c3\u9898\uff1a\u4e3a\u4ec0\u4e48Transformer\u80fd\u5e76\u884c\u800cRNN\u4e0d\u884c\uff1f\uff08\u8981\u70b9\uff1aRNN\u7b2ct\u6b65\u5fc5\u987b\u7b49t-1\u6b65\u7b97\u5b8c\uff1b\u81ea\u6ce8\u610f\u529b\u5bf9\u6240\u6709\u4f4d\u7f6e\u540c\u65f6\u7b97QK\u1d40V\uff0c\u5e8f\u5217\u7ef4\u5ea6\u5929\u7136\u5e76\u884c[9][13]\uff09\u3002<\/p>\n\n\n\n

\u7b2c\u4e8c\u7ec4\uff1a\u673a\u5236\u9898<\/strong>\u3002\u7b2c4\u9898\uff1aQ\u3001K\u3001V\u5206\u522b\u662f\u8f93\u5165\u81ea\u5e26\u7684\uff0c\u8fd8\u662f\u6a21\u578b\u5b66\u51fa\u6765\u7684\uff1f\uff08\u8981\u70b9\uff1a\u4e0d\u662f\u8f93\u5165\u81ea\u5e26\u6807\u7b7e\uff0c\u662f\u8f93\u5165\u5411\u91cf\u5206\u522b\u4e58\u4e09\u4e2a\u53ef\u5b66\u4e60\u7684\u6295\u5f71\u77e9\u9635Wq\/Wk\/Wv\u5f97\u5230\u7684\u4e09\u79cd”\u89c6\u89d2”[10]\uff09\u3002\u7b2c5\u9898\uff1a\u4e3a\u4ec0\u4e48\u9664\u4ee5\u221ad_k\uff1f\u7528\u81ea\u5df1\u7684\u8bdd\u8bf4\uff0c\u4e0d\u8981\u53ea\u8bf4”\u6570\u503c\u7a33\u5b9a”\u3002\uff08\u8981\u70b9\uff1a\u5f53\u7ef4\u5ea6d_k\u5927\u65f6\uff0c\u70b9\u79ef\u65b9\u5dee\u53d8\u5927\uff0c\u503c\u4f1a\u88ab\u63a8\u5f97\u5f88\u5927\uff0csoftmax\u8f93\u51fa\u8d8b\u8fd1one-hot\u3001\u68af\u5ea6\u6781\u5c0f\uff0c\u8bad\u7ec3\u6781\u6162\uff0c\u9664\u4ee5\u221ad_k\u628a\u65b9\u5dee\u62c9\u56de1[12]\uff09\u3002\u7b2c6\u9898\uff1a\u591a\u5934\u6ce8\u610f\u529b\u91cc\u516b\u4e2a\u5934\u662f\u5728\u505a\u91cd\u590d\u8ba1\u7b97\u5417\uff1f\u5b83\u4eec\u5404\u81ea\u5728\u5e72\u4ec0\u4e48\uff1f\uff08\u8981\u70b9\uff1a\u4e0d\u662f\u91cd\u590d\uff0c\u662f\u628a\u6a21\u578b\u7ef4\u5ea6\u62c6\u5230\u591a\u4e2a\u5b50\u7a7a\u95f4\uff0c\u4e0d\u540c\u5934\u5e76\u884c\u6355\u6349\u4e0d\u540c\u7c7b\u578b\u7684\u5173\u7cfb\u2014\u2014\u6307\u4ee3\u3001\u8bed\u6cd5\u3001\u8bed\u4e49\u3001\u4f4d\u7f6e\u7b49[10][16]\uff09\u3002<\/p>\n\n\n\n

\u7b2c\u4e09\u7ec4\uff1a\u6613\u9519\u9898<\/strong>\u3002\u7b2c7\u9898\uff1a\u65e2\u7136\u81ea\u6ce8\u610f\u529b\u8fd9\u4e48\u5f3a\uff0c\u4e3a\u4ec0\u4e48\u8fd8\u9700\u8981\u4f4d\u7f6e\u7f16\u7801\uff1f\u4e3e\u4e00\u4e2a\u53cd\u4f8b\u8bf4\u660e\u3002\uff08\u8981\u70b9\uff1a\u81ea\u6ce8\u610f\u529b\u672c\u8eab\u5bf9\u8f93\u5165\u987a\u5e8f\u662f”\u7f6e\u6362\u4e0d\u53d8”\u7684\uff0c\u6253\u4e71\u8bcd\u5e8f\u6ce8\u610f\u529b\u516c\u5f0f\u4e0d\u53d8\uff1b\u6ca1\u6709\u4f4d\u7f6e\u7f16\u7801\u65f6\uff0c”\u732b\u8ffd\u72d7”\u548c”\u72d7\u8ffd\u732b”\u4f1a\u88ab\u5f53\u6210\u540c\u4e00\u53e5\u8bdd[10][12]\uff09\u3002\u7b2c8\u9898\uff1a\u5728”The animal didn’t cross the street because it was too tired”\u4e2d\uff0cit\u4e3a\u4ec0\u4e48\u5e94\u8be5\u6307\u5411animal\u800c\u4e0d\u662fstreet\uff1f\u6a21\u578b\u9760\u4ec0\u4e48\u505a\u5230\u8fd9\u4e00\u70b9\uff1f\uff08\u8981\u70b9\uff1a\u9760\u81ea\u6ce8\u610f\u529b\u2014\u2014it\u7684Q\u4f1a\u548canimal\u7684K\u7b97\u51fa\u9ad8\u76f8\u4f3c\u5ea6\u3001\u548cstreet\u7684K\u7b97\u51fa\u4f4e\u76f8\u4f3c\u5ea6\uff0c\u52a0\u6743\u6c42\u548c\u540eit\u7684\u8868\u793a\u4f1a\u66f4\u591a\u6c47\u805aanimal\u7684V\uff0c\u4ece\u800c\u5b8c\u6210\u6307\u4ee3\u6d88\u89e3[14]\uff09\u3002<\/p>\n\n\n\n

\u7b54\u5b8c\u8fd9\u516b\u9898\uff0c\u82e5\u67d0\u9898\u5361\u58f3\uff0c\u5efa\u8bae\u6253\u5f00\u7b2c\u516d\u7ae0\u63d0\u5230\u7684\u52a8\u753b\u8d44\u6e90\u2014\u20143Blue1Brown\u7684Token\u5230\u6ce8\u610f\u529b\u5168\u8fc7\u7a0b\u52a8\u753b[1]\u3001Jay Alammar\u7684\u56fe\u89e3\u535a\u5ba2[7]\u3001Gal Lahat\u76f4\u63a5\u52a8\u753b\u5c55\u793aQKV\u4ea4\u4e92\u7684\u89c6\u9891[6]\u2014\u2014\u628a\u5361\u58f3\u7684\u90a3\u4e00\u6b65”\u770b”\u4e00\u904d\uff0c\u76f4\u89c9\u8865\u56de\u6765\u6bd4\u6b7b\u8bb0\u786c\u80cc\u6709\u6548\u5f97\u591a\u3002\u60f3\u7cfb\u7edf\u518d\u8fc7\u4e00\u904d\uff0c\u53ef\u4ee5\u8ddf\u4e00\u8282\u674e\u5b8f\u6bc5\u7684\u8bfe[5]\u6216\u4e00\u7ad9\u5f0f\u5165\u95e8\u8bfe\u7a0b[17]\u505a\u6536\u5c3e\u3002<\/p>\n\n\n\n

7.3 \u6545\u4e8b\u5316\u56de\u987e\uff1a\u4ece\u4e00\u53e5\u8bdd\u5230\u6574\u4e2aTransformer<\/strong><\/strong><\/h3>\n\n\n\n

\u8ba9\u6211\u4eec\u7528\u4e00\u4e2a\u5b8c\u6574\u7684\u5c0f\u6545\u4e8b\uff0c\u628a\u524d\u516d\u7ae0\u6240\u6709\u6982\u5ff5\u4e00\u6b21\u6027\u4e32\u8d77\u6765\uff0c\u4f5c\u4e3a\u4f60\u4eca\u540e\u56de\u5fc6Transformer\u65f6\u7684”\u951a\u70b9\u573a\u666f”\u3002\u5047\u8bbe\u6a21\u578b\u8bfb\u5230\u8fd9\u6837\u4e00\u53e5\u8bdd\uff1a”\u5c0f\u660e\u628a\u82f9\u679c\u9012\u7ed9\u5c0f\u7ea2\uff0c\u56e0\u4e3a\u5979\u997f\u4e86\u3002”\u6211\u4eec\u4e00\u6b65\u6b65\u770bTransformer\u662f\u600e\u4e48\u8bfb\u61c2\u5b83\u7684\u3002<\/p>\n\n\n\n

\u7b2c\u4e00\u6b65\uff1a\u8fdb\u573a\u4e0e\u8d34\u6807\u7b7e<\/strong>\u3002\u53e5\u5b50\u88ab\u5207\u6210\u4e00\u4e2a\u4e2a\u8bcd\uff08token\uff09\uff1a\u5c0f\u660e\u3001\u628a\u3001\u82f9\u679c\u3001\u9012\u3001\u7ed9\u3001\u5c0f\u7ea2\u3001\u56e0\u4e3a\u3001\u5979\u3001\u997f\u3001\u4e86\u3002\u6bcf\u4e2a\u8bcd\u5148\u88ab\u53d8\u6210\u4e00\u4e2a\u5411\u91cf\uff08embedding\uff09\uff0c\u76f8\u5f53\u4e8e\u9886\u5230\u4e00\u5f20\u5199\u7740\u81ea\u8eab\u8bed\u4e49\u7684\u540d\u7247\u3002\u4f46\u5149\u6709\u540d\u7247\u8fd8\u4e0d\u591f\u2014\u2014\u5982\u679c\u5927\u5bb6\u968f\u4fbf\u5750\uff0c\u6a21\u578b\u8ba4\u4e0d\u51fa\u8c01\u5148\u8c01\u540e\uff0c”\u5c0f\u660e\u9012\u7ed9\u5c0f\u7ea2”\u548c”\u5c0f\u7ea2\u9012\u7ed9\u5c0f\u660e”\u5c31\u5206\u4e0d\u6e05\uff0c\u4e8e\u662f\u5de5\u4f5c\u4eba\u5458\u7ed9\u6bcf\u4e2a\u4eba\u53d1\u4e86\u4e00\u4e2a\u5ea7\u4f4d\u53f7\uff0c\u8fd9\u5c31\u662f\u4f4d\u7f6e\u7f16\u7801[10][12]\u3002\u81f3\u6b64\uff0c\u6bcf\u4f4d\u53c2\u4f1a\u8005\u5e26\u7740”\u8bed\u4e49\u540d\u7247+\u5ea7\u4f4d\u53f7”\u8d70\u8fdb\u4f1a\u573a\u3002<\/p>\n\n\n\n

\u7b2c\u4e8c\u6b65\uff1a\u4e00\u8f6eQKV\u63d0\u95ee<\/strong>\u3002\u4f1a\u4e0a\u8ba8\u8bba”\u5979\u5230\u5e95\u6307\u8c01”\u3002”\u5979”\u8fd9\u4e2a\u8bcd\u5fc3\u91cc\u5e26\u7740\u4e00\u4e2a\u95ee\u9898Q\uff1a”\u6211\u662f\u4e00\u4f4d\u5973\u6027\u3001\u800c\u4e14\u5904\u4e8e’\u997f\u4e86’\u8fd9\u4e2a\u8bed\u5883\u91cc\uff0c\u524d\u9762\u8c01\u6700\u53ef\u80fd\u662f\u6211\uff1f”\u5b83\u62ac\u773c\u626b\u8fc7\u5168\u573a\uff1a\u5c0f\u660e\u7684K\u4e0a\u5199\u7740”\u7537\u6027\u4eba\u540d”\uff0c\u82f9\u679c\u7684K\u4e0a\u5199\u7740”\u6c34\u679c”\uff0c\u5c0f\u7ea2\u7684K\u4e0a\u5199\u7740”\u5973\u6027\u4eba\u540d”\u2014\u2014Q\u548c\u5c0f\u7ea2\u7684K\u4e00\u62cd\u5373\u5408\uff0c\u70b9\u79ef\u5f97\u5206\u5f88\u9ad8\uff1b\u548c\u82f9\u679c\u3001\u5c0f\u660e\u7684\u5f97\u5206\u8f83\u4f4e\u3002\u6240\u6709\u5f97\u5206\u9664\u4ee5\u221ad_k\uff0c\u9632\u6b62\u8c01\u55d3\u95e8\u592a\u5927\u628a\u5176\u4ed6\u4eba\u538b\u6ca1[12]\uff0c\u518d\u7ecf\u8fc7softmax\u6362\u7b97\u6210\u767e\u5206\u6bd4\uff1a\u5c0f\u7ea2\u536085%\uff0c\u5c0f\u660e\u536010%\uff0c\u5176\u4ed6\u8bcd\u5360\u5269\u4e0b\u76845%\u3002\u6700\u540e\u6309\u8fd9\u4e2a\u6743\u91cd\u628a\u5927\u5bb6\u7684V\u52a0\u6743\u6c47\u603b\uff0c”\u5979”\u542c\u5b8c\u4e00\u5708\u4e4b\u540e\uff0c\u81ea\u5df1\u7684\u65b0\u8868\u793a\u91cc\u5c31\u88c5\u8fdb\u4e86”\u5c0f\u7ea2”\u7684\u5927\u90e8\u5206\u4fe1\u606f\u2014\u2014\u6307\u4ee3\u6d88\u89e3\u5b8c\u6210[14][2]\u3002<\/p>\n\n\n\n

\u7b2c\u4e09\u6b65\uff1a\u591a\u526f\u773c\u955c\u540c\u65f6\u770b<\/strong>\u3002\u5f53\u7136\u4e00\u53e5\u8bdd\u91cc\u4e0d\u53ea\u6709”\u5979\u6307\u8c01”\u8fd9\u4e00\u4ef6\u4e8b\u3002\u6709\u7684\u5934\u6234\u7740”\u8bed\u6cd5\u773c\u955c”\u5728\u6293”\u628a”\u5b57\u53e5\u7ed3\u6784\uff0c\u6709\u7684\u5934\u6234\u7740”\u60c5\u611f\u773c\u955c”\u5728\u611f\u53d7”\u997f”\u662f\u8d1f\u9762\u72b6\u6001\uff0c\u6709\u7684\u5934\u6234\u7740”\u4f4d\u7f6e\u773c\u955c”\u5728\u5173\u6ce8”\u56e0\u4e3a”\u524d\u540e\u7684\u56e0\u679c\u5173\u7cfb\u2014\u2014\u516b\u526f\u773c\u955c\uff08\u591a\u5934\uff09\u5404\u770b\u5404\u7684\uff0c\u6700\u540e\u628a\u516b\u4e2a\u89c6\u89d2\u62fc\u8d77\u6765[10][16]\u3002\u62fc\u63a5\u4e4b\u540e\u518d\u8fc7\u4e00\u4e2a\u524d\u9988\u7f51\u7edcFFN\uff0c\u628a”\u8fde\u597d\u7684\u4fe1\u606f”\u505a\u4e00\u6b21\u975e\u7ebf\u6027\u601d\u8003\uff0c\u76f8\u5f53\u4e8e\u6bcf\u4e2a\u4eba\u542c\u5b8c\u610f\u89c1\u540e\u81ea\u5df1\u518d\u6d88\u5316\u4e00\u4e0b[10]\u3002\u8fd9\u6837\u4e00\u5c42”\u81ea\u6ce8\u610f\u529b+FFN”\u5c31\u8d70\u5b8c\u4e86\uff0c\u7f16\u7801\u5668\u91cc\u901a\u5e38\u58066\u5c42\uff0c\u4e00\u5c42\u4e00\u5c42\u5730\u628a\u7406\u89e3\u505a\u6df1\u3002<\/p>\n\n\n\n

\u7b2c\u56db\u6b65\uff1a\u4e3a\u4ec0\u4e48\u6bd4\u4f20\u8bdd\u6e38\u620f\u5f3a<\/strong>\u3002\u56de\u60f3RNN\uff0c\u5b83\u8981\u4ece”\u5c0f\u660e”\u5f00\u59cb\u4e00\u4e2a\u8bcd\u4e00\u4e2a\u8bcd\u4f20\u8bdd\uff0c\u4f20\u5230”\u5979”\u7684\u65f6\u5019\uff0c”\u5c0f\u7ea2”\u7684\u4fe1\u606f\u5df2\u7ecf\u8fc7\u4e86\u4e03\u516b\u5f20\u5634\uff0c\u53ef\u80fd\u65e9\u5c31\u4f20\u6210\u4e86”\u82f9\u679c”\uff1b\u800c\u4e14\u5fc5\u987b\u7b49\u524d\u4e00\u4e2a\u8bcd\u8bf4\u5b8c\u4e0b\u4e00\u4e2a\u8bcd\u624d\u80fd\u5f00\u53e3\uff0cGPU\u518d\u5f3a\u4e5f\u6ca1\u6cd5\u52a0\u901f\u3002Transformer\u7684\u7535\u8bdd\u4f1a\u8bae\u91cc\uff0c”\u5979”\u4e00\u8fdb\u573a\u5c31\u80fd\u76f4\u63a5\u548c”\u5c0f\u7ea2”\u8fde\u7ebf\uff0c\u4e00\u6b21\u901a\u8bdd\u89e3\u51b3\uff0c\u4fe1\u606f\u96f6\u635f\u8017\uff1b\u6240\u6709\u4eba\u540c\u65f6\u5f00\u53e3\uff0cGPU\u7684\u51e0\u5343\u4e2a\u6838\u5fc3\u4e00\u8d77\u5e72\u6d3b\uff0c\u901f\u5ea6\u98de\u8d77[3][4][9][13]\u3002\u8fd9\u5c31\u662f\u4e3a\u4ec0\u4e482017\u5e74\u8c37\u6b4c\u90a3\u7bc7\u300aAttention Is All You Need\u300b\u4e00\u51fa[9][14]\uff0c\u6574\u4e2aNLP\u4e43\u81f3AI\u4e16\u754c\u90fd\u88ab\u6539\u5199\u2014\u2014GPT\u3001BERT\u3001ChatGPT\u4ee5\u53ca\u6240\u6709\u4f60\u80fd\u53eb\u51fa\u540d\u5b57\u7684\u5927\u6a21\u578b\uff0c\u9aa8\u67b6\u90fd\u662f\u8fd9\u573a”\u5e26\u5ea7\u4f4d\u53f7\u3001\u6234\u591a\u526f\u773c\u955c\u3001\u5168\u5458\u540c\u65f6\u5728\u7ebf”\u7684\u7535\u8bdd\u4f1a\u8bae\u3002<\/p>\n\n\n\n

\u6545\u4e8b\u8bb2\u5b8c\uff0c\u4f60\u5df2\u7ecf\u4e0d\u662f”\u542c\u8bf4\u8fc7Transformer”\u7684\u95e8\u5916\u6c49\uff0c\u800c\u662f\u80fd\u7528\u4e00\u4e2a\u5b8c\u6574\u5fc3\u667a\u6a21\u578b\u590d\u8ff0\u5b83\u5982\u4f55\u5de5\u4f5c\u7684\u5165\u95e8\u8005\u3002\u63a5\u4e0b\u6765\u5982\u679c\u60f3\u7ee7\u7eed\u6df1\u5165\uff0c\u8def\u5f84\u4e5f\u5f88\u6e05\u6670\uff1a\u7528\u7b2c\u516d\u7ae0\u7684\u52a8\u753b\u8d44\u6e90\u53cd\u590d\u770b\u5efa\u7acb\u808c\u8089\u8bb0\u5fc6[1][6][7]\uff0c\u6311\u674e\u5b8f\u6bc5\u6216Deepseek\u90a3\u5957\u7cfb\u7edf\u8bfe\u7a0b\u8865\u7ec6\u8282[5][17]\uff0c\u518d\u6253\u5f00\u4e00\u4efd\u6700\u5c0f\u5b9e\u73b0\u4ee3\u7801\u4eb2\u624b\u8dd1\u4e00\u6b21QKV\u7684\u56db\u6b65\u8ba1\u7b97[8]\uff0c\u4f60\u5c31\u80fd\u7a33\u7a33\u8de8\u8fc7”\u770b\u61c2\u8bba\u6587\u516c\u5f0f”\u90a3\u9053\u95e8\u69db\uff0c\u8fdb\u5165\u5927\u6a21\u578b\u7684\u771f\u6b63\u5927\u95e8\u3002<\/p>\n\n\n\n

\u53c2\u8003\u8d44\u6599<\/strong><\/strong><\/h2>\n\n\n\n

[1] 3Blue1Brown. \u52a8\u753b\u8bb2\u89e3GPT\/Transformer\u7cfb\u5217\u89c6\u9891[EB\/OL]. Bilibili, 2024-05. https:\/\/www.bilibili.com<\/u><\/a><\/p>\n\n\n\n

[2] B\u7ad9UP\u4e3b. \u5927\u767d\u8bdd\u8bb2\u660e\u767dTransformer\u7684\u6ce8\u610f\u529b\u673a\u5236[EB\/OL]. Bilibili, 2025. BV1PSxYzJEkf<\/p>\n\n\n\n

[3] B\u7ad9UP\u4e3b. RNN vs CNN vs Transformer\u5bf9\u6bd4\u8bb2\u89e3[EB\/OL]. Bilibili, 2025. BV1q7Kj64EcD<\/p>\n\n\n\n

[4] B\u7ad9UP\u4e3b. RNN VS transformer \u4e00\u6761\u89c6\u9891\u641e\u61c2\u4e24\u4e2a\u673a\u5236[EB\/OL]. Bilibili, 2025. BV1S3XvBjEtk<\/p>\n\n\n\n

[5] \u674e\u5b8f\u6bc5. \u81ea\u6ce8\u610f\u529b\u673a\u5236\u4e0eTransformer\u8bfe\u7a0b[EB\/OL]. Bilibili, 2024. BV1oWbFetEfE<\/p>\n\n\n\n

[6] Gal Lahat. Transformer\u53ef\u89c6\u5316\uff1aAttention QKV\u5728\u505a\u4ec0\u4e48[EB\/OL]. Bilibili, 2025. BV1Uj4f6SE7M<\/p>\n\n\n\n

[7] Jay Alammar. The Illustrated Transformer[EB\/OL]. jalammar.github.io, 2018.<\/p>\n\n\n\n

[8] CSDN\u535a\u5ba2. \u4e00\u6587\u641e\u61c2AI\u5927\u6a21\u578bAttention\u673a\u5236\u96f6\u57fa\u7840\u5165\u95e8[EB\/OL]. CSDN, 2025-10-21.<\/p>\n\n\n\n

[9] \u5fae\u4fe1\u516c\u4f17\u53f7. Attention is all you need \u8bba\u6587\u89e3\u6790[EB\/OL]. \u5fae\u4fe1, 2022-12-13.<\/p>\n\n\n\n

[10] CSDN\u535a\u5ba2. \u81ea\u6ce8\u610f\u529b\u673a\u5236\u5230\u5e95\u5728”\u770b”\u4ec0\u4e48[EB\/OL]. CSDN, 2026-06-22.<\/p>\n\n\n\n

[11] B\u7ad9UP\u4e3b. 15\u5206\u949f\u8ba4\u8bc6\u6ce8\u610f\u529b\u00b7\u591a\u5934\u6ce8\u610f\u529b\u6570\u5b66\u539f\u7406\u8be6\u89e3[EB\/OL]. Bilibili, 2025. BV17x8jzvEm6<\/p>\n\n\n\n

[12] \u674e\u91d1\u6d2a. \u6df1\u5ea6\u5b66\u4e60\u79c1\u623f\u83dc\uff1a\u8ddf\u7740\u6848\u4f8b\u5b66TensorFlow[M]. \u5317\u4eac:\u7535\u5b50\u5de5\u4e1a\u51fa\u7248\u793e, 2019.<\/p>\n\n\n\n

[13] \u673a\u5668\u5b66\u4e60\u7814\u7a76\u516c\u4f17\u53f7. \u901a\u900f!Transformer vs RNN\u6838\u5fc3\u533a\u522b[EB\/OL]. \u5fae\u4fe1\u516c\u4f17\u53f7, 2025-12-30.<\/p>\n\n\n\n

[14] \u6280\u672f\u535a\u5ba2. \u591a\u5934\u81ea\u6ce8\u610f\u529b\u6a21\u578b\u4e4bTransformer\u8be6\u89e3[EB\/OL]. \u535a\u5ba2\u56ed, 2023-04-12.<\/p>\n\n\n\n

[15] B\u7ad9UP\u4e3b. Transformer\u81ea\u6ce8\u610f\u529b\u673a\u5236[EB\/OL]. Bilibili, 2025. BV1TmGD64Edz<\/p>\n\n\n\n

[16] CSDN\u535a\u5ba2. Transformer\u8ddf\u674e\u6c90\u5b66AI+3Blue1Brown\u76f4\u89c2\u7406\u89e3[EB\/OL]. CSDN, 2025-06-01.<\/p>\n\n\n\n

[17] B\u7ad9UP\u4e3b. Deepseek\u5e95\u5c42\u7b97\u6cd5\u2014\u2014Transformer\u67b6\u6784\u5165\u95e8\u5230\u7cbe\u901a[EB\/OL]. Bilibili, 2025. BV1p4KneqEpy<\/p>\n","protected":false},"excerpt":{"rendered":"

Transformer\u662f2017\u5e74\u8c37\u6b4c\u5728\u8bba\u6587\u300aAttention Is All You Need\u300b\u4e2d\u63d0\u51fa\u7684\u795e\u7ecf […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[30],"tags":[],"class_list":["post-2684","post","type-post","status-publish","format-standard","hentry","category-30"],"_links":{"self":[{"href":"http:\/\/blog.xtaa.cn\/index.php\/wp-json\/wp\/v2\/posts\/2684","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/blog.xtaa.cn\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/blog.xtaa.cn\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/blog.xtaa.cn\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/blog.xtaa.cn\/index.php\/wp-json\/wp\/v2\/comments?post=2684"}],"version-history":[{"count":11,"href":"http:\/\/blog.xtaa.cn\/index.php\/wp-json\/wp\/v2\/posts\/2684\/revisions"}],"predecessor-version":[{"id":2704,"href":"http:\/\/blog.xtaa.cn\/index.php\/wp-json\/wp\/v2\/posts\/2684\/revisions\/2704"}],"wp:attachment":[{"href":"http:\/\/blog.xtaa.cn\/index.php\/wp-json\/wp\/v2\/media?parent=2684"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/blog.xtaa.cn\/index.php\/wp-json\/wp\/v2\/categories?post=2684"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/blog.xtaa.cn\/index.php\/wp-json\/wp\/v2\/tags?post=2684"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}