输入现代白话或古文，找到意思接近的古诗
举例
输入 你看那长江的水从天上来
1: 李白 [鼓吹曲辞将进酒]
君不见黄河之水天上来，奔流到海不复回．．．
2: 马之纯 [新亭其二]
新亭见说在山头，看见江河衮衮流．．．
3: 释善果 [偈其五]
苏州有，常州有，吸尽西江只一口．．．
输入：忆长安
1: 宋祁 [农阁]
...看云记巫峡，望日省长安。...
2: 徐凝 [寄白司马]
...争遣江州白司马，五年风景忆长安。
3: 崔涂 [春晚怀进士韦澹]
...二年春怅望，不似在长安。
简介
- 简单来说就是用预训练的语言模型来得到诗句的向量，放到scaNN里面索引，然后输入查询语句也得到向量并查询scaNN的最近邻居。
- Based on guwenBERT (an ancient chinese pre-trained RoBERTa language model), HF Transformers (model inference), and Google sanNN (approximate nearest neighbor search)
- We fetch chinese poems from this chinese-poetry github project, and divide to sentence pieces
- Converted to simplified Chinese input using chinese-converter package, please skip if you prefer traditional Chinese
- Use the last layer hidden output as embedding to balance quality and memory constraints, literature recommends last 4 layers but cannot afford memory
- Note the colab runs successfully with high RAM GPU colab instance (paid class, Tesla P100 GPU with 16G GPU ram, 2 core CPU with 24G ram). If you encourter with OOM issue, reduce the
SAMPLE_SIZEvalue will help, or consider to upgrade to paid colab class ($9.99 per month with awesome GPU!!). - Colab takes about 1 hour to fully load/transform/initialize, after that, the inference should take <100ms per call.