[Umbrella Issue] Multilingual Support

[shaoyijia]

At present, we do not have immediate plans to introduce multilingual support for our deployed web research preview. However, we are strong advocates for making AI technology more inclusive and accessible to speakers of various languages.

As part of this commitment, we want to support the community in adapting STORM for use in other languages. This issue serves as a space for discussion, sharing experiences, and collaboration on efforts to expand STORM's multilingual capabilities.

We encourage anyone interested in this effort to share their attempts, insights, or challenges here. For example, we have already seen initial success in adding support for Arabic (#169). Team members will be actively involved in the discussion to provide support.

[zhoucheng89]

Title: Article Outline Disordered After Modifying WriteLeadSection for Chinese Output

Description:

To enable Storm to generate articles in Chinese, we modified the prompt in the WriteLeadSection method of the ArticleGenerationModule, requesting the output to be in Chinese. After making this change, the generated initial article outline became disorganized and was not assembled according to the specified outline sequence.

Steps to Reproduce:

1. Modify the prompt in WriteLeadSection to request Chinese output.
2. Generate an article using the modified module.
3. Observe the structure of the generated article outline.

Expected Behavior:

• The article outline should be generated in Chinese and assembled according to the predefined outline sequence.

Actual Behavior:

• The generated article outline is disordered and does not follow the intended sequence.

Additional Information:

• The issue began after changing the prompt for Chinese output.
• This problem may affect other non-English language outputs as well.

Request:

• Investigate the cause of the disordered outline when the prompt is set to Chinese.
• Provide a fix or guidance on how to maintain the correct outline sequence with Chinese output.

[Yucheng-Jiang]

@zhoucheng89 would you please share the full system log directory as a zip file?

[zhoucheng89]

@zhoucheng89would you please share the full system log directory as a zip file?

@Yucheng-Jiang please check this zip

finding_key_players_in_complex_networks.zip

[shaoyijia]

Thank you! I will look into this during the weekend.

[shaoyijia]

I just checked out the issue. Given Steps to Reproduce include "Modify the prompt in WriteLeadSection to request Chinese output", I think I need to do more experiment on my side to see how to better support non-English languages instead of directly debugging your provided log.

I will follow up in this thread once I have progress.

[zhoucheng89]

@shaoyijia ok,thanks. Waiting for your good news

[shaoyijia]

Hi @zhoucheng89 , sorry for the late response. I finally got time to look into this issue today. Here are the changes I made to make STORM support Chinese. (Note: This is not a very well-polished version but for demonstration-purpose only. I recommend run more experiments on your side)

In general, to make STORM support non-English languages, you need to confirm or modify the following things:

1. The retrieval module needs to be able to retrieve information in the target language.
   • For example, in my experiment, I change YouRM to make sure it can search Chinese websites and produce Chinese snippets.
   • If you look at the raw_search_results.json in your provided zip file, all the snippets are still in English. There is no way for STORM to generate the correct final article in Chinese, since STORM is a knowledge curation system grounding on the retrieved information.
2. The query generated by STORM needs to be in the target language. This can make it easier for the retrieval module to retrieve information in the target language (basically, the language of the query and the language of the results are the same).
   • To achieve this, you can require the simulated Wikipedia Writer to ask questions in Chinese and the simulated Topic Expert to break down queries and answer questions in Chinese. See my change for example.
3. The outline needs to be generated in the target language.
   • See my change for example.
4. The final article needs to be generated in the target language. To do so,
   • You need to change the embedding model to a model that process the target language; otherwise, the information table for each section would be wrong. For example, I use paraphrase-multilingual-MiniLM-L12-v2 in my example (see my change here).
   • You need to modify the prompt for generating the section. See my change here.
5. It's also suggested to change ArticleTextProcessing to handle corner cases in the target language.

Hope my answer provides you with pointers to modify STORM for your use case. If you are willing to polish my example and provide a STORM (Chinese) version, we are happy to link it in our README.md - I believe this contribution would help many people.

[zhoucheng89]

Hi @shaoyijia,I am very pleased to see your message. I saw the code you submitted in the dev-chinese branch, and I will improve the Chinese version of the story in the case you submitted in the future. At the same time, I am currently reading the source code of Storm and have debugged it several times locally. However, the word count of the article I run is always 3400 words, which cannot meet the requirements of a long academic paper. I first tried to return more snippets in the RM stage, and before AnswerQuestion, I concatenated all the snippets into the required Info in the prompt words. Before generating the outline, I recalled more similar snippets through paraphrase-multilingual-MILM-L12-v2, but these methods did not exceed 3400 words in the paper, and the final article was relatively empty. The chapter order of the article was inconsistent with the chapter order in the outline. May I know which steps I can modify to achieve the final goal? Improve the number of final output text books and maintain consistency between the order of article chapters and the outline.

[shaoyijia]

Hi @zhoucheng89 , thanks for testing it out and looking into the codebase.

However, the word count of the article I run is always 3400 words

Are you running with do_polish_article set as True? If this is set as True, after the article is generated, STORM will do a round of global refinement. This may lead to a long article being truncated due to the output length limit of the language model you use (please also check what's the value of max_tokens for the language model you use for each step.

Before generating the outline, I recalled more similar snippets through paraphrase-multilingual-MILM-L12-v2

This seems to be problematic. The embedding model is only used when STORM extends the outline into the final article. I don't think you shall call the embedding model before generating the outline.

The chapter order of the article was inconsistent with the chapter order in the outline.

Is the order for first-level heading (those headings marked with "#") correct or not? As elaborated here, STORM only forces the first-level heading to follow the planned outline and don't apply requirement for second/third/...-level heading to allow flexibility for article generation. You can change its behavior by modifying ArticleGenerationModule.

However, if the order for first-level heading does not follow the planned outline, this is not as expected and we can help look into it if you provide more details on what command you run and what results you get.

[zhoucheng89]

@shaoyijia Can you add me as a contributor，because I have discovered a bug but am unable to submit code.
[SerperRM Bug]
valid_url_to_snippets.get(url, {}) get none value.
call strip function will be fix it. just like valid_url_to_snippets.get(url.strip("\'"), {})

The result of running Storm last time was very hollow because my code was not updated. Now that I have updated to the latest code, I am planning to run it again. I am glad that you have introduced the co story mechanism, which will help users intervene in a timely manner before paper generation. I believe the biggest difference between dialogue mode and collaboration mode is the addition of a moderator and users. The moderator can guide users to intervene, which will promptly correct the confusing answers generated during the roundtable discussion process. This mechanism is great, and I will read the code of the collaboration mechanism in a timely manner and try to support it in Chinese.

[shaoyijia]

Thank you! For contribution, please follow the steps below:

1. Click "Fork" on https://github.com/stanford-oval/storm
2. Clone the forked repository. The forked repository is under your account so you have full access to it.
3. Follow CONTRIBUTING.md to install pre-commit hook which will take care of the python code format.
4. Make edits on your forked repository and commit the change.
5. Create a PR to contribute your change to our main branch.

[zhoucheng89]

@shaoyijia I want to modify the Chinese version of SerperRM. I need to modify the search API. I modified the code on the dev Chinese branch after forking. Am I submitting a pull request directly from the dev Chinese branch after forking to the dev Chinese branch of Storm?

[shaoyijia]

Hi @zhoucheng89 , you can also fork this repo and develop the Chinese STORM in your repo so that you can have full control. If everything is functioning, we are happy to link to your repo on our README so that people can find Chinese STORM if needed. Does this sound good to you?

[zhoucheng89]

@shaoyijia Are you suggesting that after I fork, I can directly modify the code in my dev-chinese branch without the need to initiate a pull request? And will you continue to maintain the dev-chinese branch of the original Storm repository?

[shaoyijia]

@zhoucheng89 Yes. Once you feel it's stable, you can ping me to take a look, so I can consider mentioning it in our README. Currently, I don't have bandwidth myself to develop dev-chinese branch.
Also, if your contribution can be customized to other languages (e.g., a multilingual plugin/mixin class), we can consider merging it directly to main as a major feature update.