Explore CodeRabbit for PR reviews 🐰
Closed this issue · 5 comments
Description
As a 1-person engineering team, I'm the bottleneck for reviewing PRs and getting code merged. Often times, we need some initial feedback on a PR that can be the first touch point before I come in.
Acceptance Criteria
We should explore CodeRabbit throughly to see if its a feasible/trustworthy option to help with code reviews. Any decision/learnings should be documented in this thread!
Additional Context
Some alternatives include What The Diff and Codium.
i think that's interesting , would love to work on that!
Awesome, I'll assign it to you! I think the main thing I'm looking for hear is a simple proof-of-concept and just documenting any learnings from it!
@ramiAbdou I am not assigned to this, but just was interested in helping out on the project so I started doing some research since I am in the area of ML and these were my finds:
"You can opt out of data storage." ~ CodeRabbit -
- Code downloaded during the review process is stored temporarily and deleted after the review processing is complete.
- However, CodeRabbit enhances future reviews for us by storing the embeddings of the learnings generated from chat conversations. It also integrates embeddings from issues in workflow systems (Liner, Jira, GitHub/GitLab issues) to assess the impact of code changes on existing issues
- Choosing to opt out will limit the personalization (and therefore performance) of the reviews.
Mostly paid service after trial however, Open Source Projects get pro for free.
FluxNinja was acquired by CodeRabbit, which is a startup that provides a platform for building scalable generative AI applications. It fuels the generative AI suggestion part of CodeRabbit.
Also thought this was important in understanding CodeRabbit's Flow:
I guess my biggest question is if it is not storing our data(we choose to opt out due to privacy concerns), how is it training itself to do better? This is a different area of AI but most components of ML need previous states/info to do better like neural networks or LSTMs. It seems like a great tool, but personally for me, I am wondering how this tool will adapt to our codebase and be more efficient without storing our code commits. It stores conversations, which could help it improve though.
When weighing it against other AI suggestors like GitHub CoPilot, it says CodeRabbit is better since copilot "may not be equipped to auto-generate meaningful code for moderately complex applications."
I do see its usefulness; it is just privacy-wise, I am wondering how flexible you would like to be for commits.
Thank you for the research you did here, @JustinhSE! Thank you for your note on privacy, it's certainly important to understand how they are protecting us and our IP. Given that they delete the code after the review processing is completed, I feel okay with it!
That being said, I still need to be convinced on the value of CodeRabbit (it sounds good, but I need to see it in action on our codebase). I think a sample PR would be what gives me the confidence in saying yes/no to it!
I am comfortable as well since they discard it. @ramiAbdou No problem though, enjoyed doing it 🫡.To your point, honestly throw up some sample code of a degenerated feature or an idea that is no longer in use and see what it does.
I would like to see the comparisons of its effectiveness compared to throwing it into Microsoft Copilot or some AI code analyzer. Also, something that just popped in my head was maybe measuring how many "convos" you would have to have with coderabbit to see if it is effective or not as a measure of ROI.