symflower/eval-dev-quality

Deal with dependencies requested by LLMs

Opened this issue · 2 comments

Assume an LLM responds with something like

	package com.eval;

	import org.junit.jupiter.api.Test;
	import org.mockito.InMockito;

	import static org.junit.jupiter.api.Assertions.*;

	class PlainTest {

	    @Test
	    void testPlain() {
	        InMockito.when(Plain::plain).thenReturn(null);
	        Plain.plain();
	        InMockito.verify(Plain::plain);
	    }
	}

The test file needs mockito to run, and we need to make sure that this is available when executing the tests.

Response from custom-fireworks_accounts_fireworks_models_qwen2-72b-instruct

What's also bad is that we just go mod tidy all dependencies for Go... so LLMs can use whatever they want, but for Java everything except Java is forbidden.

llama-3-70b-instruct responded with tests using hamcrest

	import org.junit.jupiter.api.Test;
	import static org.junit.jupiter.api.Assertions.*;
	import static org.hamcrest.CoreMatchers.is;
	import static org.hamcrest.CoreMatchers.equalTo;
	import static org.hamcrest.MatcherAssert.assertThat;