JSON mode not returning the correct output

[joshpopelka20]

I'm using NousResearch/Meta-Llama-3-8B-Instruct with JSON output. I added the system message template as specified in the docs:

{"role": "system", "content": "You are a helpful assistant that only answers in JSON. Here's the json schema you must adhere to:\n<schema>\n{{" + pydantic_schema + "}}\n<schema>\n"},

This is my schema:

class Classification(BaseModel):
    text: str
    classification: str

    class Config:
        json_schema_extra = {
            "additionalProperties": False
        }

# serialize pydantic model into json schema
pydantic_schema = Classification.schema_json()

My issue is that the output doesn't return just the JSON output. It returns an additional chat message:

'{"text": "3 has been determined to be evolving", "classification": "REPORTABLE"}\n\n{"text": "3 has been determined to be evolving", "classification": "REPORTABLE"}\nThe sentence in the context that most closely matches the given sentence is: "3 has been determined to be evolving" with a classification of "REPORTABLE".\nJSON Response: {"text": "3 has been determined to be evolving", "classification": "REPORTABLE"}\nNote: The JSON response provided is already in the required format. No further action is needed.\nJSON Response: {"text": "3 has been determined to be evolving", "classification": "REPORTABLE"}\nThe sentence in the context that most closely matches the given sentence is: "3 has been determined to be evolving" with a classification of "REPORTABLE".\nJSON Response: {"text": "3 has been determined to be evolving", "classification": "REPORTABLE"}\nNote: The JSON response provided is already in the required format. No further action is needed.\nJSON Response: {"text": "3 has been determined to be evolving", "classification": "REPORTABLE"}\nThe sentence in the context that most closely matches'

How do I get it to only output a message like this:

{"text": "3 has been determined to be evolving", "classification": "REPORTABLE"}

[teknium1]

cc @interstellarninja

[interstellarninja]

He's using Meta's model NousResearch/Meta-Llama-3-8B-Instruct, not fine-tuned with our format 😅

[teknium1]

I'm using NousResearch/Meta-Llama-3-8B-Instruct with JSON output. I added the system message template as specified in the docs:

{"role": "system", "content": "You are a helpful assistant that only answers in JSON. Here's the json schema you must adhere to:\n<schema>\n{{" + pydantic_schema + "}}\n<schema>\n"},

This is my schema:

class Classification(BaseModel):
    text: str
    classification: str

    class Config:
        json_schema_extra = {
            "additionalProperties": False
        }

# serialize pydantic model into json schema
pydantic_schema = Classification.schema_json()

My issue is that the output doesn't return just the JSON output. It returns an additional chat message:

'{"text": "3 has been determined to be evolving", "classification": "REPORTABLE"}\n\n{"text": "3 has been determined to be evolving", "classification": "REPORTABLE"}\nThe sentence in the context that most closely matches the given sentence is: "3 has been determined to be evolving" with a classification of "REPORTABLE".\nJSON Response: {"text": "3 has been determined to be evolving", "classification": "REPORTABLE"}\nNote: The JSON response provided is already in the required format. No further action is needed.\nJSON Response: {"text": "3 has been determined to be evolving", "classification": "REPORTABLE"}\nThe sentence in the context that most closely matches the given sentence is: "3 has been determined to be evolving" with a classification of "REPORTABLE".\nJSON Response: {"text": "3 has been determined to be evolving", "classification": "REPORTABLE"}\nNote: The JSON response provided is already in the required format. No further action is needed.\nJSON Response: {"text": "3 has been determined to be evolving", "classification": "REPORTABLE"}\nThe sentence in the context that most closely matches'

How do I get it to only output a message like this:

{"text": "3 has been determined to be evolving", "classification": "REPORTABLE"}

Please use NousResearch/Hermes-2-Pro-Llama-3-8B

[joshpopelka20]

Is there plans to create a hermes model for Llama-3.1-8B?