Streaming complex object fast in custom component makes bug
Closed this issue · 5 comments
MobisParkHeekang commented
Describe the bug
I define a complex object like below:
class TextSection(GradioModel):
section_type: Literal["text"]
section_name: Optional[str]
section_state: Optional[Literal["initial", "running", "complete"]]
section_id: str
feedback: Optional[bool]
section_content: str
class Bubble(GradioModel):
role: Literal["assistant", "user", "system"]
sections: List[TextSection]
class ChatbotData(GradioModel):
bubbles: List[Bubble]
status: Literal["initial", "running", "complete"]
class GradioData(GradioRootModel):
root: ChatbotDataWhen I stream data with yield, errors emerge.
def run(query, chat):
chat = {
"bubbles": [
{
"role": "user",
"sections": [
{
"section_type": "text",
"section_name": null,
"section_state": "complete",
"section_id": "b1152ee1-1349-11ef-be91-8f639724379a",
"feedback": null,
"section_content": ""
}
]
},
(assistant_bubble := {
"role": "assistant",
"sections": [
{
"section_type": "text",
"section_name": null,
"section_state": "running",
"section_id": "b11535eb-1349-11ef-ba87-8f639724379a",
"feedback": null,
"section_content": ""
}
]
})
],
"status": "running"
}
yield chat
text = "Lorem ipsum dolor sit amet"
for c in text:
assistant_bubble["sections"][0]["section_content"] += c
yield chat
#time.sleep(0.1)
assistant_bubble["sections"][0]["section_status"] = "complete"
chat["status"] = "complete"
yield chatSSE data suddenly changes from below:
{
"msg": "process_generating",
"event_id": "2c9694b94b054f76a254d05e143e7f6f",
"output": {
"data": [
{
"bubbles": [
{
"role": "user",
"sections": [
{
"section_type": "text",
"section_name": null,
"section_state": "complete",
"section_id": "b1152ee1-1349-11ef-be91-8f639724379a",
"feedback": null,
"section_content": ""
}
]
},
{
"role": "assistant",
"sections": [
{
"section_type": "text",
"section_name": null,
"section_state": "running",
"section_id": "b11535eb-1349-11ef-ba87-8f639724379a",
"feedback": null,
"section_content": ""
}
]
}
],
"status": "running"
}
],
"is_generating": true,
"duration": 0.0014843940734863281,
"average_duration": 0.0003377993901570638
},
"success": true
}to below:
{
"msg": "process_generating",
"event_id": "2c9694b94b054f76a254d05e143e7f6f",
"output": {
"data": [
[
[
"replace",
[
"bubbles",
0,
"sections",
0,
"section_id"
],
"b1155e6f-1349-11ef-ba0d-8f639724379a"
],
[
"replace",
[
"bubbles",
1,
"sections",
0,
"section_id"
],
"b11561a3-1349-11ef-a2a4-8f639724379a"
],
[
"append",
[
"bubbles",
1,
"sections",
0,
"section_content"
],
"L"
]
]
],
"is_generating": true,
"duration": 0.0003814697265625,
"average_duration": 0.0003392081106862714
},
"success": true
}However, if I put time.sleep()(currently commented out), errors do not emerge.
Have you searched existing issues? 🔎
- I have searched and found no existing issues
Reproduction
types
class TextSection(GradioModel):
section_state: Optional[Literal["initial", "running", "complete"]]
section_content: str
class Bubble(GradioModel):
role: Literal["assistant", "user", "system"]
sections: List[TextSection]
class ChatbotData(GradioModel):
bubbles: List[Bubble]
status: Literal["initial", "running", "complete"]
class GradioData(GradioRootModel):
root: ChatbotDatastreaming function
def run(query, chat):
chat = {
"bubbles": [
{
"role": "user",
"sections": [
{
"section_type": "text",
"section_content": query
}
]
},
(assistant_bubble := {
"role": "assistant",
"sections": [
{
"section_type": "text",
"section_content": ""
}
]
})
],
"status": "running"
}
yield chat
text = "Lorem ipsum dolor sit amet"
for c in text:
assistant_bubble["sections"][0]["section_content"] += c
yield chat
#time.sleep(0.1)
assistant_bubble["sections"][0]["section_status"] = "complete"
chat["status"] = "complete"
yield chatScreenshot
No response
Logs
No response
System Info
Gradio Environment Information:
------------------------------
Operating System: Linux
gradio version: 4.25.0
gradio_client version: 0.15.0
------------------------------------------------
gradio dependencies in your environment:
aiofiles: 23.2.1
altair: 5.2.0
fastapi: 0.110.0
ffmpy: 0.3.2
gradio-client==0.15.0 is not installed.
httpx: 0.27.0
huggingface-hub: 0.20.3
importlib-resources: 6.1.2
jinja2: 3.1.2
markupsafe: 2.1.3
matplotlib: 3.8.3
numpy: 1.26.4
orjson: 3.9.15
packaging: 23.1
pandas: 2.2.0
pillow: 9.5.0
pydantic: 2.6.1
pydub: 0.25.1
python-multipart: 0.0.9
pyyaml: 6.0.1
ruff: 0.2.2
semantic-version: 2.10.0
tomlkit==0.12.0 is not installed.
typer: 0.9.0
typing-extensions: 4.8.0
uvicorn: 0.27.1
authlib; extra == 'oauth' is not installed.
itsdangerous; extra == 'oauth' is not installed.
gradio_client dependencies in your environment:
fsspec: 2023.10.0
httpx: 0.27.0
huggingface-hub: 0.20.3
packaging: 23.1
typing-extensions: 4.8.0
websockets: 11.0.3Severity
Blocking usage of gradio
freddyaboulton commented
Hi @MobisParkHeekang ! What exactly is the error that's happening? It's expected that the data format sent from the server to change because we will only send the diff. Can you please share the custom component you're developing?
MobisParkHeekang commented
Thank you for your reply. Unfortunately, the component is on the company's private server, so it cannot be shared. However, your expectation sounds really plausible. Can you please tell me in which file the diff calculation and sending occur?
abidlabs commented
Hi @MobisParkHeekang you may want to check this out:
Line 1792 in 8f46556
MobisParkHeekang commented
Can you please tell me in which js(or svelte) file ‘rebuild’ value from diffs sent from FastAPI server?
freddyaboulton commented
The js client will rebuild the value based on the diff:
gradio/client/js/src/utils/submit.ts
Line 597 in 35905c5