Help with German zero shot
abasse opened this issue · 16 comments
I would like to run Sahajtomar/German_Zeroshot (https://huggingface.co/Sahajtomar/German_Zeroshot) model in spago.
The import was successful:
./huggingface-importer --model=Sahajtomar/German_Zeroshot --repo=./models
-> BERT has been converted successfully!
Can I now run the model with Bart server(as I believe supports the zero shot, not the Bart server)?
I receive:
bassea@AP15557 spago % ./bart-server server --repo=./models --model=Sahajtomar/German_Zeroshot --tls-disable
Start loading pre-trained model from "models/Sahajtomar/German_Zeroshot"
[1/2] Loading configuration... ok
panic: bart: unsupported architecture BertForSequenceClassification
goroutine 1 [running]:
github.com/nlpodyssey/spago/pkg/nlp/transformers/bart/loader.Load(0xc000038660, 0x21, 0x2, 0xc000038660, 0x21, 0x492f960)
/Users/bassea/go/src/spago/pkg/nlp/transformers/bart/loader/loader.go:43 +0x819
github.com/nlpodyssey/spago/cmd/bart/app.newServerCommandActionFor.func1(0xc00022b740, 0x0, 0x0)
/Users/bassea/go/src/spago/cmd/bart/app/server.go:106 +0x105
github.com/urfave/cli/v2.(*Command).Run(0xc000222ea0, 0xc00022b440, 0x0, 0x0)
/Users/bassea/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/command.go:163 +0x4e0
github.com/urfave/cli/v2.(*App).RunContext(0xc0000351e0, 0x4ae0aa0, 0xc000036068, 0xc0000320a0, 0x5, 0x5, 0x0, 0x0)
/Users/bassea/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/app.go:313 +0x814
github.com/urfave/cli/v2.(*App).Run(...)
/Users/bassea/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/app.go:224
main.main()
/Users/bassea/go/src/spago/cmd/bart/main.go:15 +0x72
Hi @abasse
The German_Zeroshot uses GBERT Large (German BERT) as base model.
Therefore, when you try to use it with the BART server, it gives you that error.
I think it's not hard to implement zero-shot classification with BERT (I imagine it works just like BART).
I will try as soon as I have time if you are interested.
Thanks, do you have any recommendations for a zero shot BART model ?
@abasse do you mean a zero-shot BART model supporting German?
A zero-shot BART model in English or German that I can use for testing. I tried fascebook/bart-large-mnli, but got a runtime (runtime error: index out of range [-1]) in the console when I tried to call http://127.0.0.1:1987/classify-nli
I'm sorry I'm pretty new to this, so any help is much appreciated :)
I thought this was documented... but nope!
In general, I use the valhalla/distilbart-mnli-12-3 model since it has good performance in both terms of efficiency and accuracy.
You can have a look at this repository in the meantime that I'm updating BART's README to use the API. The zero-shot classification is also included in the various demos.
If I can help further, let me know!
I'm running into the same error with the model. Here is the trace
bassea@AP15557 spago % ./bart-server server --repo=./models --model=valhalla/distilbart-mnli-12-3 --tls-disable
Start loading pre-trained model from "models/valhalla/distilbart-mnli-12-3"
[1/2] Loading configuration... ok
[2/2] Loading model weights... ok
Start non-TLS gRPC server listening on 0.0.0.0:1976.
Start non-TLS HTTP server listening on 0.0.0.0:1987.
2021/07/02 16:26:51 runtime error: index out of range [-1]
goroutine 83620 [running]:
runtime/debug.Stack(0x40e699c, 0xc0001a2050, 0x2)
/usr/local/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
/usr/local/go/src/runtime/debug/stack.go:16 +0x22
github.com/nlpodyssey/spago/pkg/utils/httphandlers.recoveryHandler.log(0x4ace600, 0xc000066000, 0x0, 0x0, 0x1, 0xc1a63030f0, 0x1, 0x1)
/Users/bassea/go/src/spago/pkg/utils/httphandlers/
...
This is the call that creates the error:
curl --location --request GET 'http://127.0.0.1:1987/classify-nli' \
--header 'Content-Type: application/json' \
--data-raw '{"text": "I have a problem with my iphone that needs to be resolved asap!!", "labels":"urgent, not urgent, phone, tablet, computer","multi-class": true}'
Any idea, what's wrong?
@abasse Please try the following curl and let me know how it goes.
curl -k -d '{"text": "I have a problem with my iphone that needs to be resolved asap!!", "possible_labels":["urgent", "not urgent", "phone", "tablet", "computer"],"multi_class": true}' -H "Content-Type: application/json" "http://127.0.0.1:1987/classify-nli"@matteo-grella Thanks! It's working.
By the way, I am using a Apple M1 and the results are not what they should be...
I'm still on Intel. Interesting that ARM shows different results. Any ideas why that is happening?
I'm investigating, and I'll open a dedicated issue for that. Did you get the expected result in the Intel architecture?
Yes, the Intel results look ok so far. It's not fast, a request takes about 10-15 seconds on my MacBook Pro i7.
Ops.. It should take less than a second! What are your CPU details (Ghz, cache) and your RAM?
3,3 GHz Dual-Core Intel Core i7, 16 GB 2133 MHz LPDDR3
@jjviana any ideas?
On my 2.3 GHZ, 8-core Intel i9 it also takes a little less than one second. Maybe the number of cores is the issue?