softonic/laravel-intelligent-scraper

Specified key was too long; max key length is 3072

Closed this issue · 4 comments

I get Base table or view not found: 1146 Table 'database.configurations' doesn't exist .

I know it contain two migration file but I do not know whether I need to copy these file to my project migration directory or we can import/create these table from vendor directory, I do not find regarding these in documentation. I am new to Laravel.

I am able to migrate it but I receive error

SQLSTATE[42000]: Syntax error or access violation: 1071 Specified key was too long; max key length is 3072 bytes (SQL: alter table scraped_datasets add primary key scraped_datasets_url_primary(url))

It is an issue with Laravel and MySQL character set.

About the migrations, you don't need to do anything but execute the php artisan migrate command. I'm going to add that to the README.

Closing because this is not a library issue.

Schema::create('scraped_datasets', function (Blueprint $table) {
            $table->string('url', 1024)->primary(); // <--- THE CULPRIT
            $table->string('type');
            $table->string('variant', 40)->index()->nullable();
            $table->json('data');
            $table->timestamps();
        });``

Per MySQL documentation :

"For example, a prefix can be up to 767 bytes long for InnoDB tables that use the REDUNDANT or COMPACT row format. The prefix length limit is 3072 bytes for InnoDB tables that use the DYNAMIC or COMPRESSED row format. For MyISAM tables, the prefix length limit is 1000 bytes."

Given that we're working with utf8mb4 by default on most laravel installations, each character is 4 bytes long. This makes the url field 1024*4 = 4096 bytes long which is over the 3072 limit even for mysql8.

Also, I'm not a DB expert but I wouldn't index on such a long field. Wouldn't it be possible to create a hash of the url and use that as an index?

You are right. An URL is not the best primary key, it should be better with some kind of hash. Could you provide a PR?