Increase product import performance

Question

Increase product import performance

Chimera16 opened this issue 5 years ago · 1 comments

Hello everyone,

I have the problem, that importing products is really slow... I need to import 40k products, which take 8 hours. Every product has 3-5 images, but every image is stored locally. Is this a usually behavior, or is it too slow? I could not find any import time values for products with images for reference...

I have tried importing the products in pieces, for example 40x1000 products, or 80x500products, but the performance remains the same. Partial indexing is deactivated, this is the part of my sript which does the import:

try {
    /** @var $import AvS_FastSimpleImport_Model_Import */
    $import = Mage::getModel('fastsimpleimport/import');
    $import->setPartialIndexing(false);
    $import->setBehavior(Mage_ImportExport_Model_Import::BEHAVIOR_REPLACE);
    $import->processProductImport($data);
} catch (Exception $e) {
    print '<pre>'; print_r($e); print '</pre>';
}

The import is not inside a loop. The server has 128GB of RAM available, the CPU is around 3% usage, while the import is running.

I'm using Magento 1.9.4.4 with FastSimpleImport 0.8.2.

I'm running out of ideas, i would highly appreciate any help..

Thanks in advance

Answer 1 · 2020-03-13T14:56:33.000Z

The problem is related to images, I dont know if there is a solution for that...
Import with images:
68.7555 1 import_p
1.5193 1 CORE::create_object_of::AvS_FastSimpleImport_Model_Import_Entity_Product
0.0009 20 EAV: Mage_Eav_Model_Config::getEntityType
0.0017 6 CORE::create_object_of::Mage_Eav_Model_Resource_Entity_Attribute_Set_Collection
0.0009 179 DISPATCH EVENT:resource_get_tablename
0.0004 63 DISPATCH EVENT:core_collection_abstract_load_before
0.0006 63 DISPATCH EVENT:core_collection_abstract_load_after
0.3354 1 CORE::create_object_of::AvS_FastSimpleImport_Model_Import_Entity_Product_Type_Simple
0.0024 5 CORE::create_object_of::Mage_Catalog_Model_Resource_Product_Attribute_Collection
0.0000 10 CORE::create_object_of::Mage_Eav_Model_Entity_Attribute_Source_Boolean
0.0151 6 CORE::create_object_of::Mage_Core_Model_Design_Package
0.0138 1 CORE::create_object_of::Mage_Core_Model_Design_Config
0.0012 5 CORE::create_object_of::Mage_Tax_Model_Resource_Class_Collection
0.0003 5 CORE::create_object_of::Mage_Directory_Model_Resource_Country_Collection
0.0001 40 CORE::create_object_of::Mage_Eav_Model_Entity_Attribute_Source_Table
0.0063 40 CORE::create_object_of::Mage_Eav_Model_Resource_Entity_Attribute_Option_Collection
0.0003 41 CORE::create_object_of::Mage_Eav_Model_Resource_Entity_Attribute_Option
0.2754 1 CORE::create_object_of::AvS_FastSimpleImport_Model_Import_Entity_Product_Type_Configurable
0.2736 1 CORE::create_object_of::AvS_FastSimpleImport_Model_Import_Entity_Product_Type_Virtual
0.2822 1 CORE::create_object_of::AvS_FastSimpleImport_Model_Import_Entity_Product_Type_Grouped
0.2908 1 CORE::create_object_of::AvS_FastSimpleImport_Model_Import_Entity_Product_Type_Bundle
0.0002 25 CORE::create_object_of::Mage_Catalog_Model_Resource_Eav_Attribute
0.0002 26 DISPATCH EVENT:model_load_after
0.0001 25 DISPATCH EVENT:catalog_entity_attribute_load_after
0.0050 1 CORE::create_object_of::Mage_Catalog_Model_Resource_Category_Collection
0.0023 1 CORE::create_object_of::Mage_Catalog_Model_Resource_Category
0.0553 24 EAV: Mage_Eav_Model_Config::getAttribute
0.0544 23 LOAD_ATTRIBUTE_BY_CODE_
0.0103 1 EAV_COLLECTION_LOAD_ENT
0.0070 1 EAV_COLLECTION_LOAD_ATTR
0.0008 1 CORE::create_object_of::Mage_Customer_Model_Resource_Group_Collection
0.0007 1 CORE::create_object_of::AvS_FastSimpleImport_Model_NestedArrayAdapter
0.0150 270 CORE::create_object_of::Mage_ImportExport_Model_Import_Proxy_Product
0.0014 10 CORE::create_object_of::Mage_ImportExport_Model_Import_Proxy_Product_Resource
0.0000 17 CORE::create_object_of::Mage_Eav_Model_Entity_Attribute_Backend_Default
0.0011 100 CORE::create_object_of::Mage_Catalog_Model_Product_Url
0.0011 100 CORE::create_object_of::Mage_CatalogInventory_Model_Stock_Item
0.0014 1 CORE::create_object_of::Mage_Index_Model_Indexer
0.0007 1 CORE::create_object_of::Mage_Index_Model_Resource_Process_Collection

Import without images:
2.4035 1 import_p
1.0598 1 CORE::create_object_of::AvS_FastSimpleImport_Model_Import_Entity_Product
0.0009 16 EAV: Mage_Eav_Model_Config::getEntityType
0.0014 6 CORE::create_object_of::Mage_Eav_Model_Resource_Entity_Attribute_Set_Collection
0.0009 167 DISPATCH EVENT:resource_get_tablename
0.0004 63 DISPATCH EVENT:core_collection_abstract_load_before
0.0006 63 DISPATCH EVENT:core_collection_abstract_load_after
0.3653 1 CORE::create_object_of::AvS_FastSimpleImport_Model_Import_Entity_Product_Type_Simple
0.0020 5 CORE::create_object_of::Mage_Catalog_Model_Resource_Product_Attribute_Collection
0.0000 10 CORE::create_object_of::Mage_Eav_Model_Entity_Attribute_Source_Boolean
0.0071 6 CORE::create_object_of::Mage_Core_Model_Design_Package
0.0059 1 CORE::create_object_of::Mage_Core_Model_Design_Config
0.0010 5 CORE::create_object_of::Mage_Tax_Model_Resource_Class_Collection
0.0003 5 CORE::create_object_of::Mage_Directory_Model_Resource_Country_Collection
0.0001 40 CORE::create_object_of::Mage_Eav_Model_Entity_Attribute_Source_Table
0.0069 40 CORE::create_object_of::Mage_Eav_Model_Resource_Entity_Attribute_Option_Collection
0.0004 41 CORE::create_object_of::Mage_Eav_Model_Resource_Entity_Attribute_Option
0.1707 1 CORE::create_object_of::AvS_FastSimpleImport_Model_Import_Entity_Product_Type_Configurable
0.1622 1 CORE::create_object_of::AvS_FastSimpleImport_Model_Import_Entity_Product_Type_Virtual
0.1491 1 CORE::create_object_of::AvS_FastSimpleImport_Model_Import_Entity_Product_Type_Grouped
0.1562 1 CORE::create_object_of::AvS_FastSimpleImport_Model_Import_Entity_Product_Type_Bundle
0.0002 25 CORE::create_object_of::Mage_Catalog_Model_Resource_Eav_Attribute
0.0002 26 DISPATCH EVENT:model_load_after
0.0001 25 DISPATCH EVENT:catalog_entity_attribute_load_after
0.0048 1 CORE::create_object_of::Mage_Catalog_Model_Resource_Category_Collection
0.0022 1 CORE::create_object_of::Mage_Catalog_Model_Resource_Category
0.0637 24 EAV: Mage_Eav_Model_Config::getAttribute
0.0628 23 LOAD_ATTRIBUTE_BY_CODE_
0.0096 1 EAV_COLLECTION_LOAD_ENT
0.0067 1 EAV_COLLECTION_LOAD_ATTR
0.0006 1 CORE::create_object_of::Mage_Customer_Model_Resource_Group_Collection
0.0007 1 CORE::create_object_of::AvS_FastSimpleImport_Model_NestedArrayAdapter
0.0042 270 CORE::create_object_of::Mage_ImportExport_Model_Import_Proxy_Product
0.0000 17 CORE::create_object_of::Mage_Eav_Model_Entity_Attribute_Backend_Default
0.0006 100 CORE::create_object_of::Mage_Catalog_Model_Product_Url
0.0010 100 CORE::create_object_of::Mage_CatalogInventory_Model_Stock_Item
0.0013 1 CORE::create_object_of::Mage_Index_Model_Indexer
0.0007 1 CORE::create_object_of::Mage_Index_Model_Resource_Process_Collection