Testing: Import value annotations
Closed this issue · 9 comments
The value-annotations branch enables the import of value annotations.
To test this, you'll need to find a remote installation that has value annotations. If you cannot find one—and you likely won't—it's fine to use an installation that you control. Technically, the remote installation and local installation can be one and the same, but I prefer that they are separate installations, because I want property ID mismatches. (If we import from the same installation, we can't test for accurate mapping since property IDs are the same.)
On the remote installation, add quite a few value annotations to multiple values. Make sure to use multiple property/data type combinations, preferably using properties from vocabularies you've imported yourself. Make sure to include "resource" data types as well. Then, on the local installation, add a new import, linking it to the remote installation. On the manage import page, take a snapshot, prepare the import by mapping data types, and import. After this is completed, the newly created items should have values that have value annotations that are identical to the ones on the remote installation.
Because I made some critical changes to the code, you should test this thoroughly, referring back to the original testing issue (#1). You should check for expected behavior for the entire import, not just value annotations. Thanks!
I'm getting this error when trying to import a snapshot:
2022-09-12T19:48:18+00:00 INFO (6): Iterating OSII media:
1666, 1667, 1668, 1669, 1670, 1671, 1672, 1673, 1674, 1675,
1676, 1677, 1678, 1679, 1680, 1681, 1682, 1683, 1684, 1685,
2022-09-12T19:48:18+00:00 INFO (6): Iterating OSII media:
1686, 1687, 1688, 1689, 1690, 1691, 1692, 1693, 1694, 1695,
1696, 1697, 1698, 1699, 1700, 1701, 1702, 1703, 1704, 1705,
2022-09-12T19:48:18+00:00 INFO (6): Iterating OSII media:
1706, 1707, 1708, 1709, 1710, 1711, 1712, 1713, 1714, 1715,
1716, 1717, 1718, 1719, 1720, 1721, 1722, 1723, 1724, 1725,
2022-09-12T19:48:18+00:00 INFO (6): Iterating OSII media:
1726, 1727, 1728, 1729, 1730, 1731, 1732, 1733, 1734, 1735,
1736, 1737, 1738, 1739, 1740, 1741, 1742, 1743, 1744, 1745,
2022-09-12T19:48:18+00:00 INFO (6): Iterating OSII media:
1746, 1747, 1748, 1749, 1750, 1751, 1752, 1753, 1754, 1755,
1756, 1757, 1758, 1759, 1760, 1761, 1762, 1763, 1764, 1765,
2022-09-12T19:48:19+00:00 INFO (6): Iterating OSII media:
1766, 1767, 1768, 1769, 1770, 1771, 1772, 1773, 1774, 1775,
1776, 1777, 1778, 1779, 1780, 1781, 1782, 1783, 1784, 1785,
2022-09-12T19:48:19+00:00 INFO (6): Iterating OSII media:
1786, 1787, 1788, 1789, 1790, 1791, 1792, 1793, 1794, 1795,
1796, 1797, 1798, 1799, 1800, 1801, 1802, 1803, 1804, 1805,
2022-09-12T19:48:19+00:00 INFO (6): Iterating OSII media:
1806, 1807, 1808, 1809, 1810, 1811, 1812, 1813, 1814, 1815,
1816, 1817, 1818, 1819, 1820, 1821, 1822, 1823, 1824, 1825,
2022-09-12T19:48:19+00:00 INFO (6): Iterating OSII media:
1826, 1827, 1828, 1829, 1830, 1831, 1832, 1833, 1834, 1835,
1836, 1837, 1838, 1839, 1840, 1841, 1842, 1843, 1844, 1845,
2022-09-12T19:48:19+00:00 INFO (6): Iterating OSII media:
1846, 1847, 1848, 1849, 1850, 1851, 1852, 1853, 1854, 1855,
1856, 1857, 1858, 1859, 1860, 1861, 1862, 1863, 1864, 1865,
2022-09-12T19:48:19+00:00 INFO (6): Iterating OSII media:
1866, 1867, 1868, 1869, 1870, 1871, 1872, 1873, 1874, 1875,
1876, 1877, 1878, 1879, 1880, 1881, 1882, 1883, 1884, 1885,
2022-09-12T19:48:19+00:00 INFO (6): Iterating OSII media:
1886, 1887, 1888, 1889, 1890, 1891, 1892, 1893, 1894, 1895,
1896, 1897, 1898, 1899, 1900, 1901, 1902, 1903, 1904, 1905,
2022-09-12T19:48:20+00:00 INFO (6): Iterating OSII media:
1906, 1907, 1908, 1909, 1910, 1911, 1912, 1913, 1914, 1915,
1916, 1917, 1918, 1919, 1920, 1921, 1922, 1923, 1924, 1925,
2022-09-12T19:48:20+00:00 INFO (6): Iterating OSII media:
1926, 1927, 1928, 1929, 1930, 1931, 1932, 1933, 1934, 1935,
1936, 1937, 1938, 1939, 1940, 1941, 1942, 1943, 1944, 1945,
2022-09-12T19:48:20+00:00 INFO (6): Iterating OSII media:
1946, 1947, 1948, 1949, 1950, 1951, 1952, 1953, 1954, 1955,
1956, 1957, 1958, 1959, 1960, 1961, 1962, 1963, 1964, 1965,
2022-09-12T19:48:20+00:00 INFO (6): Iterating OSII media:
1966, 1967, 1968, 1969, 1970, 1971, 1972, 1973, 1974, 1975,
1976, 1977, 1978, 1979, 1980, 1981, 1982, 1983, 1984, 1985,
2022-09-12T19:48:20+00:00 INFO (6): Iterating OSII media:
1986, 1987, 1988, 1989, 1990, 1991, 1992, 1993, 1994, 1995,
1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
2022-09-12T19:48:20+00:00 INFO (6): Iterating OSII media:
2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015,
2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025,
2022-09-12T19:48:20+00:00 INFO (6): Iterating OSII media:
2026, 2027, 2028, 2029, 2030, 2031, 2032, 2033, 2034, 2035,
2036, 2037, 2038, 2039, 2040, 2041, 2042, 2043, 2044, 2045,
2022-09-12T19:48:21+00:00 INFO (6): Iterating OSII media:
2046, 2047, 2048, 2049, 2050, 2051, 2052, 2053, 2054, 2055,
2056, 2057, 2058, 2059, 2060, 2061, 2062, 2063, 2064, 2065,
2022-09-12T19:48:21+00:00 INFO (6): Iterating OSII media:
2066, 2067, 2068, 2069, 2070, 2071, 2072, 2073, 2074, 2075,
2076, 2077, 2078, 2079, 2080, 2081, 2082, 2083, 2084, 2085,
2022-09-12T19:48:21+00:00 INFO (6): Iterating OSII media:
2086, 2087, 2088, 2089, 2090, 2091, 2092, 2093, 2094, 2095,
2096, 2097, 2098, 2099, 2100, 2101, 2102, 2103, 2104, 2105,
2022-09-12T19:48:21+00:00 INFO (6): Iterating OSII media:
2106, 2107, 2108, 2109, 2110, 2111, 2112, 2113, 2114, 2115,
2116, 2117, 2118, 2119, 2120, 2121, 2122, 2123, 2124, 2125,
2022-09-12T19:48:21+00:00 INFO (6): Iterating OSII media:
2126, 2127, 2128, 2129, 2130, 2131, 2132, 2133, 2134, 2135,
2136, 2137, 2138, 2139, 2140, 2141, 2142, 2143, 2144, 2145,
2022-09-12T19:48:21+00:00 INFO (6): Iterating OSII media:
2146, 2147, 2148, 2149, 2150, 2151, 2152, 2153, 2154, 2155,
2156, 2157, 2158, 2159, 2160, 2161, 2162, 2163, 2164, 2165,
2022-09-12T19:48:21+00:00 INFO (6): Iterating OSII media:
2166, 2167, 2168, 2169, 2170, 2171, 2172, 2173, 2174, 2175,
2176, 2177, 2178, 2179, 2180, 2181, 2182, 2183, 2184, 2185,
2022-09-12T19:48:21+00:00 INFO (6): Iterating OSII media:
2186, 2187, 2188,
2022-09-12T19:48:22+00:00 INFO (6): Iterating OSII items:
3239, 3240, 3241, 3242, 3243, 3244, 3245, 3246, 3247, 3248,
3249, 3250, 3251, 3252, 3253, 3254, 3255, 3256, 3257, 3258,
3259, 3260, 3261, 3262, 3263, 3264, 3265, 3266, 3267, 3268,
3269, 3270, 3271, 3272, 3273, 3274, 3275, 3276, 3277, 3278,
3279, 3280, 3281, 3282, 3283, 3284, 3285, 3286, 3287, 3288,
3289, 3290, 3291, 3292, 3293, 3294, 3295, 3296, 3297, 3298,
3299, 3300, 3301, 3302, 3303, 3304, 3305, 3306, 3307, 3308,
3309, 3310, 3311, 3312, 3313, 3314, 3315, 3316, 3317, 3318,
3319, 3320, 3321, 3322, 3323, 3324, 3325, 3326, 3327, 3328,
3329, 3330, 3331, 3332, 3333, 3334, 3335, 3336, 3337, 3338,
2022-09-12T19:48:23+00:00 INFO (6): Iterating OSII items:
3339, 3340, 3341, 3342, 3343, 3344, 3345, 3346, 3347, 3348,
3349, 3350, 3351, 3352, 3353, 3354, 3355, 3356, 3357, 3358,
3359, 3360, 3361, 3362, 3363, 3364, 3365, 3366, 3367, 3368,
3369, 3370, 3371, 3372, 3373, 3374, 3375, 3376, 3377, 3378,
3379, 3380, 3381, 3382, 3383, 3384, 3385, 3386, 3387, 3388,
3389, 3390, 3391, 3392, 3393, 3394, 3395, 3396, 3397, 3398,
3399, 3400, 3401, 3402, 3403, 3404, 3405, 3406, 3407, 3408,
3409, 3410, 3411, 3412, 3413, 3414, 3415, 3416, 3417, 3418,
3419, 3420, 3421, 3422, 3423, 3424, 3425, 3426, 3427, 3428,
3429, 3430, 3431, 3432, 3433, 3434, 3435, 3436, 3437, 3438,
2022-09-12T19:48:24+00:00 ERR (3): PDOException: SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (katknow.value, CONSTRAINT FK_1D775834549213EC FOREIGN KEY (property_id) REFERENCES property (id) ON DELETE CASCADE) in /var/www/html/katknow/OmekaSTesting/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php:112
Stack trace:
#0 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php(112): PDOStatement->execute(NULL)
#1 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/dbal/lib/Doctrine/DBAL/Statement.php(177): Doctrine\DBAL\Driver\PDOStatement->execute(NULL)
#2 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/Entity/BasicEntityPersister.php(287): Doctrine\DBAL\Statement->execute()
#3 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(1122): Doctrine\ORM\Persisters\Entity\BasicEntityPersister->executeInserts()
#4 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(420): Doctrine\ORM\UnitOfWork->executeInserts(Object(Doctrine\ORM\Mapping\ClassMetadata))
#5 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(376): Doctrine\ORM\UnitOfWork->commit(NULL)
#6 /var/www/html/katknow/OmekaSTesting/modules/Osii/src/Job/AbstractOsiiJob.php(173): Doctrine\ORM\EntityManager->flush()
#7 /var/www/html/katknow/OmekaSTesting/modules/Osii/src/Job/DoImport.php(300): Osii\Job\AbstractOsiiJob->flushClear()
#8 /var/www/html/katknow/OmekaSTesting/application/src/Job/DispatchStrategy/Synchronous.php(34): Osii\Job\DoImport->perform()
#9 /var/www/html/katknow/OmekaSTesting/application/src/Job/Dispatcher.php(105): Omeka\Job\DispatchStrategy\Synchronous->send(Object(Omeka\Entity\Job))
#10 /var/www/html/katknow/OmekaSTesting/application/data/scripts/perform-job.php(66): Omeka\Job\Dispatcher->send(Object(Omeka\Entity\Job), Object(Omeka\Job\DispatchStrategy\Synchronous))
#11 {main}
Next Doctrine\DBAL\Driver\PDO\Exception: SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (katknow.value, CONSTRAINT FK_1D775834549213EC FOREIGN KEY (property_id) REFERENCES property (id) ON DELETE CASCADE) in /var/www/html/katknow/OmekaSTesting/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDO/Exception.php:18
Stack trace:
#0 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php(114): Doctrine\DBAL\Driver\PDO\Exception::new(Object(PDOException))
#1 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/dbal/lib/Doctrine/DBAL/Statement.php(177): Doctrine\DBAL\Driver\PDOStatement->execute(NULL)
#2 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/Entity/BasicEntityPersister.php(287): Doctrine\DBAL\Statement->execute()
#3 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(1122): Doctrine\ORM\Persisters\Entity\BasicEntityPersister->executeInserts()
#4 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(420): Doctrine\ORM\UnitOfWork->executeInserts(Object(Doctrine\ORM\Mapping\ClassMetadata))
#5 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(376): Doctrine\ORM\UnitOfWork->commit(NULL)
#6 /var/www/html/katknow/OmekaSTesting/modules/Osii/src/Job/AbstractOsiiJob.php(173): Doctrine\ORM\EntityManager->flush()
#7 /var/www/html/katknow/OmekaSTesting/modules/Osii/src/Job/DoImport.php(300): Osii\Job\AbstractOsiiJob->flushClear()
#8 /var/www/html/katknow/OmekaSTesting/application/src/Job/DispatchStrategy/Synchronous.php(34): Osii\Job\DoImport->perform()
#9 /var/www/html/katknow/OmekaSTesting/application/src/Job/Dispatcher.php(105): Omeka\Job\DispatchStrategy\Synchronous->send(Object(Omeka\Entity\Job))
#10 /var/www/html/katknow/OmekaSTesting/application/data/scripts/perform-job.php(66): Omeka\Job\Dispatcher->send(Object(Omeka\Entity\Job), Object(Omeka\Job\DispatchStrategy\Synchronous))
#11 {main}
Next Doctrine\DBAL\Exception\ForeignKeyConstraintViolationException: An exception occurred while executing 'INSERT INTO value (type, lang, value, uri, is_public, resource_id, property_id, value_resource_id, value_annotation_id) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)' with params ["literal", null, "this is a title test", null, 1, 34195, 1568, null, null]:
SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (katknow.value, CONSTRAINT FK_1D775834549213EC FOREIGN KEY (property_id) REFERENCES property (id) ON DELETE CASCADE) in /var/www/html/katknow/OmekaSTesting/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/AbstractMySQLDriver.php:68
Stack trace:
#0 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/dbal/lib/Doctrine/DBAL/DBALException.php(182): Doctrine\DBAL\Driver\AbstractMySQLDriver->convertException('An exception oc...', Object(Doctrine\DBAL\Driver\PDO\Exception))
#1 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/dbal/lib/Doctrine/DBAL/DBALException.php(159): Doctrine\DBAL\DBALException::wrapException(Object(Doctrine\DBAL\Driver\PDO\MySQL\Driver), Object(Doctrine\DBAL\Driver\PDO\Exception), 'An exception oc...')
#2 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(2214): Doctrine\DBAL\DBALException::driverExceptionDuringQuery(Object(Doctrine\DBAL\Driver\PDO\MySQL\Driver), Object(Doctrine\DBAL\Driver\PDO\Exception), 'INSERT INTO va...', Array) #3 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/dbal/lib/Doctrine/DBAL/Statement.php(183): Doctrine\DBAL\Connection->handleExceptionDuringQuery(Object(Doctrine\DBAL\Driver\PDO\Exception), 'INSERT INTO va...', Array, Array)
#4 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/Entity/BasicEntityPersister.php(287): Doctrine\DBAL\Statement->execute()
#5 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(1122): Doctrine\ORM\Persisters\Entity\BasicEntityPersister->executeInserts()
#6 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(420): Doctrine\ORM\UnitOfWork->executeInserts(Object(Doctrine\ORM\Mapping\ClassMetadata))
#7 /var/www/html/katknow/OmekaSTesting/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(376): Doctrine\ORM\UnitOfWork->commit(NULL)
#8 /var/www/html/katknow/OmekaSTesting/modules/Osii/src/Job/AbstractOsiiJob.php(173): Doctrine\ORM\EntityManager->flush()
#9 /var/www/html/katknow/OmekaSTesting/modules/Osii/src/Job/DoImport.php(300): Osii\Job\AbstractOsiiJob->flushClear()
#10 /var/www/html/katknow/OmekaSTesting/application/src/Job/DispatchStrategy/Synchronous.php(34): Osii\Job\DoImport->perform()
#11 /var/www/html/katknow/OmekaSTesting/application/src/Job/Dispatcher.php(105): Omeka\Job\DispatchStrategy\Synchronous->send(Object(Omeka\Entity\Job))
#12 /var/www/html/katknow/OmekaSTesting/application/data/scripts/perform-job.php(66): Omeka\Job\Dispatcher->send(Object(Omeka\Entity\Job), Object(Omeka\Job\DispatchStrategy\Synchronous))
#13 {main}
That's odd. Did you happen to delete a vocabulary while the import process was running? Try running the import again to see if it happens regularly.
Also, to make sure that this error is unconnected with the new feature, try checking out the main branch and running the import. Are there any errors?
So there weren't errors on main, and when I switched back and tried again on the value annotation branch it did work. It imported everything correctly, including multiple kinds of annotations with vocabularies I had imported myself on different kinds of resources. I did end up importing from my own Omeka S instance since I was having trouble finding annotations, but it was a separate one I had created. However, there is some weirdness going on with how the item page looks:
I'm guessing this doesn't have anything to do with this, but since it happened while I was trying to sort out this problem I thought I'd mention it.
I'm going to revisit the original testing issue to generally check and make sure all the different elements are working as anticipated.
Other than the weird layout I think this is working okay. Tried with queries, with and without media, with and without item sets, etc.
Other than the weird layout I think this is working okay. Tried with queries, with and without media, with and without item sets, etc.
There's been a recent update to metadata styles and markup. Try a hard refresh and let me know if these issues persist.
@kimisgold yes hard refresh fixed it, thanks.
@jimsafley I played with this a couple more hours last night and was unable to replicate the error so I think we're it may be good to go. I redid the original testing issue as well and things worked as expected. The only caveat is I imported from a second instance I created myself, so everything was purely based on what I could think up myself in terms of annotations, vocabularies etc.
Thanks. It's a strange and unexpected error, which gives me pause. Anyway, as usual, I appreciate your thoroughness!