提问者:小点点

如何使用jpa repository saveAll忽略保存操作的重复插入?


我需要能够使用spring数据CrudRepository的saveAll()方法,并忽略对唯一约束的重复插入。

在我当前的项目中,我目前使用的是spring boot-

问题是我有一个唯一的约束,文件可能与旧行一起有新行。通过POI处理Excel文件非常繁重,我通过java.util.Set将计算结果保存在内存中,直到我到达必须使用CrudRepository的saveAll()方法保留Set的时间点。

我目前遇到:错误:重复的键值违反了唯一约束“csspd_avoid_duplicates”

org.springframework.transaction.UnexpectedRollbackException:事务已自动回滚,因为它已标记为仅回滚

并且导入在第一次遇到重复时停止。

对于同一项目,我在循环中单次插入(单 save())时遇到了类似的问题,我已经设法使用以下代码片段解决了这个问题:

                try {

                    repository.save(entity);
                } catch (DataIntegrityViolationException e) {
                    this.log.debug("Duplicate found, skipping");
                }

我也尝试过同样的方法,但没有运气。

这是一个有趣的方法:

public boolean triggerExcelImport(Path path) {
    try {

        ExcelDataDTO excelExtractedData;
        if (path == null) {
            excelExtractedData = excelImporterService.importExcelFiles();
        } else {
            excelExtractedData = excelImporterService.importStandardXLSXFiles(path);
        }

        staticProductDataRepository.saveAll(excelExtractedData.getProductData());  // <-- This saveAll might have duplicates

        for (Set<AllocationChartData> allocationList : excelExtractedData.getAllocationMap().values()) {
            allocationChartDataRepository.saveAll(allocationList);   // <-- This saveAll might have duplicates
        }

        performanceRepository.saveAll(excelExtractedData.getPerformanceData());    // <-- This saveAll might have duplicates

        return true;
    } catch (DataIntegrityViolationException e) {
        this.log.debug("Duplicate found, skipping");
    }
    return false;
}

我希望导入过程能够实际完成导入,避免在第一个副本上终止。该解决方案必须是独立于数据库的(我正在使用 Postgres 和 Oracle 通过 liquibase 上下文进行管理)

这是我遇到的实际例外:

        org.springframework.transaction.UnexpectedRollbackException: Transaction silently rolled back because it has been marked as rollback-only
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:755) ~[spring-tx-5.1.6.RELEASE.jar:5.1.6.RELEASE]
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:714) ~[spring-tx-5.1.6.RELEASE.jar:5.1.6.RELEASE]
        at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:533) ~[spring-tx-5.1.6.RELEASE.jar:5.1.6.RELEASE]
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:304) ~[spring-tx-5.1.6.RELEASE.jar:5.1.6.RELEASE]
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98) ~[spring-tx-5.1.6.RELEASE.jar:5.1.6.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.1.6.RELEASE.jar:5.1.6.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688) ~[spring-aop-5.1.6.RELEASE.jar:5.1.6.RELEASE]
        at com.juliusbaer.ims.toolbox.services.CenshareService$$EnhancerBySpringCGLIB$$8e7529c1.triggerCenshareExcelImport(<generated>) ~[classes/:?]
        at com.juliusbaer.ims.toolbox.services.FolderWatchService.readAndCleanUpFile(FolderWatchService.java:168) ~[classes/:?]
        at com.juliusbaer.ims.toolbox.services.FolderWatchService.initializeFileImportAtStartup(FolderWatchService.java:99) ~[classes/:?]
        at com.juliusbaer.ims.toolbox.services.FolderWatchService.setUp(FolderWatchService.java:93) ~[classes/:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:363) ~[spring-beans-5.1.6.RELEASE.jar:5.1.6.RELEASE]
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:307) ~[spring-beans-5.1.6.RELEASE.jar:5.1.6.RELEASE]
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:136) ~[spring-beans-5.1.6.RELEASE.jar:5.1.6.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:414) ~[spring-beans-5.1.6.RELEASE.jar:5.1.6.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1770) ~[spring-beans-5.1.6.RELEASE.jar:5.1.6.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593) ~[spring-beans-5.1.6.RELEASE.jar:5.1.6.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515) ~[spring-beans-5.1.6.RELEASE.jar:5.1.6.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320) ~[spring-beans-5.1.6.RELEASE.jar:5.1.6.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) [spring-beans-5.1.6.RELEASE.jar:5.1.6.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318) [spring-beans-5.1.6.RELEASE.jar:5.1.6.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) [spring-beans-5.1.6.RELEASE.jar:5.1.6.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:849) [spring-beans-5.1.6.RELEASE.jar:5.1.6.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:877) [spring-context-5.1.6.RELEASE.jar:5.1.6.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549) [spring-context-5.1.6.RELEASE.jar:5.1.6.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:142) [spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:775) [spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) [spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:316) [spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1260) [spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1248) [spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
        at com.juliusbaer.ims.toolbox.Application.main(Application.java:12) [classes/:?]

谢谢你的帮助。

编辑:为了给你一个更简单的解释,惟一约束不是基于单个标识符id,而是由多个列(总共8列)组成


共2个答案

匿名用户

您可以使用@Query注释来定义您自己的自定义repo方法来完成此操作。

@Query("INSERT IGNORE INTO table (COLUMNS) values (...)")
public List<S> saveAllInsertIgnore(List<> list);

参考:

https://stackoverflow.com/a/70574062/6784846

匿名用户

为什么不在想要唯一性的列上为对象实现equals()hashCode()函数?将您的对象存储在集合中,它将自动删除重复项,然后您可以毫无例外地使用saveAll()