When publishing over 100+, it always throws Out of Memory Error below: Error : cn=sites9-deployer cid=d257b5c69f05 [2663]: Exception in thread "http-nio2-8084-exec-37" java.lang.OutOfMemoryError: Java heap space cn=sites9-deployer cid=d257b5c69f05 [2663]: 2024-04-08 12:28:13,177 ERROR [dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] threw exception cn=sites9-deployer cid=d257b5c69f05 [2663]: java.lang.OutOfMemoryError: Java heap space cn=sites9-deployer cid=d257b5c69f05 [2663]: 2024-04-08 12:28:13,179 ERROR [localhost] - Exception Processing ErrorPage[errorCode=0, location=/error] cn=sites9-deployer cid=d257b5c69f05 [2663]: org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1055) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) cn=sites9-deployer cid=d257b5c69f05 [2663]: at javax.servlet.http.HttpServlet.service(HttpServlet.java:655) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) cn=sites9-deployer cid=d257b5c69f05 [2663]: at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:84) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:103) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:103) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:711) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:461) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:385) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:313) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:403) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:249) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:344) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) cn=sites9-deployer cid=d257b5c69f05 [2663]: at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)When issue occurs, it will require restarting deployer service. However, we cannot keep recycling deployer on Production environment. Publishing performance has improved after cleanning up State Store Database tables but the OutOfMemory exception was difference than originally seen Error Stack: cn=sites9-deployer cid=66a2d0276f9b [2011]: java.lang.OutOfMemoryError: Java heap space cn=sites9-deployer cid=66a2d0276f9b [2011]: 2024-04-09 14:14:29,904 ERROR TridionTransactionalExecutableStep - Rollback transaction due to error. ExecutionId: tcm:0-20812743-66560 error: cn=sites9-deployer cid=66a2d0276f9b [2011]: com.tridion.broker.StorageException: error while querying entity, transaction timeout has expired cn=sites9-deployer cid=66a2d0276f9b [2011]: at com.tridion.storage.persistence.JPABaseDAO.wrapStorageException(JPABaseDAO.java:603) cn=sites9-deployer cid=66a2d0276f9b [2011]: at com.tridion.storage.persistence.JPABaseDAO.prepareExceptions(JPABaseDAO.java:591) cn=sites9-deployer cid=66a2d0276f9b [2011]: at com.tridion.storage.persistence.JPABaseDAO.executeQueryListResultThrowable(JPABaseDAO.java:831) cn=sites9-deployer cid=66a2d0276f9b [2011]: at com.tridion.storage.persistence.JPABaseDAO.executeQueryListResultThrowable(JPABaseDAO.java:734) cn=sites9-deployer cid=66a2d0276f9b [2011]: at com.tridion.storage.persistence.JPABaseDAO.executeQueryListResultThrowable(JPABaseDAO.java:656) cn=sites9-deployer cid=66a2d0276f9b [2011]: at com.tridion.storage.persistence.JPABaseDAO.executeQuerySingleResult(JPABaseDAO.java:633) cn=sites9-deployer cid=66a2d0276f9b [2011]: at com.tridion.storage.persistence.JPATaxonomyDAO.getRecalculatedKeywordCount(JPATaxonomyDAO.java:581) cn=sites9-deployer cid=66a2d0276f9b [2011]: at com.tridion.storage.caching.CachedTaxonomyDAO.getRecalculatedKeywordCount(CachedTaxonomyDAO.java:320) cn=sites9-deployer cid=66a2d0276f9b [2011]: at com.tridion.storage.services.TaxonomyCounterService.updateKeywordReferenceCount(TaxonomyCounterService.java:71) cn=sites9-deployer cid=66a2d0276f9b [2011]: at com.tridion.storage.services.TaxonomyCounterService.updateKeywordsReferenceCount(TaxonomyCounterService.java:52) cn=sites9-deployer cid=66a2d0276f9b [2011]: at com.tridion.storage.services.TaxonomyCounterService.updateKeywordsReferenceCount(TaxonomyCounterService.java:34) cn=sites9-deployer cid=66a2d0276f9b [2011]: at com.tridion.storage.deploy.workers.ComponentWorker.doUnDeploy(ComponentWorker.java:161)Again, after disabling Taxonomies count as below, issue can still be reproduced when unpublishing 12+ items at a time <Taxonomies CountEnabled="false"/> New Error: 'http://javax.xml.XMLConstants/property/accessExternalStylesheet' to document builder factory cn=sites9-deployer cid=fef5795e2930 [2011]: 2024-04-15 21:36:47,264 WARN ResourceTypeProcessor - Could not set attribute 'http://javax.xml.XMLConstants/property/accessExternalStylesheet' to document builder factory cn=sites9-deployer cid=fef5795e2930 [2011]: 2024-04-15 21:36:53,693 ERROR JPABaseDAO - Error while executing query cn=sites9-deployer cid=fef5795e2930 [2011]: com.tridion.broker.StorageException: error while querying entity, transaction timeout has expired cn=sites9-deployer cid=fef5795e2930 [2011]: at com.tridion.storage.persistence.JPABaseDAO.wrapStorageException(JPABaseDAO.java:603) cn=sites9-deployer cid=fef5795e2930 [2011]: at com.tridion.storage.persistence.JPABaseDAO.prepareExceptions(JPABaseDAO.java:591) cn=sites9-deployer cid=fef5795e2930 [2011]: at com.tridion.storage.persistence.JPABaseDAO.executeQueryListResultThrowable(JPABaseDAO.java:831) cn=sites9-deployer cid=fef5795e2930 [2011]: at com.tridion.storage.persistence.JPABaseDAO.executeQueryListResultThrowable(JPABaseDAO.java:734) cn=sites9-deployer cid=fef5795e2930 [2011]: at com.tridion.storage.persistence.JPABaseDAO.executeQueryListResultThrowable(JPABaseDAO.java:656) cn=sites9-deployer cid=fef5795e2930 [2011]: at com.tridion.storage.persistence.JPABaseDAO.executeQueryListResult(JPABaseDAO.java:669) cn=sites9-deployer cid=fef5795e2930 [2011]: at com.tridion.storage.persistence.JPAReferenceEntryDAO.findByReferencedURI(JPAReferenceEntrcurrent commit workers was 24, why does it still throw OutOfMemory exception when unpublishing over 12 items? The current commit workers was 24, why does it still throw OutOfMemory exception when unpublishing over 12 items? |
-clean up State store database tables
delete from EXECUTION delete from EXECUTION_PROPERTY delete from PIPELINE delete from STEP-update setting in cd_storage_conf.xml of deployer service <Taxonomies CountEnabled="false"/-tune deployer by changing the number of commit workers from 24 to 10 |
-State store database tables have grown too large. -Taxonomies count in the cd_storage_conf.xml file was enabled. -Number of workers in the deployer set too high. |
https://gateway.sdl.com/CommunityKnowledge?articleName=000003519 https://gateway.sdl.com/CommunityKnowledge?articleName=000011841 |