- Publishing will fail randomly on either one server when publishing to 2 servers.
- There is no load-balancer through Tridion.
- There is no consistent patterns for the publisher failures. It usually will workaround if republishing the failed items.
- Sometimes, bulk publishing will crash the deployer and causing deployer services to stop.
Scenario for the setup :
- Sharing brokers across deployers but the deployers are specific to individual Publications
- There are 2 deployers on the same machine, both deployers are pointed to the same broker db and both are using the same broker db for the state store.
Errors observed are :
2018-06-19 13:49:50,473 ERROR FSBinaryStorage - The file 'D:\SDLWeb\binary\Binaries\tcm_0-324969-66560.Content.zip' could not be put to binary storage. java.nio.file.FileSystemException: D:\SDLWeb\binary\Binaries\tcm_0-324969-66560.Content.zip: The process cannot access the file because it is being used by another process. at sun.nio.fs.WindowsException.translateToIOException(Unknown Source) at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source) at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source) at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source) at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source) at java.nio.file.Files.deleteIfExists(Unknown Source) at java.nio.file.Files.copy(Unknown Source) at com.sdl.delivery.deployer.binarystorage.filesystem.FSBinaryStorage.streamToFile (FSBinaryStorage.java:210) at com.sdl.delivery.deployer.binarystorage.filesystem.FSBinaryStorage.put (FSBinaryStorage.java:139) at com.sdl.delivery.deployer.legacy.endpoint.controller.DeployerLegacyEndpointController.handleUpload (DeployerLegacyEndpointController.java:386) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke (InvocableHandlerMethod.java:221) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest (InvocableHandlerMethod.java:136)
The number render failures has exceeded its specified failure limit of 0.
2018-06-19 11:37:45,287 ERROR SpringApplication - Application startup failed org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'AkkaProcessingService' defined in URL [jar:file:/D:/SDLWeb/deployer_API.Healthyroads_ActiveandFit/services/deployer-worker/deployer-worker-8.5.0-1014.jar!/com/sdl/delivery/deployer/engine/pipeline/actor/AkkaProcessingService.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'actorProducer': Unsatisfied dependency expressed through field 'actorSystem'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'actorSystem' defined in class path resource [com/sdl/delivery/deployer/actor/spring/AkkaConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [akka.actor.ActorSystem]: Factory method 'actorSystem' threw exception; nested exception is akka.ConfigurationException: Could not start logger due to [akka.ConfigurationException: Logger specified in config can't be loaded [akka.event.Logging$DefaultLogger] due to [akka.event.Logging$LoggerInitializationException: Logger log1-Logging$DefaultLogger did not respond with LoggerInitialized, sent instead [TIMEOUT]]]
Troubleshooting steps :
- Add a new Broker DB for each Deployer that exists on the same machine so that they do not share the same one for status or content
- If there are multiple deployers on the same machine, it will need to configure a different temporary storage location for each in cd_storage_conf.xml
- In deployer-conf.xml,ensure to add "LegacyTransport=true" if using legacy publishing methods.
- In start.ps1 ($jvmoptions) or start.sh (JVM_OPTIONS), add "-Dakka.logger-startup-timeout=30s" in the script which starts the deployer service.
- If the deployer service is installed as window service, will need to add that option in the installservice.ps1 script, and then uninstall and re-install the deployer microservice
e.g. $jvmoptions = '-Xrs", "-Xms512m", "-Xmx1024m", "-Dfile.encoding=UTF-8", "Dakka.logger-startup-timeout=30s"
- Move the queue and binary folders from inside the Deployers install path to a separate folder outside of the install path
- Enable Process Monitor to trace what are the processes accessing the files when the error occurs.
- Apply the latest Hotfix for the deployer (e.g. CD_188.8.131.5245, CD_184.108.40.20639)