SDL WorldServer: Environment is unresponsive all projects are stuck with no progress. The workflow engine queue is far behind such that many requests are in the queue for hours in the In Queue state

Production environment is unresponsive all projects are stuck with no progress. The workflow engine queue is far behind such that many requests are in the queue for hours in the In Queue state

In those cases where either the concurrent Project Scope Automatic Actions for all the projects in a project group are In Queue, or when there is a dramatic backlog of tasks in the workflow engine queue, it may be the case that increasing the number of workflow engines will produce some relief for such issues.

Changes needed to increase workflow engines to 30:

pagefile size must be at least 1.5 time the size of physical memory

java heap must be set to at least -Xmx8192m

In the general.properties files of the WS_Config folder  ( at W:\WorldServer\WS_CONFIG\general.properties in hosted deployments)
   

database_connection_pool_size=current database connection pool size + 100  
total_workflow_daemons = 30
workflow_engine_cache_size = 10
workflow_daemon_project_queue_frequency = 30
workflow_daemon_projectgroup_queue_frequency = 30

In the general.properties files of the WS_Config\ws-legacy folder  ( at W:\WorldServer\WS_CONFIG\ws-legacy\general.properties)

workflow_daemon=30

These changes require an Idiom service restart. How to determine if this change helped?

You will be able to see if the change helped by reviewing the progression of the Workflow Engine restart after these changes are deployed

There are not sufficient workflow engines to allow concurrent processing and instead, long-running actions such as the Automatic Action Generate Translation Kit are taking 15 minutes or more. Those are the the requests at the top of the queue or there are a very large number of tasks scope Automatic Actions which are in the In Queue state. In either situation more workflow engines may reduce or even eliminate the workflow engine processing delay.

This article is relevant:

SDL WorldServer: About the workflow daemon project and project group queue frequency (general.properties file) and the sleeptime value in the Sdl.WorldServer.FileTypeSupport.Server.HostProcess.exe.config