Tomcat 502

 

We have some client software that uploads files using HTTPS on an authenticated session that is proxied through Apache to Tomcat. However, if an authenticated session is not used (or the session times out), the backend Tomcat server sends a HTTP 401 response. However, Apache returns a 502 error to the client instead. There was no problem at all, neither on Tomcat side, nor on proxy side. Problem was relying on the network in-between. Seems that one switch/router had no MTU configured (not sure what it is, that's what the network guy said) so the communication link would brake if packet/frame negotiation failed for packets bigger than a few hundred bytes. Ok, define 'Most of the times, only the CSS and.png requests return 502'. So its working sometimes? – 4c74356b41 Feb 19 '19 at 8:25. Yes, and that boggles my.

Timeouts HowTo

Introduction

Setting communication timeouts is very important to improve thecommunication process. They help to detect problems and stabilisea distributed system. JK can use several different timeout types, whichcan be individually configured. For historical reasons, all of them aredisabled by default. This HowTo explains their use and giveshints how to find appropriate values.

All timeouts can be configured in the workers.properties file.For a complete reference of all worker configurationitems, please consult the worker reference.This page assumes, that you are using at least version 1.2.16 of JK.Dependencies on newer versions will be mentioned where necessary.

Do not set timeouts to extreme values. Very small timeouts will likelybe counterproductive.
Long Garbage Collection pauses on the backend do not make a goodfit with some timeouts. Try to optimise your Java memory and GC settings.

JK Timeout Attributes

CPing/CPong

CPing/CPong is our notion for using small test packets to check thestatus of backend connections. JK can use such test packets directly after establishinga new backend connection (connect mode) and also directly before each request getssend to a backend (prepost mode).Starting with version 1.2.27 it can also be used when a connection was idlefor a long time (interval mode).The maximum waiting time (timeout) for a CPong answer to a CPing and the idletime in interval mode can be configured.

The test packets will be answered by the backend very fast with a minimal amount ofneeded processing resources. A positive answer tells us, that the backend can be reachedand is actively processing requests. It does not detect, if some context is deployedand working. The benefit of CPing/CPong is a fast detection of a communicationproblem with the backend. The downside is a slightly increased latency.

The worker attribute ping_mode can be set to a combination of charactersto determine, in which situations test packets are used:

  • C: connect mode, timeout ping_timeout overwritten by connect_timeout
  • P: prepost mode, timeout ping_timeout overwritten by prepost_timeout
  • I: interval mode, timeout ping_timeout, idle time connection_ping_interval
  • A: all modes
Tomcat 502エラー

Multiple values must be concatenated without any separator characters.We recommend using all CPing tests. If your application is very latency sensitive, thenyou should only use the combination of connect and interval mode.

Activating the CPing probing via ping_mode has been added in version 1.2.27.For older versions only the connect and prepost modes exist and must be activated byexplicitely setting connect_timeout and prepost_timeout.

The worker attribute ping_timeout sets the default wait timeoutin milliseconds for CPong for all modes. By default the value is '10000'milliseconds. The value only gets used, if you activate CPing/Cpong probesvia ping_mode. The default value should be fine, except if you experiencevery long Java garbage collection pauses.Depending on your network latency and stability, good custom valuesoften are between 5000 and 15000 milliseconds.You can overwrite the timeout used for connect and prepost mode withconnect_timeout and prepost_timeout.Remember: don't use extremely small values.

The worker attribute connect_timeout sets the wait timeoutin milliseconds for CPong during connection establishment. You can use itif you want to overwrite the general timeout set with ping_timeout.To use connect mode CPing, you need to enable it via ping_mode.Since JK usually uses persistent connections, opening new connections is arare event. We therefore recommend activating connect mode.Depending on your network latency and stability, good values oftenare between 5000 and 15000 milliseconds.Remember: don't use extremely small values.

The worker attribute prepost_timeout sets the wait timeoutin milliseconds for CPong before request forwarding. You can use itif you want to overwrite the general timeout set with ping_timeout.To use prepost mode CPing, you need to enable it via ping_mode.Activating this type of CPing/CPong adds a small latency to eachrequest. Usually this is small enough and the benefit of CPing/CPong is more important.So in general we also recommend using prepost_timeout.Depending on your network latency and stability, good values oftenare between 5000 and 10000 milliseconds.Remember: don't use extremely small values.

Until version 1.2.27 ping_mode and ping_timeout did notexist and to enable connect or prepost mode CPing you had to set connect_timeoutrespectively prepost_timeout to some reasonable positive value.

Low-Level TCP Timeouts

Some platforms allow to set timeouts for all operations on TCP sockets.This is available for Linux and Windows, other platforms do not support this,e.g. Solaris. If your platform supports TCP send and receive timeouts,you can set them using the worker attribute socket_timeout.You can not set the two timeouts to different values.

JK will accept this attribute even if your platform does not supportsocket timeouts. In this case setting the attribute will have no effect.By default the value is '0' and the timeout is disabled.You can set the attribute to some seconds value (not: milliseconds).JK will then set the send and the receive timeouts of the backendconnections to this value. The timeout is low-level, it isused for each read and write operation on the socket individually.

Using this attribute will make JK react faster to some types of network problems.Unfortunately socket timeouts have negative side effects, because for mostplatforms, there is no good way to recover from such a timeout, once it fired.For JK there is no way to decide, if this timeout fired because of real networkproblems, or only because it didn't receive an answer packet from a backend in time.So remember: don't use extremely small values.

For the general case of connection establishment you can usesocket_connect_timeout. It takes a millisecond value and workson most platforms, even if socket_timeout is not supported.We recommend using socket_connect_timeout because in some networkfailure situations failure detection during connection establishmentcan take several minutes due to TCP retransmits. Depending on the qualityof your network a timeout somewhere between 1000 and 5000 millisecondsshould be fine. Note that socket_timeout is in seconds, andsocket_connect_timeout in milliseconds.

Connection Pools and Idle Timeouts

JK handles backend connections in a connection pool per web server process.The connections are used in a persistent mode. After a request completedsuccessfully we keep the connection open and wait for the nextrequest to forward. The connection pool is able to grow accordingto the number of threads that want to forward requests in parallel.

Most applications have a varying load depending on the hour of the dayor the day of the month. Other reasons for a growing connection poolwould be temporary slowness of backends, leading to an increasingcongestion of the frontends like web servers. Many backends use a dedicatedthread for each incoming connection they handle. So usually one wants theconnection pool to shrink, if the load diminishes.

JK allows connections in the pool to get closed after some idle time.This maximum idle time can be configured with the attributeconnection_pool_timeout which is given in units of seconds.The default value is '0', which disables closing idle connections.

We generally recommend values around 10 minutes, so settingconnection_pool_timeout to 600 (seconds). If you use this attribute,please also set the attribute keepAliveTimeout(if it is set explicitly) or connectionTimeout in the AJPConnector element of your Tomcat server.xml configuration file toan analogous value. Caution: keepAliveTimeout andconnectionTimeout must be given in milliseconds.So if you set JK connection_pool_timeout to 600, you should set TomcatkeepAliveTimeout or connectionTimeout to 600000.

Tomcat

JK connections do not get closed immediately after the timeout passed.Instead there is an automatic internal maintenance taskrunning every 60 seconds, that checks the idle status of all connections.The 60 seconds intervalcan be adjusted with the global attribute worker.maintain. We do notrecommend to change this value, because it has a lot of side effects.Until version 1.2.26, the maintenance task only runs, if requests getprocessed. So if your web server has processes that do not receive anyrequests for a long time, there is no way to close the idle connectionsin its pool. Starting with version 1.2.27 you can configure an independentwatchdog thread when using Apache HTTP Server 2.x with threaded APR or Microsoft IIS.

The maximum connection pool size can be configured with theattribute connection_pool_size. We generally do not recommendto use this attribute in combination with Apache HTTP Server. ForApache we automatically detect the number of threads perprocess and set the maximum pool size to this value. For Microsoft IIS we usea default value of 250 (before version 1.2.20: 10),for the iPlanet Web Server the default is '1'.We strongly recommend adjusting this value for IIS and the iPlanet Web Serverto the number of requests one web server process shouldbe able to send to a backend in parallel. You should measure how many connectionsyou need during peak hours without performance problems, and then add somepercentage depending on your growth rate etc. Finally you should check,whether your web server processes are able to use at least as many threads,as you configured as the pool size.

The JK attribute connection_pool_minsize defines,how many idle connections remain when the pool gets shrunken.By default this is half of the maximum pool size.

Firewall Connection Dropping

One particular problem with idle connections comes from firewalls, thatare often deployed between the web server layer and the backend.Depending on their configuration, they will silently dropconnections from their status table if they are idle for to long.

From the point of view of JK and of the web server, the other sidesimply doesn't answer any traffic. Since TCP is a reliable protocolit detects the missing TCP ACKs and tries to resend the packets fora relatively long time, typically several minutes.Therefore you should always useconnection_pool_timeout andconnection_pool_minsize on the JK side and keepAliveTimeoutor connectionTimeout on the Tomcat side to prevent idleconnection drop.

Furthermore using the boolean attribute socket_keepalive you canset a standard socket option, that automatically sends TCP keepalive packetsafter some idle time on each connection. By default this is set to false.If you suspect idle connection drops by firewalls you should set this totrue.

Unfortunately the default intervals and algorithms for these packetsare platform specific. You might need to inspect TCP tuning options foryour platform on how to control TCP keepalive.Often the default intervals are much longer than the firewall timeoutsfor idle connections. Nevertheless we recommend talking to your firewalladministration and your platform administration in order to make them agreeon good configuration values for the firewall and the platform TCP tuning.

TomcatTomcat 502

In case none of our recommendations help and you are definitively havingproblems with idle connection drops, you can disable the use of persistentconnections when using JK together with Apache HTTP Server. For this you set'JkOptions +DisableReuse' in your Apache configuration.The amount of performance impact this will have depends on the details ofyour network and your firewall.

Reply Timeout

JK can also use a timeout on request replies. This timeout does notmeasure the full processing time of the response. Instead it controls,how much time between consecutive response packets is allowed.

In most cases, this is what one actually wants. Consider for examplelong running downloads. You would not be able to set an effective globalreply timeout, because downloads could last for many minutes.Most applications though have limited processing time before startingto return the response. For those applications you could set an explicitreply timeout. Applications that do not harmonise with reply timeoutsare batch type applications, data warehouse and reporting applicationswhich are expected to observe long processing times.

Nginx Tomcat 502

If JK aborts waiting for a response, because a reply timeout fired,there is no way to stop processing on the backend. Although you freeprocessing resources in your web server, the requestwill continue to run on the backend - without any way to send back aresult once the reply timeout fired.

JK uses the worker attribute reply_timeout to set reply timeouts.The default value is '0' (timeout disabled) and you can set it to anymillisecond value.

In combination with Apache HTTP Server, you can also set a more flexible reply_timeoutusing an Apache environment variable. If you set the variable JK_REPLY_TIMEOUTto some integer value, this value will be used instead of the value inthe worker configuration. This way you can set reply timeouts more flexiblewith mod_setenvif and mod_rewrite depending on URI, query string etc.If the environment variable JK_REPLY_TIMEOUT is not set, or is set to anegative value, the default reply timeout of the worker will be used. IfJK_REPLY_TIMEOUT contains the value '0', then the reply timeout will be disabledfor the request.

Tomcat

In combination with a load balancing worker, JK will disable a memberworker of the load balancer if a reply timeout fires. The worker will thenno longer be used until it gets recovered during the next automaticmaintenance task. Starting with JK 1.2.24 you can improve this behaviour usingmax_reply_timeouts. Thisattribute will allow occasional long running requests without disabling theworker. Only if those requests happen to often, the worker gets disabled by theload balancer.

Load Balancer Error Detection

Local and Global Error States

A load balancer worker does not only have the ability to balance load.It also handles stickyness and failover of requests in case of errors.When a load balancer detects an error on one of its members, it needs todecide, whether the error is serious, or only a temporary error or maybeonly related to the actual request that was processed. Temporary errorsare called local errors, serious errors will be called global errors.

If the load balancer decides that a backend should be put into the global errorstate, then the web server will not send any more requests there. If no sessionreplication is used, this means that all user sessions located on the respectivebackend are no longer available. The users will be send to another backendand will have to login again. So the global error state is not transparent to theusers. The application is still available, but users might loose some work.

In some cases the decision between local error and global error is easy.For instance if there is an error sending back the response to the client (browser),then it is very unlikely that the backend is broken.So this situation is a typical example of a local error.

Some situations are harder to decide though. If the load balancer can't establisha new connection to a backend, it could be because of a temporary overload situation(so no more free threads in the backend), or because the backend isn't alive any more.Depending on the details, the right state could either be local error or global error.

Error Escalation Time

Tomcat 502エラー

Until version 1.2.26 most errors were interpreted as global errors.Starting with version 1.2.27 many errors which were previously interpreted as globalwere switched to being local whenever the backend is still busy. Busy means, thatother concurrent requests are send to the same backend (successful or not).

In many cases there is no perfect way of making the decisionbetween local and global error. The load balancer simply doesn't have enough information.In version 1.2.28 you can now tune, how fast the load balancer switches from local error toglobal error. If a member of a load balancer stays in local error state for too long,the load balancer will escalate it into global error state.

The time tolerated in local error state is controlled by the load balancer attributeerror_escalation_time (in seconds). The default value is half of recover_time,so unless you changed recover_time the default is 30 seconds.

Using a smaller value for error_escalation_time will make the load balancer reactfaster to serious errors, but also carries the risk of more often loosing sessionsin not so serious situations. You can lower error_escalation_time down to 0 seconds,which means all local errors which are potentially serious are escalated to global errorsimmediately.

Erreur 502 Tomcat

Note that without good basic error detection the whole escalation procedure is useless.So you should definitely use socket_connect_timeout and activate CPing/CPongwith ping_mode and ping_timeout before thinking about also tuningerror_escalation_time.

Hello,
I have a Tomcat behind IIS, and I get the 502: Bad Gateway error for long running requests.
After some experiments I've found out, that if I use connector 8082 (see below), I have this problem, but when I switch to 8084 (see below), the problem disappears. The only difference is that proxyName, proxyPort and scheme are set in the first connector.
Obviously I need both, to have those set and to serve the log running requests without error... Any idea?
Many thanks,
Petr Nemecek
<Connector port='8082' protocol='HTTP/1.1'
connectionTimeout='20000'
redirectPort='8444'
URIEncoding='UTF-8'
proxyName='www.xyz.eu'
proxyPort='80'
scheme='http' />
<Connector port='8084' protocol='HTTP/1.1'
connectionTimeout='20000'
redirectPort='8444'
URIEncoding='UTF-8' />
---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]