today we discovered teh following error in the Djigzo logfile:
JVM appears hung: Timed out waiting for signal from JVM.
JVM did not exit on request, terminated
JVM received a signal SIGKILL (9).
Launching a JVM...
java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.5) (6b24-1.11.5-0ubuntu1~10.04.2)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
spring config file: conf/spring/djigzo.xml
additional spring config dir: conf/spring/spring.d/
log4j config file: conf/log4j.properties
Obviously Djigzo decided to restart but the strange thing is that it
was working normal at this point. I have checked the logfiles and it
looks like happend some times already but with increasing frequency.
The djigzo backend is started by the Java wrapper process. The Java
wrapper process monitors the JVM to see whether the JVM is still running
and if not it will restart it. Somehow the JVM was hanging or the Java
wrapper incorrectly assumed it was hanging and restarted the backend.
I have seen this happen when the gateway runs under VMWare with
overcommitted memory and not setting all memory of the virtual machine
as reserved. If memory is overcommitted and the memory of the virtual
machine is not fully reserved the virtual machine swaps the memory to
disk which kill the Java performance.
Are you running the gateway virtualized or on real hardware?
Kind regards,
Martijn
···
On 12/05/2012 10:11 AM, lst_hoe02(a)kwsoft.de wrote:
today we discovered teh following error in the Djigzo logfile:
JVM appears hung: Timed out waiting for signal from JVM.
JVM did not exit on request, terminated
JVM received a signal SIGKILL (9).
Launching a JVM...
java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.5)
(6b24-1.11.5-0ubuntu1~10.04.2)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
spring config file: conf/spring/djigzo.xml
additional spring config dir: conf/spring/spring.d/
log4j config file: conf/log4j.properties
Obviously Djigzo decided to restart but the strange thing is that it was
working normal at this point. I have checked the logfiles and it looks
like happend some times already but with increasing frequency.
Zitat von Martijn Brinkers <martijn(a)djigzo.com>:
···
On 12/05/2012 10:11 AM, lst_hoe02(a)kwsoft.de wrote:
today we discovered teh following error in the Djigzo logfile:
JVM appears hung: Timed out waiting for signal from JVM.
JVM did not exit on request, terminated
JVM received a signal SIGKILL (9).
Launching a JVM...
java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.5)
(6b24-1.11.5-0ubuntu1~10.04.2)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
spring config file: conf/spring/djigzo.xml
additional spring config dir: conf/spring/spring.d/
log4j config file: conf/log4j.properties
Obviously Djigzo decided to restart but the strange thing is that it was
working normal at this point. I have checked the logfiles and it looks
like happend some times already but with increasing frequency.
Any idea what could be wrong?
The djigzo backend is started by the Java wrapper process. The Java
wrapper process monitors the JVM to see whether the JVM is still running
and if not it will restart it. Somehow the JVM was hanging or the Java
wrapper incorrectly assumed it was hanging and restarted the backend.
I have seen this happen when the gateway runs under VMWare with
overcommitted memory and not setting all memory of the virtual machine
as reserved. If memory is overcommitted and the memory of the virtual
machine is not fully reserved the virtual machine swaps the memory to
disk which kill the Java performance.
Are you running the gateway virtualized or on real hardware?
Kind regards,
Martijn
This is real Hardware and no memory shortage either. Djigzo was
working as expected at the time it was killed. Today where it was
noticed i was even working in the admin console, so it should be no
stall of Djigzo but more of a false restart trigger.
Although 3.4.1 is already old, I did not immediately see a bug fix which
might explain what happens. This looks the closest:
Version 3.5.9:
Fix a problem where the Wrapper was not correctly yielding control back
to its main loop when very large amounts of continuous output was being
logged from the JVM. Introduced in 3.4.0. In versions prior to 3.5.8,
this could have caused the JVM to timeout and restart itself. That
particular issue was resolved but the Wrapper process in 3.5.8 would
still have been unresponsive when this was happening. The Wrapper will
now always yeild back to its main loop after 250 milliseconds of
continuous logging.
You might try to use a newer Java wrapper release to see whether this
fixes the problem. The Java wrapper files are stored in:
/usr/share/djigzo/wrapper
The wrapper tar is extracted to a sub dir and then the following
symlinks link to the required files inside the version specific sub dir:
If you want to test with a newer release of Java wrapper, untar the file
and overwrite the symlinks and then restart djigzo.
Kind regards,
Martijn
···
On 12/05/2012 04:07 PM, lst_hoe02(a)kwsoft.de wrote:
Are you running the gateway virtualized or on real hardware?
This is real Hardware and no memory shortage either. Djigzo was working
as expected at the time it was killed. Today where it was noticed i was
even working in the admin console, so it should be no stall of Djigzo
but more of a false restart trigger.
Zitat von Martijn Brinkers <martijn(a)djigzo.com>:
···
On 12/05/2012 04:07 PM, lst_hoe02(a)kwsoft.de wrote:
Are you running the gateway virtualized or on real hardware?
This is real Hardware and no memory shortage either. Djigzo was working
as expected at the time it was killed. Today where it was noticed i was
even working in the admin console, so it should be no stall of Djigzo
but more of a false restart trigger.
Strange. I have never seen this problem on my own servers (well at least
not that I know, since it automatically restarts
I have checked the Java Wrapper releases notes at:
Although 3.4.1 is already old, I did not immediately see a bug fix which
might explain what happens. This looks the closest:
Version 3.5.9:
Fix a problem where the Wrapper was not correctly yielding control back
to its main loop when very large amounts of continuous output was being
logged from the JVM. Introduced in 3.4.0. In versions prior to 3.5.8,
this could have caused the JVM to timeout and restart itself. That
particular issue was resolved but the Wrapper process in 3.5.8 would
still have been unresponsive when this was happening. The Wrapper will
now always yeild back to its main loop after 250 milliseconds of
continuous logging.
You might try to use a newer Java wrapper release to see whether this
fixes the problem. The Java wrapper files are stored in:
/usr/share/djigzo/wrapper
The wrapper tar is extracted to a sub dir and then the following
symlinks link to the required files inside the version specific sub dir:
On 12/05/2012 04:35 PM, lst_hoe02(a)kwsoft.de wrote:
Zitat von Martijn Brinkers <martijn(a)djigzo.com>:
On 12/05/2012 04:07 PM, lst_hoe02(a)kwsoft.de wrote:
Are you running the gateway virtualized or on real hardware?
This is real Hardware and no memory shortage either. Djigzo was working
as expected at the time it was killed. Today where it was noticed i was
even working in the admin console, so it should be no stall of Djigzo
but more of a false restart trigger.
Strange. I have never seen this problem on my own servers (well at least
not that I know, since it automatically restarts
I have checked the Java Wrapper releases notes at:
Although 3.4.1 is already old, I did not immediately see a bug fix which
might explain what happens. This looks the closest:
Version 3.5.9:
Fix a problem where the Wrapper was not correctly yielding control back
to its main loop when very large amounts of continuous output was being
logged from the JVM. Introduced in 3.4.0. In versions prior to 3.5.8,
this could have caused the JVM to timeout and restart itself. That
particular issue was resolved but the Wrapper process in 3.5.8 would
still have been unresponsive when this was happening. The Wrapper will
now always yeild back to its main loop after 250 milliseconds of
continuous logging.
You might try to use a newer Java wrapper release to see whether this
fixes the problem. The Java wrapper files are stored in:
/usr/share/djigzo/wrapper
The wrapper tar is extracted to a sub dir and then the following
symlinks link to the required files inside the version specific sub dir:
Zitat von Martijn Brinkers <martijn(a)djigzo.com>:
···
On 12/05/2012 04:35 PM, lst_hoe02(a)kwsoft.de wrote:
Zitat von Martijn Brinkers <martijn(a)djigzo.com>:
On 12/05/2012 04:07 PM, lst_hoe02(a)kwsoft.de wrote:
Are you running the gateway virtualized or on real hardware?
This is real Hardware and no memory shortage either. Djigzo was working
as expected at the time it was killed. Today where it was noticed i was
even working in the admin console, so it should be no stall of Djigzo
but more of a false restart trigger.
Strange. I have never seen this problem on my own servers (well at least
not that I know, since it automatically restarts
I have checked the Java Wrapper releases notes at:
Although 3.4.1 is already old, I did not immediately see a bug fix which
might explain what happens. This looks the closest:
Version 3.5.9:
Fix a problem where the Wrapper was not correctly yielding control back
to its main loop when very large amounts of continuous output was being
logged from the JVM. Introduced in 3.4.0. In versions prior to 3.5.8,
this could have caused the JVM to timeout and restart itself. That
particular issue was resolved but the Wrapper process in 3.5.8 would
still have been unresponsive when this was happening. The Wrapper will
now always yeild back to its main loop after 250 milliseconds of
continuous logging.
You might try to use a newer Java wrapper release to see whether this
fixes the problem. The Java wrapper files are stored in:
/usr/share/djigzo/wrapper
The wrapper tar is extracted to a sub dir and then the following
symlinks link to the required files inside the version specific sub dir:
Zitat von Martijn Brinkers <martijn(a)djigzo.com>:
Zitat von Martijn Brinkers <martijn(a)djigzo.com>:
Are you running the gateway virtualized or on real hardware?
This is real Hardware and no memory shortage either. Djigzo was working
as expected at the time it was killed. Today where it was noticed i was
even working in the admin console, so it should be no stall of Djigzo
but more of a false restart trigger.
Strange. I have never seen this problem on my own servers (well at least
not that I know, since it automatically restarts
I have checked the Java Wrapper releases notes at:
Although 3.4.1 is already old, I did not immediately see a bug fix which
might explain what happens. This looks the closest:
Version 3.5.9:
Fix a problem where the Wrapper was not correctly yielding control back
to its main loop when very large amounts of continuous output was being
logged from the JVM. Introduced in 3.4.0. In versions prior to 3.5.8,
this could have caused the JVM to timeout and restart itself. That
particular issue was resolved but the Wrapper process in 3.5.8 would
still have been unresponsive when this was happening. The Wrapper will
now always yeild back to its main loop after 250 milliseconds of
continuous logging.
You might try to use a newer Java wrapper release to see whether this
fixes the problem. The Java wrapper files are stored in:
/usr/share/djigzo/wrapper
The wrapper tar is extracted to a sub dir and then the following
symlinks link to the required files inside the version specific sub dir:
to the file /usr/share/djigzo/wrapper/djigzo.wrapper.conf
and then restart djigzo
You will now see the ping's to the JVM and hopefully some more
information about why it restarts.
Kind regards,
Martijn
···
On 12/05/2012 05:34 PM, lst_hoe02(a)kwsoft.de wrote:
Zitat von Martijn Brinkers <martijn(a)djigzo.com>:
On 12/05/2012 04:35 PM, lst_hoe02(a)kwsoft.de wrote:
Zitat von Martijn Brinkers <martijn(a)djigzo.com>:
On 12/05/2012 04:07 PM, lst_hoe02(a)kwsoft.de wrote:
Are you running the gateway virtualized or on real hardware?
This is real Hardware and no memory shortage either. Djigzo was
working
as expected at the time it was killed. Today where it was noticed i
was
even working in the admin console, so it should be no stall of Djigzo
but more of a false restart trigger.
Strange. I have never seen this problem on my own servers (well at
least
not that I know, since it automatically restarts
I have checked the Java Wrapper releases notes at:
Although 3.4.1 is already old, I did not immediately see a bug fix
which
might explain what happens. This looks the closest:
Version 3.5.9:
Fix a problem where the Wrapper was not correctly yielding control back
to its main loop when very large amounts of continuous output was being
logged from the JVM. Introduced in 3.4.0. In versions prior to 3.5.8,
this could have caused the JVM to timeout and restart itself. That
particular issue was resolved but the Wrapper process in 3.5.8 would
still have been unresponsive when this was happening. The Wrapper will
now always yeild back to its main loop after 250 milliseconds of
continuous logging.
You might try to use a newer Java wrapper release to see whether this
fixes the problem. The Java wrapper files are stored in:
/usr/share/djigzo/wrapper
The wrapper tar is extracted to a sub dir and then the following
symlinks link to the required files inside the version specific sub
dir:
Downloading of CRLs are done on a separate thread. The only think I can
think of is that downloading the CRL somehow uses all the CPU for some
time although that should not happen since all threads have the same
priority. Do you have the certificate that contains that CRL
distribution point so I can test whether I can download the CRL?
Kind regards,
Martijn
···
On 12/05/2012 05:44 PM, lst_hoe02(a)kwsoft.de wrote:
Is it possible that it is related to the CRL downloading. Most of the
time we have something like this nearby:
Downloading of CRLs are done on a separate thread. The only think I can
think of is that downloading the CRL somehow uses all the CPU for some
time although that should not happen since all threads have the same
priority. Do you have the certificate that contains that CRL
distribution point so I can test whether I can download the CRL?
Kind regards,
Martijn
There is more than one with crappy CRL download URL. I will try to
pick them and send you in private mail. There is obviously something
wrong with the CRLs anyway:
/var/log/old/james.wrapper.log.1:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: GC overhead limit exceeded
/var/log/old/james.wrapper.log.1:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.1:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.1:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.1:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.1:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.1:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.2.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: GC overhead limit exceeded
/var/log/old/james.wrapper.log.2.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.2.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: GC overhead limit exceeded
/var/log/old/james.wrapper.log.3.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.3.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.3.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.4.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.4.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
I have now raised the Memory Limit in wrapper.conf from 512MB to 700MB
to see if this fixes at least the OutOfMemory
And yes, one CPU is at 100% load for the time of CRL updates but at
least we have two of them
Regards
Andreas
···
On 12/05/2012 05:44 PM, lst_hoe02(a)kwsoft.de wrote:
Downloading of CRLs are done on a separate thread. The only think I can
think of is that downloading the CRL somehow uses all the CPU for some
time although that should not happen since all threads have the same
priority. Do you have the certificate that contains that CRL
distribution point so I can test whether I can download the CRL?
Kind regards,
Martijn
There is more than one with crappy CRL download URL. I will try to pick
them and send you in private mail. There is obviously something wrong
with the CRLs anyway:
/var/log/old/james.wrapper.log.1:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: GC overhead limit exceeded
/var/log/old/james.wrapper.log.1:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.1:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.1:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.1:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.1:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.1:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.2.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: GC overhead limit exceeded
/var/log/old/james.wrapper.log.2.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.2.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: GC overhead limit exceeded
/var/log/old/james.wrapper.log.3.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.3.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.3.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.4.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.4.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
I have now raised the Memory Limit in wrapper.conf from 512MB to 700MB
to see if this fixes at least the OutOfMemory
And yes, one CPU is at 100% load for the time of CRL updates but at
least we have two of them
That might explain it. Unfortunately CRLs can be large and memory
consuming. I think 512 MB is too low for some large CRLs. For the
Virtual Appliance I have enabled "dynamic memory allocation" which uses
more memory if the system is configured with more memory. It's really
simple but effective (see /etc/default/djigzo). This is not enabled by
default since djigzo might be installed on systems that run other things
as well.
This is a warning from the JVM that the system spends more time on
garbage collection then some max value. I think that the system is then
restarted. Somehow you have a CA with a very large CRL. The system sets
a max size of a downloaded CRL (see djigzo.properties parameter:
system.cRLDownloadParameters.maxBytes). You can lower this value, but
then it might be that a CRL is no longer downloaded since it's too big.
I'll need to spend time refactoring the CRL part. The problem is that
Java tries to load the CRL in memory which can take a lot of memory. The
thing I have been thinking of for some time is to make a database backed
CRL. But this is not trivial to do.
Can you see which CRL is giving you grieve?
Kind regards,
Martijn
···
On 12/05/2012 06:03 PM, lst_hoe02(a)kwsoft.de wrote:
On 12/05/2012 05:44 PM, lst_hoe02(a)kwsoft.de wrote:
Downloading of CRLs are done on a separate thread. The only think I can
think of is that downloading the CRL somehow uses all the CPU for some
time although that should not happen since all threads have the same
priority. Do you have the certificate that contains that CRL
distribution point so I can test whether I can download the CRL?
Kind regards,
Martijn
There is more than one with crappy CRL download URL. I will try to pick
them and send you in private mail. There is obviously something wrong
with the CRLs anyway:
/var/log/old/james.wrapper.log.1:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: GC overhead limit exceeded
/var/log/old/james.wrapper.log.1:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.1:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.1:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.1:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.1:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.1:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.2.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: GC overhead limit exceeded
/var/log/old/james.wrapper.log.2.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.2.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: GC overhead limit exceeded
/var/log/old/james.wrapper.log.3.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.3.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.3.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.4.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.4.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
I have now raised the Memory Limit in wrapper.conf from 512MB to 700MB
to see if this fixes at least the OutOfMemory
And yes, one CPU is at 100% load for the time of CRL updates but at
least we have two of them
That might explain it. Unfortunately CRLs can be large and memory
consuming. I think 512 MB is too low for some large CRLs. For the
Virtual Appliance I have enabled "dynamic memory allocation" which uses
more memory if the system is configured with more memory. It's really
simple but effective (see /etc/default/djigzo). This is not enabled by
default since djigzo might be installed on systems that run other things
as well.
This is a warning from the JVM that the system spends more time on
garbage collection then some max value. I think that the system is then
restarted. Somehow you have a CA with a very large CRL. The system sets
a max size of a downloaded CRL (see djigzo.properties parameter:
system.cRLDownloadParameters.maxBytes). You can lower this value, but
then it might be that a CRL is no longer downloaded since it's too big.
I'll need to spend time refactoring the CRL part. The problem is that
Java tries to load the CRL in memory which can take a lot of memory. The
thing I have been thinking of for some time is to make a database backed
CRL. But this is not trivial to do.
Can you see which CRL is giving you grieve?
Kind regards,
Martijn
I don't know how to quickly find out the size of the CRLs. I can
provide you a sample list with all CAs for which CRL trouble is
listed. What about CRLs from non-root CAs??
Downloading of CRLs are done on a separate thread. The only think I can
think of is that downloading the CRL somehow uses all the CPU for some
time although that should not happen since all threads have the same
priority. Do you have the certificate that contains that CRL
distribution point so I can test whether I can download the CRL?
Kind regards,
Martijn
There is more than one with crappy CRL download URL. I will try to pick
them and send you in private mail. There is obviously something wrong
with the CRLs anyway:
/var/log/old/james.wrapper.log.1:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: GC overhead limit exceeded
/var/log/old/james.wrapper.log.1:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.1:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.1:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.1:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.1:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.1:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.2.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: GC overhead limit exceeded
/var/log/old/james.wrapper.log.2.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.2.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: GC overhead limit exceeded
/var/log/old/james.wrapper.log.3.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.3.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.3.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.4.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
/var/log/old/james.wrapper.log.4.gz:Exception in thread "CRL Updater
thread" java.lang.OutOfMemoryError: Java heap space
I have now raised the Memory Limit in wrapper.conf from 512MB to 700MB
to see if this fixes at least the OutOfMemory
And yes, one CPU is at 100% load for the time of CRL updates but at
least we have two of them
That might explain it. Unfortunately CRLs can be large and memory
consuming. I think 512 MB is too low for some large CRLs. For the
Virtual Appliance I have enabled "dynamic memory allocation" which uses
more memory if the system is configured with more memory. It's really
simple but effective (see /etc/default/djigzo). This is not enabled by
default since djigzo might be installed on systems that run other things
as well.
This is a warning from the JVM that the system spends more time on
garbage collection then some max value. I think that the system is then
restarted. Somehow you have a CA with a very large CRL. The system sets
a max size of a downloaded CRL (see djigzo.properties parameter:
system.cRLDownloadParameters.maxBytes). You can lower this value, but
then it might be that a CRL is no longer downloaded since it's too big.
I'll need to spend time refactoring the CRL part. The problem is that
Java tries to load the CRL in memory which can take a lot of memory. The
thing I have been thinking of for some time is to make a database backed
CRL. But this is not trivial to do.
Can you see which CRL is giving you grieve?
Kind regards,
Martijn
I don't know how to quickly find out the size of the CRLs. I can provide
you a sample list with all CAs for which CRL trouble is listed. What
about CRLs from non-root CAs??
