[Unicode]   Common Locale Data Repository : Bug Tracking Home | Site Map | Search
 
Modify

CLDR Ticket #11290(accepted unknown)

Opened 4 weeks ago

Last modified 6 days ago

Survey tool log

Reported by: kristi Owned by: tbishop
Component: unknown Data Locale:
Phase: rc Review:
Weeks: Data Xpath:
Xref:

Description

Survey tool log backup keeps bringing down the tool

Attachments

Change History

comment:1 Changed 4 weeks ago by kristi

  • Owner changed from anybody to tbishop
  • Status changed from new to accepted

comment:2 Changed 4 weeks ago by emmons

~/tomcat/logs is the location of the logs

comment:3 Changed 4 weeks ago by tbishop

See also 10295

comment:4 Changed 4 weeks ago by tbishop

Another ginormous log file: /home/surveytool/tomcat/cldr/exception.log -- 14,253,911,269 bytes

surveytool@CLDRUBS14-PROD2:~/tomcat$ head /home/surveytool/tomcat/cldr/exception.log
!!!***!!! SURVEY_EXCEPTION Including template abbreviated_days.jsp
*** DATE 1526579804978 Thu May 17 17:56:44 UTC 2018
*** REVISION 14017 34 SUBMISSION PRODUCTION
*** UPTIME uptime: 36 sec.
*** LOGSITE org.unicode.cldr.web.SurveyLog.logException(SurveyLog.java:96)
org.unicode.cldr.web.SurveyLog.logException(SurveyLog.java:57)
org.unicode.cldr.web.WebContext.includeFragment(WebContext.java:1813)
org.apache.jsp.WEB_002dINF.tmpl.r_005fsteps_jsp._jspService(r_005fsteps_jsp.java:432)
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
javax.servlet.http.HttpServlet.service(HttpServlet.java:729)

surveytool@CLDRUBS14-PROD2:~/tomcat$ tail /home/surveytool/tomcat/cldr/exception.log
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:521)
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096)
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:674)
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
java.lang.Thread.run(Thread.java:748)
Last edited 4 weeks ago by tbishop (previous) (diff)

comment:5 Changed 2 weeks ago by tbishop

Another huge file is /home/surveytool/tomcat/logs/catalina.out which has grown to more than 1GB in less than a month.

cldradmin@st:~$ df -hT /home

Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sda1      ext4   29G   13G   16G  44% /

cldradmin@st:~$ ls -l /home/surveytool/tomcat/logs
total 1980700
-rw-r--r-- 1 surveytool cldr 166428759 Aug 4 23:59 catalina.2018-08-04.log
-rw-r--r-- 1 surveytool cldr 10357373 Aug 5 02:20 catalina.2018-08-05.log
-rw-r--r-- 1 surveytool cldr 1186514823 Aug 5 04:40 catalina.out
-rw-r--r-- 1 surveytool cldr 47248832 Aug 4 23:59 localhost.2018-08-04.log
-rw-r--r-- 1 surveytool cldr 2940552 Aug 5 01:17 localhost.2018-08-05.log
-rw-r--r-- 1 surveytool cldr 23369809 Jul 27 00:00 localhost_access_log.2018-07-26.txt
-rw-r--r-- 1 surveytool cldr 14907224 Jul 27 23:59 localhost_access_log.2018-07-27.txt
-rw-r--r-- 1 surveytool cldr 5307756 Jul 29 00:00 localhost_access_log.2018-07-28.txt
-rw-r--r-- 1 surveytool cldr 5753515 Jul 30 00:00 localhost_access_log.2018-07-29.txt
-rw-r--r-- 1 surveytool cldr 14208799 Jul 30 23:59 localhost_access_log.2018-07-30.txt
-rw-r--r-- 1 surveytool cldr 4758439 Jul 31 23:59 localhost_access_log.2018-07-31.txt
-rw-r--r-- 1 surveytool cldr 6620413 Aug 2 00:00 localhost_access_log.2018-08-01.txt
-rw-r--r-- 1 surveytool cldr 5123281 Aug 3 00:00 localhost_access_log.2018-08-02.txt
-rw-r--r-- 1 surveytool cldr 3979203 Aug 3 23:59 localhost_access_log.2018-08-03.txt
-rw-r--r-- 1 surveytool cldr 5432247 Aug 5 00:00 localhost_access_log.2018-08-04.txt
-rw-r--r-- 1 surveytool cldr 788689 Aug 5 04:41 localhost_access_log.2018-08-05.txt
-rw-r--r-- 1 root root 52437838 Jul 30 11:26 tomcat.log.1
-rw-r--r-- 1 root root 52429677 Jul 28 12:41 tomcat.log.10
-rw-r--r-- 1 root root 52430148 Jul 30 06:26 tomcat.log.2
-rw-r--r-- 1 root root 52428923 Jul 30 01:54 tomcat.log.3
-rw-r--r-- 1 root root 52435900 Jul 29 20:53 tomcat.log.4
-rw-r--r-- 1 root root 52437562 Jul 29 15:10 tomcat.log.5
-rw-r--r-- 1 root root 52438700 Jul 29 09:46 tomcat.log.6
-rw-r--r-- 1 root root 52429821 Jul 29 04:56 tomcat.log.7
-rw-r--r-- 1 root root 52436076 Jul 28 23:27 tomcat.log.8
-rw-r--r-- 1 root root 52431124 Jul 28 18:02 tomcat.log.9
cldradmin@st:~$ head /home/surveytool/tomcat/logs/catalina.out
10-Jul-2018 13:53:30.909 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version: Apache Tomcat/8.0.30
10-Jul-2018 13:53:30.913 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Dec 1 2015 22:30:46 UTC
10-Jul-2018 13:53:30.913 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server number: 8.0.30.0
10-Jul-2018 13:53:30.914 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Linux
10-Jul-2018 13:53:30.914 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 4.4.0-71-generic
10-Jul-2018 13:53:30.914 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64
10-Jul-2018 13:53:30.914 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: /usr/lib/jvm/java-8-oracle/jre
10-Jul-2018 13:53:30.914 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 1.8.0_161-b12
10-Jul-2018 13:53:30.914 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: Oracle Corporation
10-Jul-2018 13:53:30.914 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: /home/surveytool/tomcat
cldradmin@st:~$ tail /home/surveytool/tomcat/logs/catalina.out
MailSender: processing mail queue
MailSender: processing mail queue
MailSender: processing mail queue
MailSender: processing mail queue
MailSender: processing mail queue
MailSender: processing mail queue
MailSender: processing mail queue
MailSender: processing mail queue
MailSender: processing mail queue
MailSender: processing mail queue

cldradmin@st:~$ sudo rm /home/surveytool/tomcat/logs/catalina.out
cldradmin@st:~$ ls -l /home/surveytool/tomcat/cldr/exception.log
-rw-r--r-- 1 surveytool cldr 234185468 Aug 5 01:17 /home/surveytool/tomcat/cldr/exception.log
cldradmin@st:~$ head /home/surveytool/tomcat/cldr/exception.log
!!***!!! SURVEY_EXCEPTION Including template eras.jsp
...
cldradmin@st:~$ sudo rm /home/surveytool/tomcat/cldr/exception.log

comment:6 Changed 2 weeks ago by tbishop

Plan something like this. Create file named: /etc/logrotate.d/surveytool-logs

Contents of that file:

/home/surveytool/tomcat/cldr/exception.log {
    daily
    missingok
    rotate 14
    compress
    notifempty
    dateext
    create 0640 surveytool cldr
}

Plus similar entries, maybe in the same file, for /home/surveytool/tomcat/logs/catalina.out and maybe everything in the logs folder.

comment:7 Changed 2 weeks ago by tbishop

I created a file /etc/logrotate.d/surveytool-logs with these contents:

/home/surveytool/tomcat/cldr/exception.log /home/surveytool/tomcat/logs/catalina.out {
    daily
    rotate 7
    compress
    missingok
    notifempty
    dateext
    create 0640 surveytool cldr
}

I did:

sudo chown root:root /etc/logrotate.d/surveytool-logs
sudo chmod 644 /etc/logrotate.d/surveytool-logs

Still to do:

  • monitor operation for the next few days
  • rotate other files in /home/surveytool/tomcat/logs
  • rsync to a different location

comment:8 Changed 12 days ago by tbishop

Log rotation appears to be working now for /home/surveytool/tomcat/cldr/exception.log; we have newly-created exception.log-20180807.gz.

However, there is no /home/surveytool/tomcat/logs/catalina.out yet, nor any rotated version of it. Maybe it's necessary to restart tomcat, possibly with prerotate and postrotate sections? For example:

prerotate

/home/surveytool/tomcat/bin/shutdown.sh

endscript
postrotate

/home/surveytool/tomcat/bin/startup.sh

endscript

Alternatively it might suffice to add "copytruncate". "...It can be used when some program cannot be told to close its logfile and thus might continue writing (appending) to the previous log file forever. Note that there is a very small time slice between copying the file and truncating it, so some logging data might be lost...."

Last edited 12 days ago by tbishop (previous) (diff)

comment:9 Changed 11 days ago by tbishop

I added copytruncate, and ran shutdown.sh and startup.sh. A new catalina.out appeared. I'll keep an eye on it.

comment:10 Changed 10 days ago by tbishop

Update:

cldradmin@st:~$ ls -l /home/surveytool/tomcat/logs
...
-rw-r--r-- 1 surveytool cldr 126586766 Aug 9 14:01 catalina.out
-rw-r--r-- 1 surveytool cldr 2048692 Aug 9 06:24 catalina.out-20180809.gz
...

cldradmin@st:~$ ls -l /home/surveytool/tomcat/cldr
...
-rw-r----- 1 surveytool cldr 94294574 Aug 9 14:02 exception.log
-rw-r--r-- 1 surveytool cldr 3012039 Aug 7 06:24 exception.log-20180807.gz
-rw-r----- 1 surveytool cldr 3103955 Aug 8 06:24 exception.log-20180808.gz
-rw-r----- 1 surveytool cldr 3661160 Aug 9 06:24 exception.log-20180809.gz
...

comment:11 Changed 6 days ago by tbishop

Usage of /: 50.6% of 28.80GB
cldradmin@st:~$ ls -l /home/surveytool/tomcat/logs
...
-rw-r--r-- 1 surveytool cldr 75866227 Aug 14 12:44 catalina.out
-rw-r--r-- 1 surveytool cldr 2048692 Aug 9 06:24 catalina.out-20180809.gz
-rw-r--r-- 1 surveytool cldr 5406026 Aug 10 06:24 catalina.out-20180810.gz
-rw-r--r-- 1 surveytool cldr 4621011 Aug 11 06:24 catalina.out-20180811.gz
-rw-r--r-- 1 surveytool cldr 3815213 Aug 12 06:24 catalina.out-20180812.gz
-rw-r--r-- 1 surveytool cldr 4077717 Aug 13 06:24 catalina.out-20180813.gz
-rw-r--r-- 1 surveytool cldr 3459016 Aug 14 06:24 catalina.out-20180814.gz
...
cldradmin@st:~$ ls -l /home/surveytool/tomcat/cldr
...
-rw-r----- 1 surveytool cldr 56673641 Aug 14 12:45 exception.log
-rw-r----- 1 surveytool cldr 3103955 Aug 8 06:24 exception.log-20180808.gz
-rw-r----- 1 surveytool cldr 3661160 Aug 9 06:24 exception.log-20180809.gz
-rw-r----- 1 surveytool cldr 3371090 Aug 10 06:24 exception.log-20180810.gz
-rw-r----- 1 surveytool cldr 2899152 Aug 11 06:24 exception.log-20180811.gz
-rw-r----- 1 surveytool cldr 2469279 Aug 12 06:24 exception.log-20180812.gz
-rw-r----- 1 surveytool cldr 2530489 Aug 13 06:24 exception.log-20180813.gz
-rw-r----- 1 surveytool cldr 2198054 Aug 14 06:24 exception.log-20180814.gz
...

View

Add a comment

Modify Ticket

Action
as accepted
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.