[Unicode]   Common Locale Data Repository : Bug Tracking Home | Site Map | Search
 
Modify

CLDR Ticket #5062(accepted defect)

Opened 5 years ago

Last modified 20 months ago

Smoketest resolves parent locale against CLDR_OLDVERSION data

Reported by: pedberg Owned by: srl
Component: survey Data Locale:
Phase: dsub Review:
Weeks: Data Xpath:
Xref:

Description

This arises from my attempt to address cldrbug 4745: as originally intended, which was to move the contents of en_BE (which had parent en_GB) to a new en_150 (which has en_GB as a parent), and making en_150 the parent of en_GB. This worked fine when tested with ConsoleCheckCLDR. Testing in SmokeTest, en_150 worked fine, but in en_BE, trying to look at (for example) Gregorian calendar data resulted in this:

Exception on getSection:java.lang.NullPointerException:
{org.unicode.cldr.util.SimpleFactory dirs=/home/st-bitten/src/tomcat/cldr/21/common/main /home/st-bitten/src/tomcat/cldr/21/seed/main }.handleMake
returned a null CLDRFile for en_150

The specific reason is that the xmlFile.canRead() in SimpleFactory.getSourceDirectoryForLocale("en_150") fails because en_150.xml does not exist in /home/st-bitten/src/tomcat/cldr/21/common/main; per Steven the 21 in the path comes from CLDR_OLDVERSION in smoke test's cldr/cldr.properties

I am not sure why it is looking for parent locales in the previous CLDR data, but that makes it impossible to add new parents as in this case.

Attachments

Change History

comment:1 Changed 5 years ago by srl

The Survey Tool uses three factories: 1. CLDR_OLDVERSION (21) data, for last release .. 2. current disk data (trunk) .. and 3, a "STFactory" which is a view onto the vetted results, based on the current disk data, but also based on voting results taking the CLDR_OLDVERSION data into account.

The Survey Tool is only "looking for parent locales in the previous CLDR data" when it is attempting to resolve previous CLDR Data. The stack trace shows that it is calling Factory.make from within getOldFile (which references CLDR_OLDVERSION data). So it is simply attempting to call make("en_BE") against the version 21 factory - not against the current data factory.

As Mark pointed out, this means that we need to not make parentage changes while the Survey Tool ( or, I might add, any tooling that would need to read a previous or future version's data ) is still open, until this bug is fixed. This seems to be a factory bug.

 on loading en_BE:null

1342138046817 Thu Jul 12 19:07:26 CDT 2012
uptime: 57 sec.

MESSAGE

java.lang.NullPointerException: # SimpleFactory.getSourceDirectoryForLocale en_150, can't read file, isSupplemental false, dir /home/st-bitten/src/tomcat/cldr/21/common/main, xmlFile /home/st-bitten/src/tomcat/cldr/21/common/main/en_150.xml # SimpleFactory.getSourceDirectoryForLocale en_150, can't read file, isSupplemental false, dir /home/st-bitten/src/tomcat/cldr/21/common/main, xmlFile /home/st-bitten/src/tomcat/cldr/21/common/main/en_150.xml

STACK

org.unicode.cldr.util.SimpleFactory.getSourceDirectoryForLocale(SimpleFactory.java:242)
org.unicode.cldr.util.SimpleFactory.handleMake(SimpleFactory.java:122)
org.unicode.cldr.util.Factory.makeResolvingSource(Factory.java:138)
org.unicode.cldr.util.SimpleFactory.handleMake(SimpleFactory.java:117)
org.unicode.cldr.util.Factory.make(Factory.java:90)
org.unicode.cldr.util.Factory.make(Factory.java:98)
org.unicode.cldr.web.STFactory$PerLocaleData.getOldFile(STFactory.java:417)
org.unicode.cldr.web.STFactory.getOldFile(STFactory.java:1214)
org.unicode.cldr.web.DataSection.populateFrom(DataSection.java:2781)
org.unicode.cldr.web.DataSection.make(DataSection.java:2318)
org.unicode.cldr.web.WebContext.getSection(WebContext.java:1497)
org.unicode.cldr.web.WebContext.getSection(WebContext.java:1438)
org.apache.jsp.RefreshRow_jsp._jspService(RefreshRow_jsp.java:150)
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390)
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334)
javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
org.apache.catalina.filters.ExpiresFilter.doFilter(ExpiresFilter.java:1179)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:200)
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:539)
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:298)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:636)

LOGSITE

org.unicode.cldr.web.SurveyLog.logException(SurveyLog.java:104)
org.unicode.cldr.web.SurveyLog.logException(SurveyLog.java:64)
org.apache.jsp.RefreshRow_jsp._jspService(RefreshRow_jsp.java:161)
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390)
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334)
javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
org.apache.catalina.filters.ExpiresFilter.doFilter(ExpiresFilter.java:1179)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:200)
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:539)
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:298)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:636)

comment:2 Changed 5 years ago by mark

  • Cc mark added
  • Owner changed from anybody to srl
  • Priority changed from assess to critical
  • Status changed from new to assigned
  • Milestone changed from UNSCH to 23dsub

comment:3 Changed 4 years ago by emmons

  • Milestone changed from 23dsub to 23dres

comment:4 Changed 4 years ago by emmons

  • Milestone changed from 23dres to 24dsub

Moving all Steven's 23dres to 24dsub

comment:5 Changed 4 years ago by srl

Basically need to make sure the OldFactory uses old supplemental data to control its fallback. Make sure this has a unit test.

comment:6 Changed 4 years ago by srl

  • Milestone changed from 24dsub to 24dsub2

not dsub1

comment:7 Changed 4 years ago by srl

  • Milestone changed from 24dsub2 to 24dvet

triage

comment:8 Changed 4 years ago by srl

  • Milestone changed from 24dvet to 25dsub

comment:9 Changed 3 years ago by emmons

  • Milestone changed from 25dsub to 25M1

Moving all 25dsub to 25M1. Please adjust the milestone if you are not planning to complete the item in the 25M1 time frame.

comment:10 Changed 3 years ago by srl

  • Milestone changed from 25M1 to 25rc

comment:11 Changed 3 years ago by srl

  • Milestone changed from 25rc to 26dsub

rolling

comment:12 Changed 3 years ago by emmons

  • Milestone changed from 26dsub to 26rc

comment:13 Changed 3 years ago by srl

  • Milestone changed from 26rc to 27dsub

roll

comment:14 Changed 3 years ago by markus

  • Phase set to dsub
  • Milestone changed from 27dsub to 27

comment:15 Changed 2 years ago by srl

  • Milestone changed from 27 to 28

roll

comment:16 Changed 2 years ago by srl

  • Status changed from assigned to accepted

comment:17 Changed 2 years ago by srl

Tested by making en_MT inherit from (a new ) en_054

Index: common/supplemental/supplementalData.xml
===================================================================
--- common/supplemental/supplementalData.xml	(revision 11723)
+++ common/supplemental/supplementalData.xml	(working copy)
@@ -5163,10 +5163,12 @@
 
 	<parentLocales>
 		<parentLocale parent="root" locales="az_Cyrl bm_Nkoo bs_Cyrl en_Dsrt en_Shaw ha_Arab iu_Latn mn_Mong ms_Arab pa_Arab shi_Latn sr_Latn uz_Arab uz_Cyrl vai_Latn zh_Hant"/>
-		<parentLocale parent="en_001" locales="en_150 en_AG en_AI en_AU en_BB en_BE en_BM en_BS en_BW en_BZ en_CA en_CC en_CK en_CM en_CX en_DG en_DM en_ER en_FJ en_FK en_FM en_GB en_GD en_GG en_GH en_GI en_GM en_GY en_HK en_IE en_IM en_IN en_IO en_JE en_JM en_KE en_KI en_KN en_KY en_LC en_LR en_LS en_MG en_MO en_MS en_MT en_MU en_MW en_MY en_NA en_NF en_NG en_NR en_NU en_NZ en_PG en_PH en_PK en_PN en_PW en_RW en_SB en_SC en_SD en_SG en_SH en_SL en_SS en_SX en_SZ en_TC en_TK en_TO en_TT en_TV en_TZ en_UG en_VC en_VG en_VU en_WS en_ZA en_ZM en_ZW"/>
+		<parentLocale parent="en_001" locales="en_150 en_AG en_AI en_AU en_BB en_BE en_BM en_BS en_BW en_BZ en_CA en_CC en_CK en_CM en_CX en_DG en_DM en_ER en_FJ en_FK en_FM en_GB en_GD en_GG en_GH en_GI en_GM en_GY en_HK en_IE en_IM en_IN en_IO en_JE en_JM en_KE en_KI en_KN en_KY en_LC en_LR en_LS en_MG en_MO en_MS en_MU en_MW en_MY en_NA en_NF en_NG en_NR en_NU en_NZ en_PG en_PH en_PK en_PN en_PW en_RW en_SB en_SC en_SD en_SG en_SH en_SL en_SS en_SX en_SZ en_TC en_TK en_TO en_TT en_TV en_TZ en_UG en_VC en_VG en_VU en_WS en_ZA en_ZM en_ZW"/>
 		<parentLocale parent="es_419" locales="es_AR es_BO es_CL es_CO es_CR es_CU es_DO es_EC es_GT es_HN es_MX es_NI es_PA es_PE es_PR es_PY es_SV es_US es_UY es_VE"/>
 		<parentLocale parent="pt_PT" locales="pt_AO pt_CV pt_GW pt_MO pt_MZ pt_ST pt_TL"/>
 		<parentLocale parent="zh_Hant_HK" locales="zh_Hant_MO"/>
+		<parentLocale parent="tlh" locales="de"/>
+		<parentLocale parent="en_054" locales="en_MT"/>
 	</parentLocales>

Seems to work - i.e., doesn't crash. r11728 needs some careful review, as it is in critical code.

  • I found an unrelated bug fixed in r11726 ( ST failed if a Region could not be formatted, such as a bad locale id ). Split into ticket:8688
  • r11727 adds GUI for adding new locales, for testing. Split this into ticket:8687
Last edited 2 years ago by srl (previous) (diff)

comment:18 Changed 22 months ago by emmons

  • Milestone changed from 28 to 29

Moving all survey related to 29dsub

comment:19 Changed 20 months ago by emmons

  • Milestone changed from 29 to upcoming

Automatic move of all 29 -> upcoming

View

Add a comment

Modify Ticket

Action
as accepted
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.