1. 场景描述
最近在 SpringBoot 程序中使用 Hive Metastore 的时候踩到了两个坑。比如下面这段代码:
@Service
public class HiveMetaServiceImpl implements HiveMetaService {
private final HiveMetaStoreClient client;
/**
* Instantiates a new Hive meta service.
*
* @param hiveUris the meta store uris
* @throws MetaException the meta exception
*/
public HiveMetaServiceImpl(@Value("${hive.metastore.uris}") String hiveUris)
throws MetaException {
HiveConf conf = new HiveConf();
conf.setVar(ConfVars.METASTOREURIS, hiveUris);
this.client = new HiveMetaStoreClient(conf);
}
}
当然要增加 Maven 的依赖:
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-metastore</artifactId>
<version>2.1.1</version>
</dependency>
如果在 IDEA 这样的集成开发环境里 debug 或者单元测试是没有问题的,但是一旦打成 JAR 包运行的时候就会出现各种幺蛾子。
2. 问题一:java.lang.NoClassDefFoundError: org/eclipse/jetty/server/RequestLog$Writer
使用 java -jar
运行的时候抛出了如下异常栈信息:
ERROR||2020-07-21 11:41:00.386||SpringApplication.java:837:org.springframework.boot.SpringApplication:main||Application run failed
java.lang.NoClassDefFoundError: org/eclipse/jetty/server/RequestLog$Writer
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getDeclaredConstructors(Class.java:2020)
at org.springframework.boot.context.properties.ConfigurationPropertiesBindConstructorProvider.findConstructorBindingAnnotatedConstructor(ConfigurationPropertiesBindConstructorProvider.java:62)
at org.springframework.boot.context.properties.ConfigurationPropertiesBindConstructorProvider.getBindConstructor(ConfigurationPropertiesBindConstructorProvider.java:48)
at org.springframework.boot.context.properties.ConfigurationPropertiesBean$BindMethod.forType(ConfigurationPropertiesBean.java:311)
at org.springframework.boot.context.properties.ConfigurationPropertiesBeanDefinitionValidator.validate(ConfigurationPropertiesBeanDefinitionValidator.java:63)
at org.springframework.boot.context.properties.ConfigurationPropertiesBeanDefinitionValidator.postProcessBeanFactory(ConfigurationPropertiesBeanDefinitionValidator.java:45)
at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:291)
at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:175)
at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:707)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:533)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:143)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
at com.xxx.platform.makoto.Application.main(Application.java:39)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:109)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)
Caused by: java.lang.ClassNotFoundException: org.eclipse.jetty.server.RequestLog$Writer
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:151)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 26 common frames omitted
原因呢,其实就是缺依赖。我们可以搜索一下缺失的类 org.eclipse.jetty.server.RequestLog$Writer
可以找到对应的页面 https://www.eclipse.org/jetty/javadoc/current/org/eclipse/jetty/server/RequestLog.Writer.html。
所以加上对应的依赖和版本就好:
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-server</artifactId>
<version>9.4.30.v20200611</version>
</dependency>
3. 问题二:URI is not hierarchical
Spring 在启动构造 Bean 的阶段就会抛出异常,异常站信息如下:
Failed to instantiate [com.xxx.platform.makoto.common.sync.service.HiveMetaServiceImpl]: Constructor threw exception; nested exception is java.lang.ExceptionInInitializerError
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:313)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:294)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1358)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1204)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:557)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:893)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:879)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:551)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:143)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
at com.xxx.platform.makoto.Application.main(Application.java:39)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:109)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.xxx.platform.makoto.common.sync.service.HiveMetaServiceImpl]: Constructor threw exception; nested exception is java.lang.ExceptionInInitializerError
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:217)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:117)
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:309)
... 26 common frames omitted
Caused by: java.lang.ExceptionInInitializerError: null
at com.xxx.platform.makoto.common.sync.service.HiveMetaServiceImpl.<init>(HiveMetaServiceImpl.java:42)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:204)
... 28 common frames omitted
Caused by: java.lang.IllegalArgumentException: URI is not hierarchical
at java.io.File.<init>(File.java:418)
at org.apache.hadoop.hive.conf.HiveConf.findConfigFile(HiveConf.java:171)
at org.apache.hadoop.hive.conf.HiveConf.<clinit>(HiveConf.java:139)
... 34 common frames omitted
结合代码来看出问题的在于构造 HiveConf
的这一段代码 HiveConf conf = new HiveConf()
。
具体的描述可以看这个 issue:HiveConf static initialization fails when JAR URI is opaque。概括来说,fat jar 在使用 Metastore 时都会遇到这个问题。解决的办法也很简单,打上一个很简单的 patch,替换 hive-common
包即可,架构同学工具人简单打个包发布一把即可。
最后的 pom 依赖为:
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-metastore</artifactId>
<version>2.1.1</version>
<exclusions>
<exclusion>
<artifactId>hive-common</artifactId>
<groupId>org.apache.hive</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-common-makoto</artifactId>
<version>2.1.1</version>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-server</artifactId>
<version>9.4.30.v20200611</version>
</dependency>
4. 吐槽
都知道 Hive 代码写的糟,但是坑太 TM 多了。