在 Spring Boot 中使用 Hive Metastore 踩到的坑

1. 场景描述

最近在 SpringBoot 程序中使用 Hive Metastore 的时候踩到了两个坑。比如下面这段代码:

@Service
public class HiveMetaServiceImpl implements HiveMetaService {

  private final HiveMetaStoreClient client;

  /**
   * Instantiates a new Hive meta service.
   *
   * @param hiveUris the meta store uris
   * @throws MetaException the meta exception
   */
  public HiveMetaServiceImpl(@Value("${hive.metastore.uris}") String hiveUris)
      throws MetaException {
    HiveConf conf = new HiveConf();

    conf.setVar(ConfVars.METASTOREURIS, hiveUris);
    this.client = new HiveMetaStoreClient(conf);
  }
}

当然要增加 Maven 的依赖:

<dependency>
  <groupId>org.apache.hive</groupId>
  <artifactId>hive-metastore</artifactId>
  <version>2.1.1</version>
</dependency>

如果在 IDEA 这样的集成开发环境里 debug 或者单元测试是没有问题的,但是一旦打成 JAR 包运行的时候就会出现各种幺蛾子。

2. 问题一:java.lang.NoClassDefFoundError: org/eclipse/jetty/server/RequestLog$Writer

使用 java -jar 运行的时候抛出了如下异常栈信息:

ERROR||2020-07-21 11:41:00.386||SpringApplication.java:837:org.springframework.boot.SpringApplication:main||Application run failed
java.lang.NoClassDefFoundError: org/eclipse/jetty/server/RequestLog$Writer
        at java.lang.Class.getDeclaredConstructors0(Native Method)
        at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
        at java.lang.Class.getDeclaredConstructors(Class.java:2020)
        at org.springframework.boot.context.properties.ConfigurationPropertiesBindConstructorProvider.findConstructorBindingAnnotatedConstructor(ConfigurationPropertiesBindConstructorProvider.java:62)
        at org.springframework.boot.context.properties.ConfigurationPropertiesBindConstructorProvider.getBindConstructor(ConfigurationPropertiesBindConstructorProvider.java:48)
        at org.springframework.boot.context.properties.ConfigurationPropertiesBean$BindMethod.forType(ConfigurationPropertiesBean.java:311)
        at org.springframework.boot.context.properties.ConfigurationPropertiesBeanDefinitionValidator.validate(ConfigurationPropertiesBeanDefinitionValidator.java:63)
        at org.springframework.boot.context.properties.ConfigurationPropertiesBeanDefinitionValidator.postProcessBeanFactory(ConfigurationPropertiesBeanDefinitionValidator.java:45)
        at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:291)
        at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:175)
        at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:707)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:533)
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:143)
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758)
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750)
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
        at com.xxx.platform.makoto.Application.main(Application.java:39)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:109)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)
Caused by: java.lang.ClassNotFoundException: org.eclipse.jetty.server.RequestLog$Writer
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:151)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 26 common frames omitted

原因呢,其实就是缺依赖。我们可以搜索一下缺失的类 org.eclipse.jetty.server.RequestLog$Writer 可以找到对应的页面 https://www.eclipse.org/jetty/javadoc/current/org/eclipse/jetty/server/RequestLog.Writer.html。

所以加上对应的依赖和版本就好:

<dependency>
  <groupId>org.eclipse.jetty</groupId>
  <artifactId>jetty-server</artifactId>
  <version>9.4.30.v20200611</version>
</dependency>

3. 问题二:URI is not hierarchical

Spring 在启动构造 Bean 的阶段就会抛出异常,异常站信息如下:

Failed to instantiate [com.xxx.platform.makoto.common.sync.service.HiveMetaServiceImpl]: Constructor threw exception; nested exception is java.lang.ExceptionInInitializerError
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:313)
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:294)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1358)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1204)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:557)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:893)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:879)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:551)
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:143)
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758)
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750)
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
        at com.xxx.platform.makoto.Application.main(Application.java:39)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:109)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.xxx.platform.makoto.common.sync.service.HiveMetaServiceImpl]: Constructor threw exception; nested exception is java.lang.ExceptionInInitializerError
        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:217)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:117)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:309)
        ... 26 common frames omitted
Caused by: java.lang.ExceptionInInitializerError: null
        at com.xxx.platform.makoto.common.sync.service.HiveMetaServiceImpl.<init>(HiveMetaServiceImpl.java:42)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:204)
        ... 28 common frames omitted
Caused by: java.lang.IllegalArgumentException: URI is not hierarchical
        at java.io.File.<init>(File.java:418)
        at org.apache.hadoop.hive.conf.HiveConf.findConfigFile(HiveConf.java:171)
        at org.apache.hadoop.hive.conf.HiveConf.<clinit>(HiveConf.java:139)
        ... 34 common frames omitted

结合代码来看出问题的在于构造 HiveConf 的这一段代码 HiveConf conf = new HiveConf()

具体的描述可以看这个 issue:HiveConf static initialization fails when JAR URI is opaque。概括来说,fat jar 在使用 Metastore 时都会遇到这个问题。解决的办法也很简单,打上一个很简单的 patch,替换 hive-common 包即可,架构同学工具人简单打个包发布一把即可。

最后的 pom 依赖为:

<dependency>
  <groupId>org.apache.hive</groupId>
  <artifactId>hive-metastore</artifactId>
  <version>2.1.1</version>
  <exclusions>
    <exclusion>
      <artifactId>hive-common</artifactId>
      <groupId>org.apache.hive</groupId>
    </exclusion>
  </exclusions>
</dependency>
<dependency>
  <groupId>org.apache.hive</groupId>
  <artifactId>hive-common-makoto</artifactId>
  <version>2.1.1</version>
</dependency>
<dependency>
  <groupId>org.eclipse.jetty</groupId>
  <artifactId>jetty-server</artifactId>
  <version>9.4.30.v20200611</version>
</dependency>

4. 吐槽

都知道 Hive 代码写的糟,但是坑太 TM 多了。

Spring 切面与用户认证

1. 需求概述

一般做用户侧的项目的时候,用户身份认证和登录是免不了要做的。

每个方法都加一个身份认证的方法太过繁琐,我们可以通过实现一个 @RequireAuthentication 注解来简化这个过程。

下面给出了一个简单的基于 Java 注解和切面实现的 OAuth 认证的实现方案。

2. 技术方案

2.1 定义注解

元注解:

  1. @Target:修饰范围。Annotation可被用于 packages、types(类、接口、枚举、Annotation类型)、类型成员(方法、构造方法、成员变量、枚举值)、方法参数和本地变量(如循环变量、catch参数)。
  2. @Retention:定义了该Annotation被保留的时间长短。a)SOURCE:在源文件中有效(即源文件保留);b)CLASS:在class文件中有效(即class保留);c)RUNTIME:在运行时有效(即运行时保留)。
  3. @Documented:描述其它类型的 annotation 应该被作为被标注的程序成员的公共API,因此可以被例如 javadoc 此类的工具文档化。
  4. @Inherited:标记注解,@Inherited阐述了某个被标注的类型是被继承的。如果一个使用了 @Inherited 修饰的 annotation 类型被用于一个 class,则这个 annotation 将被用于该 class 的子类。

自定义注解:

package xxx.common.authentication.annotation;

import java.lang.annotation.Documented;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

/**
 * The interface Require authentication.
 *
 * @Author: wuzhiyu.
 * @Date: 2020-07-08 15:19:44.
 * @Description:
 */
@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface RequireAuthentication {
}

2.2 相关切面

首先加上 Maven 依赖:

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-aop</artifactId>
</dependency>

基于注解定义切面以及 @Before 操作:

package xxx.common.authentication.aspect;

import com.avaloninc.web.commons.api.util.RequestUtils;
import com.avaloninc.web.log.audit.exception.UnauthorizedException;
import xxx.common.authentication.domain.Session;
import xxx.common.authentication.service.AuthenticationService;
import java.util.Objects;
import javax.servlet.http.HttpServletRequest;
import org.aspectj.lang.JoinPoint;
import org.aspectj.lang.annotation.Aspect;
import org.aspectj.lang.annotation.Before;
import org.aspectj.lang.annotation.Pointcut;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import org.springframework.util.StringUtils;
import org.springframework.web.context.request.RequestContextHolder;
import org.springframework.web.context.request.ServletRequestAttributes;

/**
 * The type Authentication aspect.
 *
 * @Author: wuzhiyu.
 * @Date: 2020-07-08 15:35:59.
 * @Description:
 */
@Aspect
@Component
public class AuthenticationAspect {

  private final AuthenticationService authenticationService;

  /**
   * Instantiates a new Authentication aspect.
   *
   * @param authenticationService the authentication service
   */
  @Autowired
  public AuthenticationAspect(AuthenticationService authenticationService) {
    this.authenticationService = authenticationService;
  }

  /**
   * Require authentication.
   */
  @Pointcut(
      "@annotation(xxx.common.authentication.annotation.RequireAuthentication)")
  public void requireAuthentication() {

  }

  /**
   * Authenticate.
   *
   * @param joinPoint the join point
   */
  @Before("requireAuthentication()")
  public void authenticate(JoinPoint joinPoint) {
    final HttpServletRequest request =
        ((ServletRequestAttributes) RequestContextHolder.currentRequestAttributes()).getRequest();
    final String authorization = request.getHeader("Authorization");
    if (!StringUtils.isEmpty(authorization)) {
      final String  token       = authorization.replace("Bearer", "").trim();
      final Session session = this.authenticationService.getSessionByToken(token);
      if (Objects.nonNull(session)) {
        final String user = session.getUser().getId();
        RequestUtils.setUserName(user);
        return;
      }
    }
    throw new UnauthorizedException("Need to login!");
  }
}

在上面的代码中,我们从 request 的 Header 中获取 token。然后通过 token 判断用户是否已经登录(Session 对象是否存在)。

如果 Session 对象不存在则认为用户没有登录,抛出一个自定义的异常。外围通过一个 @ControllerAdvice 注解的全局异常处理器捕捉这个异常,然后返回 401 的 response。

2.3 OAuth 2.0 获取 Token 以及 Profile

首先定义一个 Service:

package xxx.common.authentication.service;

import xxx.common.authentication.domain.Session;

/**
 * The interface Authentication service.
 *
 * @Author: wuzhiyu.
 * @Date: 2020-07-08 15:43:09.
 * @Description:
 */
public interface AuthenticationService {
  /**
   * Gets token.
   *
   * @param code the code
   * @return the token
   */
  String getToken(String code);

  /**
   * Gets user by token.
   *
   * @param token the token
   * @return the user by token
   */
  Session getSessionByToken(String token);
}

这个 Service 定义了两个方法.

getToken 方法根据前端传递的 code 从 OAuth 的服务器获取一个 token,然后通过 token 获取用户的 profile。并随之创建 Session 对象,进行持久化存储以及过期时间的设置。

getSessionByToken 方法则根据 token 查找 Session 对象来判断登录是否过期。

下面给出实现类:

package xxx.common.authentication.service;

import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import xxx.common.authentication.domain.Profile;
import xxx.common.authentication.domain.Session;
import xxx.common.authentication.domain.Token;
import xxx.domain.user.entity.User;
import xxx.domain.user.service.UserService;
import java.io.IOException;
import java.time.Duration;
import java.time.LocalDateTime;
import lombok.SneakyThrows;
import okhttp3.FormBody;
import okhttp3.OkHttpClient;
import okhttp3.Request;
import okhttp3.Request.Builder;
import okhttp3.RequestBody;
import okhttp3.Response;
import okhttp3.ResponseBody;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.stereotype.Service;

/**
 * The type Authentication service.
 *
 * @Author: wuzhiyu.
 * @Date: 2020-07-08 16:01:40.
 * @Description:
 */
@Service
public class AuthenticationServiceImpl implements AuthenticationService {

  private static final String       SESSION_PREFIX = "prefix:";
  private static final OkHttpClient CLIENT         = new OkHttpClient();
  private static final Duration     EXPIRATION     = Duration.ofDays(3);

  private final UserService                   userService;
  private final RedisTemplate<String, String> redisTemplate;
  private final ObjectMapper                  objectMapper;

  private final String clientId;
  private final String clientSecret;
  private final String grantType;
  private final String tokenUrl;
  private final String redirectUrl;
  private final String profileUrl;

  /**
   * Instantiates a new Authentication service.
   *
   * @param userService   the user service
   * @param redisTemplate the redis template
   * @param objectMapper  the object mapper
   * @param clientId      the client id
   * @param clientSecret  the client secret
   * @param grantType     the grant type
   * @param tokenUrl      the token url
   * @param redirectUrl   the redirect url
   * @param profileUrl    the profile url
   */
  @Autowired
  public AuthenticationServiceImpl(UserService userService,
                                   RedisTemplate<String, String> redisTemplate,
                                   ObjectMapper objectMapper,
                                   @Value("${OAuth2.client_id}") String clientId,
                                   @Value("${OAuth2.client_secret}") String clientSecret,
                                   @Value("${OAuth2.grant_type}") String grantType,
                                   @Value("${OAuth2.token_url}") String tokenUrl,
                                   @Value("${OAuth2.redirect_uri}") String redirectUrl,
                                   @Value("${OAuth2.profile_url}") String profileUrl) {
    this.userService = userService;
    this.redisTemplate = redisTemplate;
    this.objectMapper = objectMapper;
    this.clientId = clientId;
    this.clientSecret = clientSecret;
    this.grantType = grantType;
    this.tokenUrl = tokenUrl;
    this.redirectUrl = redirectUrl;
    this.profileUrl = profileUrl;
  }


  @SneakyThrows
  @Override
  public String getToken(String code) {

    final Token   token       = this.getTokenFromSso(code);
    final String  accessToken = token.getAccessToken();
    final Profile profile     = this.getProfileFromSso(accessToken);

    this.recordUserLogin(profile);

    final Session session = this.getSession(code, token, profile);
    this.cacheSession(accessToken, session);
    return accessToken;
  }

  private void cacheSession(String accessToken, Session session) throws JsonProcessingException {
    final String sessionStr = this.objectMapper.writeValueAsString(session);
    this.redisTemplate.opsForValue().set(this.getKey(accessToken), sessionStr, EXPIRATION);
  }

  private void recordUserLogin(Profile profile) {
    User               user     = new User();
    final Profile.User userInfo = profile.getUser();
    user.setUserId(userInfo.getId());
    user.setName(userInfo.getName());
    user.setEmail(userInfo.getEmail());
    user.setPhone(userInfo.getPhone());
    this.userService.add(user);
  }

  private Session getSession(String code, Token token, Profile profile) {
    Session session = new Session();
    session.setUser(profile.getUser());
    session.setCode(code);
    session.setAccessToken(token.getAccessToken());
    final LocalDateTime now = LocalDateTime.now();
    session.setLoginTime(now);
    session.setExpiredAt(now.plus(EXPIRATION));
    return session;
  }

  private String getKey(String token) {
    return SESSION_PREFIX + token;
  }

  private Profile getProfileFromSso(String token) throws IOException {
    final Request request = new Builder()
        .url(this.profileUrl).addHeader("Authorization", "Bearer " + token).get().build();
    final ResponseBody body = CLIENT.newCall(request).execute().body();
    return this.objectMapper.readValue(body.byteStream(), Profile.class);
  }

  private Token getTokenFromSso(String code) throws IOException {
    RequestBody form = new FormBody.Builder()
        .add("code", code)
        .add("client_id", this.clientId)
        .add("client_secret", this.clientSecret)
        .add("redirect_uri", this.redirectUrl)
        .add("grant_type", this.grantType)
        .build();

    final Request  request  = new Builder().url(this.tokenUrl).post(form).build();
    final Response response = CLIENT.newCall(request).execute();
    return this.objectMapper.readValue(response.body().byteStream(), Token.class);
  }

  @SneakyThrows
  @Override
  public Session getSessionByToken(String token) {
    final String key   = this.getKey(token);
    final String value = this.redisTemplate.opsForValue().get(key);
    return this.objectMapper.readValue(value, Session.class);
  }
}

3. 使用实例

package xxx.api.controllers;

import com.avaloninc.web.commons.api.responses.Response;
import com.avaloninc.web.commons.api.responses.Responses;
import xxx.common.authentication.annotation.RequireAuthentication;
import xxx.api.requests.SearchUserRequest;
import xxx.domain.user.entity.User;
import xxx.domain.user.service.UserService;
import java.util.List;
import javax.validation.Valid;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

/**
 * The type User controller.
 *
 * @Author: wuzhiyu.
 * @Date: 2020-07-09 12:27:45.
 * @Description:
 */
@RestController
@RequestMapping("api/users")
public class UserController {

  private final UserService userService;

  /**
   * Instantiates a new User controller.
   *
   * @param userService the user service
   */
  @Autowired
  public UserController(UserService userService) {
    this.userService = userService;
  }

  /**
   * Search response.
   *
   * @param request the request
   * @return the response
   */
  @GetMapping
  @RequireAuthentication
  public Response<List<User>> search(@Valid SearchUserRequest request) {
    final List<User> users = this.userService.search(request.getKeyword());
    return Responses.successResponse(users);
  }
}

以上~