标签搜索

目 录CONTENT

文章目录

apisix配置需要证书的etcd集群

陈铭
2021-07-22 / 0 评论 / 0 点赞 / 403 阅读 / 2,312 字 / 正在检测是否收录...

生成各种证书

openssl方式生成

可以参考之前写的《生成自签名证书为服务器证书签名》那篇博客,注意一下CN这个字段的填写,对应宿主机的ip或者域名。最后我们得到ca.crt、server.key、server.crt三个文件

cfssl方式生成

可以参考之前写的《为etcd集群配置证书》那篇博客。需要注意的是,ca-config.json里面server的配置一定要加上“client auth”,因为新建集群时候,会默认做一次节点间健康检查,此时并没有像etcdctl那样提供client的证书和私钥,所以节点会用server的证书和私钥来进行通信,此时server证书就互为服务端和客户端了。
并且apisix和etcd集群通信时们,etcd由于grpc-gateway的原因,也会把server证书当成客户端证书来验证。

{
    "signing": {
        "default": {
            "expiry": "16800h"
        },
        "profiles": {
            "server": {
                "expiry": "8760h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth", # 就是这个
                    "client auth"
                ]
            },
            "client": {
                "expiry": "8760h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"
                ]
            }
        }
    }
}

新建集群

为了方便,还是新建两个节点的集群

# 各种证书私钥都放在根目录,etcd安装就不多赘述了
# 新建节点1
./etcd -name etcd1 \
  --auto-tls=true \
  --client-cert-auth=true \
  --cert-file=/server.crt \
  --key-file=/server.key \
  --trusted-ca-file=/ca.crt \
  -advertise-client-urls https://0.0.0.0:2379 \
  -listen-client-urls https://0.0.0.0:2379 \
  -listen-peer-urls http://0.0.0.0:2380 \
  -initial-advertise-peer-urls http://159.75.26.246:2380 \
  -initial-cluster-token etcd-cluster \
  -initial-cluster "etcd1=http://159.75.26.246:2380,etcd2=http://159.75.26.246:2480" \
  -initial-cluster-state new

# 新建节点2
./etcd -name etcd2 \
  --auto-tls=true \
  --client-cert-auth=true \
  --cert-file=/server.crt \
  --key-file=/server.key \
  --trusted-ca-file=/ca.crt \
  -advertise-client-urls https://0.0.0.0:2379 \
  -listen-client-urls https://0.0.0.0:2379 \
  -listen-peer-urls http://0.0.0.0:2380 \
  -initial-advertise-peer-urls http://159.75.26.246:2480 \
  -initial-cluster-token etcd-cluster \
  -initial-cluster "etcd1=http://159.75.26.246:2380,etcd2=http://159.75.26.246:2480" \
  -initial-cluster-state new

# etcdctl查看健康状态
./etcdctl \
  --cacert=/ca.crt \
  --cert=/server.crt \
  --key=/server.key \
  --endpoints=https://159.75.26.246:2379  endpoint health

# curl查看健康状态
curl --cacert /ca.crt --cert /server.crt --key /server.key https://159.75.26.246:2379/health

重构apisix的openresty

我们以docker官方镜像为基础,此时最新版本是2.7
docker pull apache/apisix:lastest
重构openresty需要安装一些命令和环境变量

yum-config-manager --add-repo https://openresty.org/package/centos/openresty.repo
yum -y install gcc gcc-c++ patch wget git make sudo
yum -y install openresty-openssl111-devel openresty-pcre-devel openresty-zlib-devel

export openssl_prefix=/usr/local/openresty/openssl111
export zlib_prefix=/usr/local/openresty/zlib
export pcre_prefix=/usr/local/openresty/pcre

export cc_opt="-DNGX_LUA_ABORT_AT_PANIC -I${zlib_prefix}/include -I${pcre_prefix}/include -I${openssl_prefix}/include"
export ld_opt="-L${zlib_prefix}/lib -L${pcre_prefix}/lib -L${openssl_prefix}/lib -Wl,-rpath,${zlib_prefix}/lib:${pcre_prefix}/lib:${openssl_prefix}/lib"

执行重构脚本build-apisix-openresty.sh,这个脚本可以在github下载
这个脚本内容如下

# !/usr/bin/env bash指的是去找环境中的bash路径,而不用我们直接指定。不同机器的bash不都是在/bin/bash
#!/usr/bin/env bash

# 设置shell,只要有一条指令返回值不为0,则整个shell都中断推出
set -euo pipefail
set -x

# $#指的是执行sh脚本的参数个数 -gt指的是大于 $0指的是sh脚本名字 $1指的是执行sh脚本的第一个参数 $2同理
if [ $# -gt 0 ] && [ "$1" == "latest" ]; then
    ngx_multi_upstream_module_ver=""
    mod_dubbo_ver=""
    apisix_nginx_module_ver=""
    lua_var_nginx_module_ver=""
    debug_args="--with-debug"
    OR_PREFIX=${OR_PREFIX:="/usr/local/openresty-debug"}
else
    # 定义好各种版本和路径
    ngx_multi_upstream_module_ver="-b 1.0.0"
    mod_dubbo_ver="-b 1.0.0"
    apisix_nginx_module_ver="-b 1.1.0"
    lua_var_nginx_module_ver="-b v0.5.2"
    debug_args=${debug_args:-}
    OR_PREFIX=${OR_PREFIX:="/usr/local/openresty"}
fi

prev_workdir="$PWD"
repo=$(basename "$prev_workdir")
workdir=$(mktemp -d)
cd "$workdir" || exit 1

or_ver="1.19.3.2"
wget https://openresty.org/download/openresty-${or_ver}.tar.gz
tar -zxvpf openresty-${or_ver}.tar.gz

if [ "$repo" == ngx_multi_upstream_module ]; then
    cp -r "$prev_workdir" .
else
    git clone --depth=1 $ngx_multi_upstream_module_ver \
        https://github.com/api7/ngx_multi_upstream_module.git
fi

if [ "$repo" == mod_dubbo ]; then
    cp -r "$prev_workdir" .
else
    git clone --depth=1 $mod_dubbo_ver \
        https://github.com/api7/mod_dubbo.git
fi

if [ "$repo" == apisix-nginx-module ]; then
    cp -r "$prev_workdir" .
else
    git clone --depth=1 $apisix_nginx_module_ver \
        https://github.com/api7/apisix-nginx-module.git
fi

if [ "$repo" == lua-var-nginx-module ]; then
    cp -r "$prev_workdir" .
else
    git clone --depth=1 $lua_var_nginx_module_ver \
        https://github.com/api7/lua-var-nginx-module
fi

# 语句1 || exit 1指的是语句1执行失败则exit 1,成功则不退出
cd ngx_multi_upstream_module || exit 1
./patch.sh ../openresty-${or_ver}
cd ..

cd apisix-nginx-module/patch || exit 1
./patch.sh ../../openresty-${or_ver}
cd ../..

version=${version:-0.0.0}
cc_opt=${cc_opt:-}
ld_opt=${ld_opt:-}
luajit_xcflags=${luajit_xcflags:="-DLUAJIT_NUMMODE=2 -DLUAJIT_ENABLE_LUA52COMPAT"}
no_pool_patch=${no_pool_patch:-}

cd openresty-${or_ver} || exit 1
./configure --prefix="$OR_PREFIX" \
    --with-cc-opt="-DAPISIX_OPENRESTY_VER=$version $cc_opt" \
    --with-ld-opt="$ld_opt" \
    --add-module=../mod_dubbo \
    --add-module=../ngx_multi_upstream_module \
    --add-module=../apisix-nginx-module \
    --add-module=../lua-var-nginx-module \
    $debug_args \
    --with-poll_module \
    --with-pcre-jit \
    --without-http_rds_json_module \
    --without-http_rds_csv_module \
    --without-lua_rds_parser \
    --with-stream \
    --with-stream_ssl_module \
    --with-stream_ssl_preread_module \
    --with-http_v2_module \
    --without-mail_pop3_module \
    --without-mail_imap_module \
    --without-mail_smtp_module \
    --with-http_stub_status_module \
    --with-http_realip_module \
    --with-http_addition_module \
    --with-http_auth_request_module \
    --with-http_secure_link_module \
    --with-http_random_index_module \
    --with-http_gzip_static_module \
    --with-http_sub_module \
    --with-http_dav_module \
    --with-http_flv_module \
    --with-http_mp4_module \
    --with-http_gunzip_module \
    --with-threads \
    --with-compat \
    --with-luajit-xcflags="$luajit_xcflags" \
    $no_pool_patch \
    -j`nproc`

make -j`nproc`
sudo make install
cd ..

cd apisix-nginx-module || exit 1
sudo OPENRESTY_PREFIX="$OR_PREFIX" make install

里面会使用git clone命令,这个命令常常拉不下来(github能不能拉看脸)。所以这里我改了一下脚本。在执行这个脚本前请先下载里面用到的4个ngx模块和我改的脚本(我已经放进gitee),都放在5个文件/build下执行脚本

我改的脚本如下:

#!/usr/bin/env bash
set -euo pipefail
set -x

if [ $# -gt 0 ] && [ "$1" == "latest" ]; then
    ngx_multi_upstream_module_ver=""
    mod_dubbo_ver=""
    apisix_nginx_module_ver=""
    lua_var_nginx_module_ver=""
    debug_args="--with-debug"
    OR_PREFIX=${OR_PREFIX:="/usr/local/openresty-debug"}
else
    ngx_multi_upstream_module_ver="-b 1.0.0"
    mod_dubbo_ver="-b 1.0.0"
    apisix_nginx_module_ver="-b 1.1.0"
    lua_var_nginx_module_ver="-b v0.5.2"
    debug_args=${debug_args:-}
    OR_PREFIX=${OR_PREFIX:="/usr/local/openresty"}
fi

prev_workdir="$PWD"
repo=$(basename "$prev_workdir")
workdir=$(mktemp -d)
cd "$workdir" || exit 1

or_ver="1.19.3.2"
wget https://openresty.org/download/openresty-${or_ver}.tar.gz
tar -zxvpf openresty-${or_ver}.tar.gz

if [ "$repo" == ngx_multi_upstream_module ]; then
    cp -r "$prev_workdir" .
else
    cp /build/ngx_multi_upstream_module-master.zip . && unzip ngx_multi_upstream_module-master.zip && mv ngx_multi_upstream_module-master ngx_multi_upstream_module
fi

if [ "$repo" == mod_dubbo ]; then
    cp -r "$prev_workdir" .
else
    cp /build/mod_dubbo-master.zip . && unzip mod_dubbo-master.zip && mv mod_dubbo-master mod_dubbo
fi

if [ "$repo" == apisix-nginx-module ]; then
    cp -r "$prev_workdir" .
else
    cp /build/apisix-nginx-module-main.zip . && unzip apisix-nginx-module-main.zip && mv apisix-nginx-module-main apisix-nginx-module
fi

if [ "$repo" == lua-var-nginx-module ]; then
    cp -r "$prev_workdir" .
else
    cp /build/lua-var-nginx-module-master.zip . && unzip lua-var-nginx-module-master.zip && mv lua-var-nginx-module-master lua-var-nginx-module
fi

cd ngx_multi_upstream_module || exit 1
./patch.sh ../openresty-${or_ver}
cd ..

cd apisix-nginx-module/patch || exit 1
./patch.sh ../../openresty-${or_ver}
cd ../..

version=${version:-0.0.0}
cc_opt=${cc_opt:-}
ld_opt=${ld_opt:-}
luajit_xcflags=${luajit_xcflags:="-DLUAJIT_NUMMODE=2 -DLUAJIT_ENABLE_LUA52COMPAT"}
no_pool_patch=${no_pool_patch:-}

cd openresty-${or_ver} || exit 1
./configure --prefix="$OR_PREFIX" \
    --with-cc-opt="-DAPISIX_OPENRESTY_VER=$version $cc_opt" \
    --with-ld-opt="$ld_opt" \
    --add-module=../mod_dubbo \
    --add-module=../ngx_multi_upstream_module \
    --add-module=../apisix-nginx-module \
    --add-module=../lua-var-nginx-module \
    $debug_args \
    --with-poll_module \
    --with-pcre-jit \
    --without-http_rds_json_module \
    --without-http_rds_csv_module \
    --without-lua_rds_parser \
    --with-stream \
    --with-stream_ssl_module \
    --with-stream_ssl_preread_module \
    --with-http_v2_module \
    --without-mail_pop3_module \
    --without-mail_imap_module \
    --without-mail_smtp_module \
    --with-http_stub_status_module \
    --with-http_realip_module \
    --with-http_addition_module \
    --with-http_auth_request_module \
    --with-http_secure_link_module \
    --with-http_random_index_module \
    --with-http_gzip_static_module \
    --with-http_sub_module \
    --with-http_dav_module \
    --with-http_flv_module \
    --with-http_mp4_module \
    --with-http_gunzip_module \
    --with-threads \
    --with-compat \
    --with-luajit-xcflags="$luajit_xcflags" \
    $no_pool_patch \
    -j`nproc`

make -j`nproc`
sudo make install
cd ..

cd apisix-nginx-module || exit 1
sudo OPENRESTY_PREFIX="$OR_PREFIX" make install

配置config.yaml

如果是cfssl生成的证书,可能需要对所有证书和私钥(ca、server和client)进行授权chmod 777 *.pem。避免apisix启动时可能出现的permission deny

整个yaml如下

apisix:
  admin_key:
    - name: "admin"
      key: edd1c9f034335f136f87ad84b625c8f1
      role: admin
  node_listen: 80 # http监听端口
  ssl:
    listen_port: 8443 # https监听端口
    ssl_trusted_certificate: /ca.crt # 受信任的的ca根证书
etcd:
  host: # etcd集群各节点
    - "https://159.75.26.246:2379"
    - "https://159.75.26.246:2479"
  tls: # etcd集群各节点需要的证书和私钥
    cert: /server.crt
    key: /server.key

启动

没报错,直接完事

apisix init
apisix init_etcd
apisix start
0

评论区