生成各种证书
openssl方式生成
可以参考之前写的《生成自签名证书为服务器证书签名》那篇博客，注意一下CN这个字段的填写，对应宿主机的ip或者域名。最后我们得到ca.crt、server.key、server.crt三个文件
cfssl方式生成
可以参考之前写的《为etcd集群配置证书》那篇博客。需要注意的是，ca-config.json里面server的配置一定要加上“client auth”，因为新建集群时候，会默认做一次节点间健康检查，此时并没有像etcdctl那样提供client的证书和私钥，所以节点会用server的证书和私钥来进行通信，此时server证书就互为服务端和客户端了。
并且apisix和etcd集群通信时们，etcd由于grpc-gateway的原因，也会把server证书当成客户端证书来验证。
{
"signing": {
"default": {
"expiry": "16800h"
},
"profiles": {
"server": {
"expiry": "8760h",
"usages": [
"signing",
"key encipherment",
"server auth", # 就是这个
"client auth"
]
},
"client": {
"expiry": "8760h",
"usages": [
"signing",
"key encipherment",
"client auth"
]
}
}
}
}
新建集群
为了方便，还是新建两个节点的集群
# 各种证书私钥都放在根目录，etcd安装就不多赘述了
# 新建节点1
./etcd -name etcd1 \
--auto-tls=true \
--client-cert-auth=true \
--cert-file=/server.crt \
--key-file=/server.key \
--trusted-ca-file=/ca.crt \
-advertise-client-urls https://0.0.0.0:2379 \
-listen-client-urls https://0.0.0.0:2379 \
-listen-peer-urls http://0.0.0.0:2380 \
-initial-advertise-peer-urls http://159.75.26.246:2380 \
-initial-cluster-token etcd-cluster \
-initial-cluster "etcd1=http://159.75.26.246:2380,etcd2=http://159.75.26.246:2480" \
-initial-cluster-state new
# 新建节点2
./etcd -name etcd2 \
--auto-tls=true \
--client-cert-auth=true \
--cert-file=/server.crt \
--key-file=/server.key \
--trusted-ca-file=/ca.crt \
-advertise-client-urls https://0.0.0.0:2379 \
-listen-client-urls https://0.0.0.0:2379 \
-listen-peer-urls http://0.0.0.0:2380 \
-initial-advertise-peer-urls http://159.75.26.246:2480 \
-initial-cluster-token etcd-cluster \
-initial-cluster "etcd1=http://159.75.26.246:2380,etcd2=http://159.75.26.246:2480" \
-initial-cluster-state new
# etcdctl查看健康状态
./etcdctl \
--cacert=/ca.crt \
--cert=/server.crt \
--key=/server.key \
--endpoints=https://159.75.26.246:2379 endpoint health
# curl查看健康状态
curl --cacert /ca.crt --cert /server.crt --key /server.key https://159.75.26.246:2379/health
重构apisix的openresty
我们以docker官方镜像为基础，此时最新版本是2.7
docker pull apache/apisix:lastest
重构openresty需要安装一些命令和环境变量
yum-config-manager --add-repo https://openresty.org/package/centos/openresty.repo
yum -y install gcc gcc-c++ patch wget git make sudo
yum -y install openresty-openssl111-devel openresty-pcre-devel openresty-zlib-devel
export openssl_prefix=/usr/local/openresty/openssl111
export zlib_prefix=/usr/local/openresty/zlib
export pcre_prefix=/usr/local/openresty/pcre
export cc_opt="-DNGX_LUA_ABORT_AT_PANIC -I${zlib_prefix}/include -I${pcre_prefix}/include -I${openssl_prefix}/include"
export ld_opt="-L${zlib_prefix}/lib -L${pcre_prefix}/lib -L${openssl_prefix}/lib -Wl,-rpath,${zlib_prefix}/lib:${pcre_prefix}/lib:${openssl_prefix}/lib"
执行重构脚本build-apisix-openresty.sh，这个脚本可以在github下载
这个脚本内容如下
# !/usr/bin/env bash指的是去找环境中的bash路径，而不用我们直接指定。不同机器的bash不都是在/bin/bash
#!/usr/bin/env bash
# 设置shell，只要有一条指令返回值不为0，则整个shell都中断推出
set -euo pipefail
set -x
# $#指的是执行sh脚本的参数个数 -gt指的是大于 $0指的是sh脚本名字 $1指的是执行sh脚本的第一个参数 $2同理
if [ $# -gt 0 ] && [ "$1" == "latest" ]; then
ngx_multi_upstream_module_ver=""
mod_dubbo_ver=""
apisix_nginx_module_ver=""
lua_var_nginx_module_ver=""
debug_args="--with-debug"
OR_PREFIX=${OR_PREFIX:="/usr/local/openresty-debug"}
else
# 定义好各种版本和路径
ngx_multi_upstream_module_ver="-b 1.0.0"
mod_dubbo_ver="-b 1.0.0"
apisix_nginx_module_ver="-b 1.1.0"
lua_var_nginx_module_ver="-b v0.5.2"
debug_args=${debug_args:-}
OR_PREFIX=${OR_PREFIX:="/usr/local/openresty"}
fi
prev_workdir="$PWD"
repo=$(basename "$prev_workdir")
workdir=$(mktemp -d)
cd "$workdir" || exit 1
or_ver="1.19.3.2"
wget https://openresty.org/download/openresty-${or_ver}.tar.gz
tar -zxvpf openresty-${or_ver}.tar.gz
if [ "$repo" == ngx_multi_upstream_module ]; then
cp -r "$prev_workdir" .
else
git clone --depth=1 $ngx_multi_upstream_module_ver \
https://github.com/api7/ngx_multi_upstream_module.git
fi
if [ "$repo" == mod_dubbo ]; then
cp -r "$prev_workdir" .
else
git clone --depth=1 $mod_dubbo_ver \
https://github.com/api7/mod_dubbo.git
fi
if [ "$repo" == apisix-nginx-module ]; then
cp -r "$prev_workdir" .
else
git clone --depth=1 $apisix_nginx_module_ver \
https://github.com/api7/apisix-nginx-module.git
fi
if [ "$repo" == lua-var-nginx-module ]; then
cp -r "$prev_workdir" .
else
git clone --depth=1 $lua_var_nginx_module_ver \
https://github.com/api7/lua-var-nginx-module
fi
# 语句1 || exit 1指的是语句1执行失败则exit 1，成功则不退出
cd ngx_multi_upstream_module || exit 1
./patch.sh ../openresty-${or_ver}
cd ..
cd apisix-nginx-module/patch || exit 1
./patch.sh ../../openresty-${or_ver}
cd ../..
version=${version:-0.0.0}
cc_opt=${cc_opt:-}
ld_opt=${ld_opt:-}
luajit_xcflags=${luajit_xcflags:="-DLUAJIT_NUMMODE=2 -DLUAJIT_ENABLE_LUA52COMPAT"}
no_pool_patch=${no_pool_patch:-}
cd openresty-${or_ver} || exit 1
./configure --prefix="$OR_PREFIX" \
--with-cc-opt="-DAPISIX_OPENRESTY_VER=$version $cc_opt" \
--with-ld-opt="$ld_opt" \
--add-module=../mod_dubbo \
--add-module=../ngx_multi_upstream_module \
--add-module=../apisix-nginx-module \
--add-module=../lua-var-nginx-module \
$debug_args \
--with-poll_module \
--with-pcre-jit \
--without-http_rds_json_module \
--without-http_rds_csv_module \
--without-lua_rds_parser \
--with-stream \
--with-stream_ssl_module \
--with-stream_ssl_preread_module \
--with-http_v2_module \
--without-mail_pop3_module \
--without-mail_imap_module \
--without-mail_smtp_module \
--with-http_stub_status_module \
--with-http_realip_module \
--with-http_addition_module \
--with-http_auth_request_module \
--with-http_secure_link_module \
--with-http_random_index_module \
--with-http_gzip_static_module \
--with-http_sub_module \
--with-http_dav_module \
--with-http_flv_module \
--with-http_mp4_module \
--with-http_gunzip_module \
--with-threads \
--with-compat \
--with-luajit-xcflags="$luajit_xcflags" \
$no_pool_patch \
-j`nproc`
make -j`nproc`
sudo make install
cd ..
cd apisix-nginx-module || exit 1
sudo OPENRESTY_PREFIX="$OR_PREFIX" make install
里面会使用git clone命令，这个命令常常拉不下来（github能不能拉看脸）。所以这里我改了一下脚本。在执行这个脚本前请先下载里面用到的4个ngx模块和我改的脚本（我已经放进gitee），都放在5个文件/build下执行脚本
我改的脚本如下：
#!/usr/bin/env bash
set -euo pipefail
set -x
if [ $# -gt 0 ] && [ "$1" == "latest" ]; then
ngx_multi_upstream_module_ver=""
mod_dubbo_ver=""
apisix_nginx_module_ver=""
lua_var_nginx_module_ver=""
debug_args="--with-debug"
OR_PREFIX=${OR_PREFIX:="/usr/local/openresty-debug"}
else
ngx_multi_upstream_module_ver="-b 1.0.0"
mod_dubbo_ver="-b 1.0.0"
apisix_nginx_module_ver="-b 1.1.0"
lua_var_nginx_module_ver="-b v0.5.2"
debug_args=${debug_args:-}
OR_PREFIX=${OR_PREFIX:="/usr/local/openresty"}
fi
prev_workdir="$PWD"
repo=$(basename "$prev_workdir")
workdir=$(mktemp -d)
cd "$workdir" || exit 1
or_ver="1.19.3.2"
wget https://openresty.org/download/openresty-${or_ver}.tar.gz
tar -zxvpf openresty-${or_ver}.tar.gz
if [ "$repo" == ngx_multi_upstream_module ]; then
cp -r "$prev_workdir" .
else
cp /build/ngx_multi_upstream_module-master.zip . && unzip ngx_multi_upstream_module-master.zip && mv ngx_multi_upstream_module-master ngx_multi_upstream_module
fi
if [ "$repo" == mod_dubbo ]; then
cp -r "$prev_workdir" .
else
cp /build/mod_dubbo-master.zip . && unzip mod_dubbo-master.zip && mv mod_dubbo-master mod_dubbo
fi
if [ "$repo" == apisix-nginx-module ]; then
cp -r "$prev_workdir" .
else
cp /build/apisix-nginx-module-main.zip . && unzip apisix-nginx-module-main.zip && mv apisix-nginx-module-main apisix-nginx-module
fi
if [ "$repo" == lua-var-nginx-module ]; then
cp -r "$prev_workdir" .
else
cp /build/lua-var-nginx-module-master.zip . && unzip lua-var-nginx-module-master.zip && mv lua-var-nginx-module-master lua-var-nginx-module
fi
cd ngx_multi_upstream_module || exit 1
./patch.sh ../openresty-${or_ver}
cd ..
cd apisix-nginx-module/patch || exit 1
./patch.sh ../../openresty-${or_ver}
cd ../..
version=${version:-0.0.0}
cc_opt=${cc_opt:-}
ld_opt=${ld_opt:-}
luajit_xcflags=${luajit_xcflags:="-DLUAJIT_NUMMODE=2 -DLUAJIT_ENABLE_LUA52COMPAT"}
no_pool_patch=${no_pool_patch:-}
cd openresty-${or_ver} || exit 1
./configure --prefix="$OR_PREFIX" \
--with-cc-opt="-DAPISIX_OPENRESTY_VER=$version $cc_opt" \
--with-ld-opt="$ld_opt" \
--add-module=../mod_dubbo \
--add-module=../ngx_multi_upstream_module \
--add-module=../apisix-nginx-module \
--add-module=../lua-var-nginx-module \
$debug_args \
--with-poll_module \
--with-pcre-jit \
--without-http_rds_json_module \
--without-http_rds_csv_module \
--without-lua_rds_parser \
--with-stream \
--with-stream_ssl_module \
--with-stream_ssl_preread_module \
--with-http_v2_module \
--without-mail_pop3_module \
--without-mail_imap_module \
--without-mail_smtp_module \
--with-http_stub_status_module \
--with-http_realip_module \
--with-http_addition_module \
--with-http_auth_request_module \
--with-http_secure_link_module \
--with-http_random_index_module \
--with-http_gzip_static_module \
--with-http_sub_module \
--with-http_dav_module \
--with-http_flv_module \
--with-http_mp4_module \
--with-http_gunzip_module \
--with-threads \
--with-compat \
--with-luajit-xcflags="$luajit_xcflags" \
$no_pool_patch \
-j`nproc`
make -j`nproc`
sudo make install
cd ..
cd apisix-nginx-module || exit 1
sudo OPENRESTY_PREFIX="$OR_PREFIX" make install
配置config.yaml
如果是cfssl生成的证书，可能需要对所有证书和私钥（ca、server和client）进行授权chmod 777 *.pem。避免apisix启动时可能出现的permission deny
整个yaml如下
apisix:
admin_key:
- name: "admin"
key: edd1c9f034335f136f87ad84b625c8f1
role: admin
node_listen: 80 # http监听端口
ssl:
listen_port: 8443 # https监听端口
ssl_trusted_certificate: /ca.crt # 受信任的的ca根证书
etcd:
host: # etcd集群各节点
- "https://159.75.26.246:2379"
- "https://159.75.26.246:2479"
tls: # etcd集群各节点需要的证书和私钥
cert: /server.crt
key: /server.key
启动
没报错，直接完事
apisix init
apisix init_etcd
apisix start
评论区