Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update: add service for v2 protocol (ConfigServer) #1793

Open
wants to merge 8 commits into
base: ospp_dev
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions config_server/service/.gitignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
DB
router/cmd
ConfigServer
license.sh
config-server
cmd/config/dev/*
.idea
39 changes: 13 additions & 26 deletions config_server/service/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,39 +1,26 @@
# Copyright 2023 iLogtail Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

FROM sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/ilogtail-community-edition/ilogtail-build-linux:2.0.3 as build

USER root
WORKDIR /src
WORKDIR /backend
COPY . .
Takuka0311 marked this conversation as resolved.
Show resolved Hide resolved
RUN go env -w GOPROXY="https://goproxy.cn,direct" && \
go build -o ConfigServer

FROM centos:centos7.9.2009
MAINTAINER TomYu [email protected]
RUN go env -w GOPROXY="https://goproxy.cn,direct" \
&& go mod tidy \
&& go build -o ConfigServer ./cmd

ENV container docker

FROM centos:centos7.9.2009
RUN curl -L -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

RUN yum update -y && yum upgrade -y && yum -y clean all && rm -fr /var/cache && rm -rf /core.*

WORKDIR /config_server
COPY --from=build /src/ConfigServer /config_server/ConfigServer
COPY --from=build /src/setting/setting.json /config_server/setting/setting.json
RUN chmod 755 /config_server/ConfigServer && \
sed -i 's/127\.0\.0\.1/0\.0\.0\.0/' setting/setting.json
WORKDIR /backend
COPY --from=build /backend/cmd /backend/cmd
COPY --from=build /backend/ConfigServer /backend/ConfigServer
ENV GIN_MODE=release
ENV GO_ENV=prod

CMD sh -c "./ConfigServer"



CMD ["./ConfigServer"]
13 changes: 13 additions & 0 deletions config_server/service/Dockerfile-dev
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
FROM sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/ilogtail-community-edition/ilogtail-build-linux:2.0.3 as build

USER root
WORKDIR /backend
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里的目录变了,得改一下


COPY . .

RUN go env -w GOPROXY="https://goproxy.cn,direct" \
&& go mod tidy

CMD sh -c "go build -o ConfigServer ./cmd && ./ConfigServer"


69 changes: 69 additions & 0 deletions config_server/service/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# Config-server

本项目针对https://github.com/alibaba/ilogtail/tree/main/config_server/protocol/v2 中提到的Agent行为进行了V2版本的适配,基本实现了能力报告、心跳压缩、配置状态上报等功能;并适配用户端实现了Config-server的前端页面Config-server-ui,项目见[ui](../ui/README.md)。
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

文档移到docs目录里,注意把对应的链接修改一下。另外markdown格式化一下

## 快速开始
### 基本配置
下载ilogtail源码,进入`config_server/service`目录
#### 数据库配置

由于要求ConfigServer存储Agent的基本信息、PipelineConfig、InstanceConfig等配置信息,所以首先需要对数据库进行配置。打开`cmd/config/prod`文件夹,编辑`databaseConfig.json`(若不存在,请先创建),填入以下信息,其中`type`为数据库的基本类型 (基于gorm我们适配了`mysql`、`sqlite`、`postgres`、`sqlserver`四种数据库);其余均为数据库配置信息;`autoMigrate`字段默认为`true`,若先前建好业务相关的表,则设置为`false`。配置完成后,请自行创建可与ConfigServer连通的数据库实例,并创建名字为`dbName`的数据库(若按照docker compose方式启动,无需任何操作,参考[deployment](../deployment/README.md))。
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

leveldb还支持吗

```json
{
"type": "mysql",
"userName": "root",
"password": "123456",
"host": "mysql",
"port": 3306,
"dbName": "test",
"autoMigrate": true
}
```

#### ConfigServer配置

打开`cmd/config/prod`文件夹,编辑`serverConfig.json`(若不存在,请先创建),填入以下信息,其中`address`为应用程序运行的ip和端口;`capabilities`标识Server端拥有的能力,`responseFlags`支持两种配置采集方式,详情见[protocol](https://github.com/alibaba/ilogtail/blob/main/config_server/protocol/v2/README.md);若长时间没接收到某个Agent的心跳检测(超时上限为`timeLimit`,单位是秒),server端自动下线该Agent。
```json
{
"address": "0.0.0.0:9090",
"capabilities": {
"rememberAttribute": true,
"rememberPipelineConfigStatus": true,
"rememberInstanceConfigStatus": true,
"rememberCustomCommandStatus": false
},
"responseFlags": {
"fetchPipelineConfigDetail": true,
"fetchInstanceConfigDetail": false
},
"timeLimit":60
}
```
### 运行

按照**基本配置**进行数据库与ConfigServer的配置,进入根目录,运行下面的命令

```shell
go build -o ConfigServer ./cmd

./ConfigServer
```
本项目同样支持docker方式启动应用,Dockerfile见`Dockerfile`(不建议使用,通过docker compose一键部署脚本,可以更快实现应用搭建,参考[deployment](../deployment/README.md))。
### 启动Agent

Agent 侧需要配置 ConfigServer 信息,才能使用管控功能。从[releases](https://github.com/alibaba/ilogtail/releases) 下载latest版本,配置`ilogtail_config.json`(若不存在,请先创建),添加如下信息,其中`endpoint_list`为ConfigServer的集群地址,并启动Agent。
```json
{
"config_server_list": [
{
"cluster": "community",
"endpoint_list": [
"127.0.0.1:9090"
]
}
]
}
```
若Agent以docker方式启动(安装与启动见[start-with-container](https://github.com/alibaba/ilogtail/blob/main/docs/cn/installation/start-with-container.md)),务必保证Agent与ConfigServer的网络连通,可选择以下两种配置方案:
* 启动时配置Agent与ConfigServer在同一docker网络`server`中,此时`endpoint_list:["config-server:9090"]`。
* 启动时配置`docker.internal.host`映射,此时`endpoint_list:["docker.internal.host:9090"]`。

9 changes: 9 additions & 0 deletions config_server/service/cmd/config/prod/databaseConfig.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"type": "mysql",
"userName": "root",
"password": "123456",
"host": "mysql",
"port": 3306,
"dbName": "test",
"autoMigrate": true
}
14 changes: 14 additions & 0 deletions config_server/service/cmd/config/prod/serverConfig.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"address": "0.0.0.0:9090",
"capabilities": {
"rememberAttribute": true,
"rememberPipelineConfigStatus": true,
"rememberInstanceConfigStatus": true,
"rememberCustomCommandStatus": false
},
"responseFlags": {
"fetchPipelineConfigDetail": true,
"fetchInstanceConfigDetail": false
},
"timeLimit": 60
}
3 changes: 3 additions & 0 deletions config_server/service/cmd/env.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"env": "prod"
}
16 changes: 16 additions & 0 deletions config_server/service/cmd/main.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
package main

import (
"config-server/config"
"config-server/router"
"github.com/gin-gonic/gin"
)

func main() {
r := gin.Default()
router.InitAllRouter(r)
err := r.Run(config.ServerConfigInstance.Address)
if err != nil {
panic(err)
}
}
63 changes: 63 additions & 0 deletions config_server/service/common/api_error.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
package common

import (
"errors"
"fmt"
)

type ApiError struct {
Code int
Message string
}

func (apiError ApiError) Error() string {
return fmt.Sprintf("Code:%d,Messgae:%s", apiError.Code, apiError.Message)
}

func ErrorWithMsg(code int, msg string) *ApiError {
var apiErrorResult = ApiError{
Code: Failed.Code,
Message: Failed.Message,
}
if code != 0 {
apiErrorResult.Code = code
}
if msg != "" {
apiErrorResult.Message = msg
}
PrintLog(apiErrorResult.Code, apiErrorResult.Message, 3)
return &apiErrorResult
}

func ServerErrorWithMsg(msg string, a ...any) *ApiError {
if a == nil || len(a) == 0 {
return ErrorWithMsg(Failed.Code, msg)
}
return ErrorWithMsg(Failed.Code, fmt.Sprintf(msg, a...))
}

func ServerError() *ApiError {
return ErrorWithMsg(Failed.Code, Failed.Message)
}

//这里切记不要返回APIError,否则会出现问题(因为(ApiError)(nil)!=nil)

func SystemError(err error) error {
if err == nil {
return nil
}

var apiError *ApiError
if errors.As(err, &apiError) {
return ErrorWithMsg(apiError.Code, apiError.Message)
}
return ErrorWithMsg(SystemFailed.Code, err.Error())
}

func ValidateErrorWithMsg(msg string) *ApiError {
return ErrorWithMsg(ValidateFailed.Code, msg)
}

func ValidateError() *ApiError {
return ErrorWithMsg(ValidateFailed.Code, ValidateFailed.Message)
}
23 changes: 0 additions & 23 deletions config_server/service/common/data_type.go

This file was deleted.

35 changes: 0 additions & 35 deletions config_server/service/common/http_response.go

This file was deleted.

25 changes: 25 additions & 0 deletions config_server/service/common/http_status.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
package common

import (
"github.com/gin-gonic/gin"
)

type HttpStatus struct {
Code int
Message string
}

var (
Takuka0311 marked this conversation as resolved.
Show resolved Hide resolved
Success = HttpStatus{Code: 200, Message: "success"}
Failed = HttpStatus{Code: 500, Message: "failed"}
SystemFailed = HttpStatus{Code: 505, Message: "systemFailed"}
ValidateFailed = HttpStatus{Code: 404, Message: "parameter is not"}
)

func ErrorProtobufRes(c *gin.Context, res any, err error) {
c.ProtoBuf(Success.Code, res)
}

func SuccessProtobufRes(c *gin.Context, res any) {
c.ProtoBuf(Success.Code, res)
}
Loading
Loading