Requirement: We wanted to send logs to splunk (A LOT of logs) and to do it with a docker container running fluentbit.
*for a more detailed discussion of why use Fluent-bit vs FluentD, you can check out https://logz.io/blog/fluentd-vs-fluent-bit/.
However for this scenario Fluent-Bit was the better choice.
Deliverable: Each VM has 5-20 Docker containers on it, and one of them will be the fluent-bit container. It will send the logs from all the containers required, not exceeding 500Gb a day, a quota shared by the amount of containers. Example: if there are 10 containers, then each can send 50 a day, however if there are 15 containers, each can send 33.3Gb a day.
If a container is stopped, we need to know at what point the stop happened (this happens through the fluentbit configuration).
Architecture of System:
The Dockerfile builds a container which monitors logs on a VM which runs between 1 and 20 containers on it. (there is no “hard” limit, can be 100 containers or more)
Since docker container terminate after the main process on them is sent a SIGHUP or any signal 1-15, we had to use a supervisord program which run as PID 1, and encapsulates the fluentbit process within it.
Supervisord:
https://gdevillele.github.io/engine/admin/using_supervisord/
This is the structure of the container build:
• Dockerfile – the file from which the container is built
• config.sh – main configuration file for the monitor-log.sh script
• fluent-bit.conf – initial fluent-bit configuration
• inputs.conf – the file which defines all the containers
• metadata.lua – filtering file
• monitor-log.sh – main script which runs every 10min
• oneopsscript.sh – script file that strips the logs to what is needed.
• supervisord.conf – the supervisord daemon config
• uncomment-fluent-config.sh – script that runs at 12am and resets the count for the next day.
Running the container:
To run the monitoring container you need to do the following:
1. # docker build -t fbit .
The arguments for this command are:
The docker build command builds Docker images from a Dockerfile and a “context”. A build’s context is the set of files located in the specified PATH
-t is: Name and optionally a tag in the 'name:tag' format
2. # docker run -v /data/weiotadm/docker/lib/containers/:/data/we/docker/lib/containers/ -v /data/:/home/prom/ --name=splunk_fbit122 fbit
This runs the container you just compiled in step #1, with mounting the directories
-v or --volume: Consists of three fields, separated by colon characters (:). The fields must be in the correct order, and the meaning of each field is not immediately obvious.
• In the case of bind mounts, the first field is the path to the file or directory on the host machine.
• The second field is the path where the file or directory is mounted in the container
• The third field is optional, and is a comma-separated list of options, such as ro, z, and Z
Note: If you use -v or --volume to bind-mount a file or directory that does not yet exist on the Docker host, -v creates the endpoint for you. It is always created as a directory.
If you use --mount to bind-mount a file or directory that does not yet exist on the Docker host, Docker does not automatically create it for you, but generates an error.
The Docker File explained:
config.sh explained
#!/bin/bash
# run every 1-5 minutes to monitor log
## cat config.v2.json | grep -o '"Name":"[^"]*' | grep -o '[^"]*$'
# load config
source ./config.sh
echo "Monitoring Log Size"
count=0
for entry in "$log_dir"/*; do
if [ -d $entry ]; then
count=$((count+1))
container_folder_name=`basename $entry`
main_log_file_name=$container_folder_name-json.log
main_log_file_path=$entry/$main_log_file_name
if [ ! -f $main_log_file_path ]; then
continue
fi
check_config="$(grep -wn "\(^\s.*\|^\|^\#.*\|^\s.*\#.*\)Path.*$container_folder_name" $fluent_config_file | cut -d: -f1)"
echo $check_config
if [ -z "$check_config" ]; then
## add more INPUT configure if it does not exits
echo "
[INPUT]
name tail
Path $entry/*.log
Parser json
Skip_Empty_Lines true
Tag_Regex (.*\/(?<container_id>.*)-json\.log)
Tag docker.<container_id>
Docker_Mode true
Read_from_Head true
Mem_Buf_Limit 5000MB
Buffer_Chunk_Size 250k
Buffer_Max_Size 500k
Refresh_Interval 10" >> $fluent_config_file
fi
## if file_size > 50MB --> remove fluent config
##
tag_lines="$(grep -wn "^\s*.*\[\([A-Z]\)*\]\|^\s*\@[A-Z].*" $fluent_config_file | cut -d: -f1)"
log_config_line="$(grep -wn "\(^\s.*\|^\)Path.*$container_folder_name" $fluent_config_file | cut -d: -f1)"
echo "tag_line=" $tag_lines
if [ ! -z "$log_config_line" ]; then
echo "Log config line: " $log_config_line
today=`date +"%Y-%m-%d"`
## get container name & max size
config_json_file=$entry'/config.v2.json'
container_name=`cat ${config_json_file} | grep -o '"Name":"[^"]*' | grep -o '[^"/]*$'`
if [ ${!container_name} ]; then
max_file_size_byte=$((${!container_name}*1024*1024))
else
## get default max size
max_file_size_byte=$(($default_max_file_size*1024*1024))
fi
echo "max_file_size_byte=" $max_file_size_byte
## calculate log size today using grep, cat, sed
##file_size=`grep "${today}" $main_log_file_path | egrep "${log_type_pattern}" | wc -c`
file_size=`sed -n "/${today}/p" $main_log_file_path | egrep "${log_type_pattern}" | wc -c`
##file_size=`cat $main_log_file_path | grep "${today}" | egrep "${log_type_pattern}" | wc -c`
# write file_size into file
date_format=`date +"%Y%m%d"`
#echo ${file_size} > $entry'/size'${date_format}'.txt'
echo "log size of container: $container_name=" $file_size "Byte, Max="$max_file_size_byte;
if [ $file_size -lt $max_file_size_byte ]; then
continue
fi
#start line & end_line to remove configure
start_line=0
end_line=0
for input_line in $tag_lines
do
if [ $log_config_line -gt $input_line ]; then
echo "[INPUT] start at line:" $input_line
start_line=$input_line
continue
else
# less than input_line
end_line=$((input_line-1))
break
fi
done
if [[ $start_line -gt 0 && $end_line == 0 ]]; then
end_line=`wc -l $fluent_config_file | cut -d' ' -f1`
fi
if [[ $start_line -gt 0 && $end_line -gt 0 ]]; then
echo "Comment from: "$start_line "to" $end_line
sed -i -e ''"${start_line}"','"${end_line}"'s/^/#/' $fluent_config_file
killall -HUP fluent-bit
fi
fi
fi
done
This is the uncomment script:
# It uncomments fluent-bit [INPUT] configuration
# load config
source ./config.sh
# find comment INPUT & remove the comment from them.
tag_lines="$(grep -wn "^\s*.*\[\([A-Z]\)*\]\|^\s*\@[A-Z].*" $fluent_config_file | cut -d: -f1)"
echo "tag_lines="$tag_lines
count=0
for entry in "$log_dir"/*; do
if [ -d $entry ]; then
count=$((count+1))
container_folder_name=`basename $entry`
main_log_file_name=$container_folder_name-json.log
main_log_file_path=$entry/$main_log_file_name
log_config_line="$(grep -wn "\(^\#.*\|^\s.*\#.*\)Path.*$container_folder_name" $fluent_config_file | cut -d: -f1)"
#start line & end_line to remove configure
start_line=0
end_line=0
if [ -z "$log_config_line" ]; then
continue
fi
echo "log_config_line=" $log_config_line
for tag_line in $tag_lines
do
if [ $log_config_line -gt $tag_line ]; then
echo "[INPUT] start at line:" $tag_line
start_line=$tag_line
continue
else
end_line=$((tag_line-1))
break
fi
done
if [[ $start_line -gt 0 && $end_line == 0 ]]; then
end_line=`wc -l $fluent_config_file | cut -d' ' -f1`
fi
if [[ $start_line -gt 0 && $end_line -gt 0 ]]; then
echo "uncomment from: "$start_line " to " $end_line
sed -i -e ''"${start_line}"','"${end_line}"'s/.*#//' $fluent_config_file
killall -HUP fluent-bit
fi
fi
done
No comments:
Post a Comment