Open in app

Sign in

Write

Sign in

AWS 環境部署：mongoDB

莉森羊
13 min readAug 22, 2017

--

目標

將工作站的mongoDB轉移到AWS環境中

流程

AWS userdata(環境部屬) → AMI → 將DB資料塞入 → 建置備份還原機制 → 檢查測試

創建Mongo instance：使用Userdata將原先從工作站倒出的資料從 S3 bucket引入，並安裝docker、cloudwatch agent。

使用docker重建mongo環境，使用cloudwatch agent回傳mongo log，每兩日備份資料回 S3 bucket。

Tool

Docker, AWS EC2, AWS S3, MongoDB

實作時間

2016.08

作法

✍ 前置作業：工作站搬移到AWS

・於local測試環境使用centOS7裝docker 1.11.2

・先登入工作站將Mongo的data複製並壓縮至本機端

cp -r <工作站路徑>/mongo/. <本機端路徑>/mongo
tar -jcv -f mongodata.tar.bz2 mongodata

・scp至工作站(M1)，再用M1機上傳至AWS S3

scp mongodata.tar.bz2 root@<ip>:/tmp/mongodata/
cd /tmp/

・準備aws accesskey

curl -O https://bootstrap.pypa.io/get-pip.py
sudo python get-pip.py
sudo pip install awscli
aws configure
# 輸入aws key 來上傳

✍ Userdata：

✎ version 1 : 第一次建置空環境，無公司資料的 mongo userdata 1

設定檔位置: s3://<bucket_name>/maintain/mongo/userdata/MG_AMI.sh

#!/bin/bash
# mongo environment install
# install docker and start docker on boot let ec2-user use
destination=/home/ec2-user
yum update -y
python -m pip install pymongo
yum install -y docker
service docker start
usermod -a -G docker ec2-user
chkconfig docker on# copy s3 docker to ec2
aws s3 cp s3://<bucket_name>/maintain/mongo/docker/mongo.zip $destination# unzip mongo docker data
unzip $destination/mongo.zip -d $destination# build docker image
chmod -R 755 $destination/mongo
docker build -t mongo3.2 $destination/mongo

✎ version 2: 用AMI後所用的userdata 2 ( 第一階段部屬 )

把工作站資料從S3拉下，並開啟container，將之匯入，每次開機時，這個container都會被啟動，且這個userdata 將資料自動備份回S3 (兩天一次的頻率)。

設定檔位置:

s3://<bucket_name>/maintain/mongo/userdata/MG_MVDATA.sh

#!/bin/bash
destination=/home/ec2-user
# copy AWS S3 mongo database data to ec2
aws s3 cp s3://<bucket_name>/maintain/mongo/backup/mongodata.tar.bz2 $destination# unzip mongo database data to mongo docker folder
tar -C $destination/mongo -jxvf $destination/mongodata.tar.bz2# delete mongodata.tar.bz2
rm -rf $destination/mongodata.tar.bz2# start docker container and on boot start container
docker run — name mongo — restart=always -p 27017:27017 -v "$destination/mongo/mongodata":/data/db -v "/tmp/logs":/var/log/mongodb/ -d mongo3.2 — logpath /var/log/mongodb/app.log# copy crontab data and setting
aws s3 cp s3://<bucket_name>/maintain/mongo/cronjob/autobackup.sh $destination
aws s3 cp s3://<bucket_name>/maintain/mongo/cronjob/flush.py $destination
aws s3 cp s3://<bucket_name>/maintain/mongo/cronjob/unlock.py $destination
aws s3 cp s3://<bucket_name>/maintain/mongo/cronjob/autodeletelog.sh $destination
aws s3 cp s3://<bucket_name>/maintain/mongo/cronjob/crontab-setting.conf /var/spool/cron/ec2-user

✎ version 3: 用AMI後所用的userdata 3( 第二階段執行例行性工作日常備份及損壞啟動 )

設定檔位置:

s3://<bucket_name>/maintain/mongo/userdata/MG_RE.sh

#!/bin/bash 
destination=/home/ec2-user
# sync s3backupdata to ec2
aws s3 sync s3://<bucket_name>/maintain/mongo/backup/mongodata-lastest
$destination/mongo/mongodata# start docker container and on boot start container
docker run — name mongo — restart=always -p 27017:27017 -v "$destination/mongo/mongodata":/data/db -v "/tmp/logs":/var/log/mongodb/ -d mongo3.2 — logpath /var/log/mongodb/app.log# copy crontab data and setting
aws s3 cp s3://<bucket_name>/maintain/mongo/cronjob/autobackup.sh $destination
aws s3 cp s3://<bucket_name>/maintain/mongo/cronjob/flush.py $destination
aws s3 cp s3://<bucket_name>/maintain/mongo/cronjob/unlock.py $destination
aws s3 cp s3://<bucket_name>/maintain/mongo/cronjob/autodeletelog.sh $destination
aws s3 cp s3://<bucket_name>/maintain/mongo/cronjob/crontab-setting.conf /var/spool/cron/ec2-user

✍ 備份MongoDB腳本：

aws s3 sync

設定檔位置:

s3://<bucket_name>/maintain/mongo/cronjob/autobackup.sh

#!/bin/bash
destination=/home/ec2-user
# flush database in memory data to disk
python $destination/flush.py
 
# sync data to s3
DATE=`date +%Y%m%d`
aws s3 sync $destination/mongo/mongodata s3://<bucket_name>/maintain/mongo/backup/mongodata-$DATE --delete
 
# update s3 latest data
aws s3 sync s3://<bucket_name>/maintain/mongo/backup/mongodata-$DATE
s3://<bucket_name>/maintain/mongo/backup/mongodata-lastest --delete
 
#unlock db
python $destination/unlock.py

pymongo-flush & lock db

設定檔位置:

s3://<bucket_name>/maintain/mongo/cronjob/flush.py

import pymongo
from pymongo import MongoClient#connect the mongo database
client = MongoClient(“localhost”, 27017)
the_client =MongoClient()
the_client.fsync(lock=True)
#the_client.is_locked 可檢查是否有lock

pymongo-unlock db

設定檔位置:

s3://<bucket_name>/maintain/mongo/cronjob/unlock.py

import pymongo
from pymongo import MongoClient#connect the mongo database
client = MongoClient(“localhost”, 27017)
the_client =MongoClient()
the_client.unlock()
 #the_client.is_locked 可檢查是否有lock

連上ssh agent

#測試環境
eval $(ssh-agent)
ssh-add KeyPair_Bastion.pem
ssh-add KeyPair_MongoDB.pem
ssh-add KeyPair_Maintain.pem
ssh -A -i KeyPair_Bastion.pem ec2-user@<Bastion_Server_ip>
ssh ec2-user@<mongo_ip>
 
#生產環境
eval $(ssh-agent)
ssh-add KeyPair_Bastion.pem
ssh-add KeyPair_MongoDB.pem
ssh-add KeyPair_Maintain.pem
ssh -A -i KeyPair_Bastion.pem ec2-user@<Bastion_Server_domain>
ssh ec2-user@<mongo_domain>

✍ 備份Log腳本

note:

  1. rm 掉log檔後 要docker restart mongo才會有全新的log
  2. 定期(每五天)備份與刪除/tmp/logs中log

設定檔位置:
s3://<bucket_name>/maintain/mongo/cronjob/autodeletelog.sh

#!/bin/bash
destination=/tmp/logs
#backup log from ec2 to s3
DATE=`date +%Y%m%d`
aws s3 sync $destination s3://<bucket_name>/maintain/mongo/backup/log-$DATE --delete#remove old log on ec2
sudo rm -r $destination
 
#restart container to generate new app.log
docker restart mongo

例行工作設定(crontab)
設定檔位置:
s3://<bucket_name>/maintain/mongo/cronjob/crontab-setting.conf

# for data backup 每隔2天凌晨1點執行備份一次 
0 1 */2 * * sh /home/ec2-user/autobackup.sh 
# for log backup 每5天凌晨1點執行備份與清理 
0 1 */5 * * sh /home/ec2-user/autodeletelog.sh

✍ 檢查測試

✎ 檢查用userdata 1 建置的空環境

# check mongo的docker image是否存在
docker images# check pymongo是否安裝 
python
>> import pymongo  #順利import的話就ok

✎ 檢查用userdata 2 的功能

# docker container是否啟動
docker ps# 腳本檔案是否進到ec2
ls  #看看是否出現autobackup.sh flush.py mongo mongo.zip unlock.py autodeletelog.sh
sudo cat /var/spool/cron/ec2-user# 檢查data資料是否匯入&log檔
# 檢查host端的db和log檔
cd mongo/mongodata/
ll
cat /tmp/logs/app.log 
 
# 檢查docker的mongo端的db和log檔
docker exec -it mongo bash 
cd /data/db
ls
cat /var/log/mongodb/app.log
 
# 同時進shell檢查是否有mapping進資料庫<重要!!>
mongo
show dbs# 備份功能測試
sh /home/ec2-user/autobackup.sh
sh /home/ec2-user/autodeletelog.sh
 
# 查看今日備份是否有更新到mongodata-lastest
aws s3 ls s3://<bucket_name>/maintain/mongo/backup/mongodata-20160708/
aws s3 ls s3://<bucket_name>/maintain/mongo/backup/mongodata-lastest/
aws s3 ls s3://<bucket_name>/maintain/mongo/backup/log/

✎ 檢查用userdata 3 的功能

# docker container是否啟動
docker ps# 腳本檔案是否進到ec2
ls # 看看是否出現autobackup.sh flush.py mongo mongo.zip unlock.py autodeletelog.sh
sudo cat /var/spool/cron/ec2-user# 損壞啟動，檢查最新備份資料是否有還原進ec2
aws s3 ls s3://cirrusdi-sw-log-backup/maintain/mongo/backup/mongodata-lastest
cd mongo/mongodata/
ll# 備份功能測試
sh /home/ec2-user/autobackup.sh
sh /home/ec2-user/autodeletelog.sh
AWS
Docker
Mongodb
Backend

--

--

莉森羊
莉森羊

Written by 莉森羊

65 Followers
56 Following

No responses yet

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams