With Ansible and Git Umut Bozkurt, 20.11...Standardizing Oracle Environment With Ansible and Git...

Standardizing Oracle Environment

With Ansible and Git

Umut Bozkurt, 20.11.2019

−Problem

− Increasing number of databases

− Increasing security requirements

− Consistent number of DBAs

−Solution

− Increasing number of DBAs

− Standardization

− Automation

30

40

50

60

70

80

90Oracle Database Size (TB)

About me…

17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank4

−Umut Bozkurt

− Database Engineer

− Swiss National Bank

−More than 18 years of Oracle Database experience

− from database development to Oracle hardware

Integrating Ansible Tower

Oracle Infrastructure

−Current platform: IBM AIX, LPAR

−Next platform: Red Hat, VMware ESX

−Oracle technologies

− Single instance

− Multitenant

− Data Guard

− Automatic Storage Management

− Oracle Enterprise Manager

− Automation with shell scripts


What is Ansible?

− Ansible is an automation language and engine

− Ansible is agentless and uses OpenSSH or WinRM

− Ansible terminology:

− Playbooks files contains plays, in YAML

− YAML is a data serialization standard that can be used in

conjunction with all programming languages

− Plays consists of tasks and ensures that managed hosts

are in a particular state

− Tasks calls modules

− Tasks runs sequentially

− Playbooks are executed from the control node

− Managed hosts are listed in the inventory


- name: httpd server installation

hosts: all

become: yes

become_user: root

tasks:

- name: httpd server is installed

yum:

name: httpd

state: present

- name: httpd is started and enabled

service:

name: httpd

state: started

enabled: true

What is Ansible Tower?

− Ansible Tower is a web console and REST API

− Ansible Tower is a commercial offering from Red Hat

− Ansible AWX is the upstream project of the Ansible Tower

− Ansible Tower features:

− Role based access control

− Job scheduling

− Workflows

− Credential management

− Logging and auditing

− Real time and historical job status reporting

− Notifications

− REST API


Ansible Tower Organization

−Credentials

− Git repository credentials

− Server credentials

− Projects

− Git connection

− Git credential

− Git branch

− Inventories

− Managed hosts

− Groups

− Variables

Templates


Security Concept - Personal Users


Authentication

AD account 1 +

Smart Card certificate

Authentication

AD account 2 +

Smart Card certificate

Authentication

Unix account + RSA SecurID

Authorization

Database Server privileges

Change Management

># Database Server># ---------------> logged in as umut> sudo su - oracle

># Database Server># ---------------> logged in as umut> sudo su -

># Database Server># ---------------> logged in as umut> sudo su -

># Jump Host># ---------------

> ssh umut@dbsrv

Authentication

AD account 3

−Running Ad-Hoc commands only with the personal accounts

Security Concept - Jobs

− Scheduling jobs only with a technical user and a privileged

credential

− Scheduler user

− Only execute privileges on the assigned template

−Credential

− Encrypted and stored in the Ansible Tower database

− Can only be used from the Tower GUI or API

− Exclusive for the scheduler user


# Database Server# ---------------> cat authorized_users...from "10.24.23.11" ssh-rsa AKCB3NzaC1yc...

# Database Server# ---------------> cat sshd_config... AllowUsers [email protected]...

# Database Server# ---------------> sudo su -

Security Concept - Operations


− Execute privileges on a template allows running the template. Viewing or modifications are

not possible

− Surveys allows operations team or end users to run the templates with input parameters

Standardizing Oracle Environment

Standardization

− IT standardization is a strategy for minimizing the IT

costs by keeping the hardware and software as

consistent as possible

−Our goal is to consistently push and (if changed)

overwrite both our scripts and configuration files to the

database servers

− This will lead us to

− easy automation

− increase security

− reduce human errors


># Development Server># ------------------

># Database Server># ---------------

># Database Server># ---------------

># Database Server># ---------------

># Database Server># ---------------

># Database Server># ---------------

># Database Server># ---------------

># Database Server># ---------------

># Database Server># ---------------

># Database Server># ---------------

># Database Server># ---------------

Standardization with Ansible and Git

−Git

− Playbooks, inventory scripts

− Files to push to the servers

− LDAP

− Server and lifecycle information

−CMDB

− Oracle instance information

−Oracle Enterprise Manager

− Monitoring


># Database Server># ----------------

># Database Server># ----------------

># Database Server># ----------------

LDAP CMDB

># Ansible Tower ># Rest API># ---------------

curl -X GET …

Git Projects

Files to push to the servers


oracle_distribution

oracle_automation

Scripts for automation, like playbooks,

dynamic inventory scripts,..

Protected branches cannot be deleted and cannot be force pushed

Inventory

−Managed hosts are defined in the inventory

−Hosts are organized by groups

− The inventory can be defined either

− in a static text file

− dynamically by scripts, from the external sources


# My Static Inventory# -------------------

[PROD]

dbserver01

dbserver02

dbserver03

[TEST]

dbserver05

dbserver06

[AVALOQ]

dbserver01

dbserver05

[SAP]

dbserver03

dbserver06

Instance Information in the Inventory

− In most cases, extending the inventory with the instance

information is helpful

− Instance information can be gathered either

− directly from the managed hosts (custom facts)

− from the external inventories


# My Static Inventory# -------------------

[PROD]

dbserver01 instance=AAAL HOME=/u01/...

dbserver02 instance=DB01 HOME=/u01/...

dbserver03 instance=PRQ HOME=/PRQ/...

[TEST]

dbserver05 instance=AAAQ HOME=/u01/...

dbserver06 instance=SRQ HOME=/SRQ/...

[AVALOQ]

dbserver01 instance=AAAL HOME=/u01/...

dbserver05 instance=AAAQ HOME=/u01/...

[SAP]

dbserver03 instance=PRQ HOME=/PRQ/...

dbserver06 instance=SRQ HOME=/SRQ/...

Ansible Facts

− Facts are

− variables that are automatically discovered on the

managed host

− collected before executing the first task

− Fact information can be accessed later in the tasks


PLAY [Install Oracle PDB] *************

TASK [Gathering Facts]***************

ok: [birs1z.snb.ch]

TASK [Create PDB] *******************

changed: [birs1z.snb.ch]

PLAY [Install Oracle PDB] *************

TASK [Gathering Facts]***************

ok: [server1.ubozkurt.com]

TASK [Create PDB] *******************

changed: [server1.ubozkurt.com]

Custom Facts


−Custom facts

− allows us to gather additional information (like instance information)

from the managed hosts

− are either static files or scripts stored in the managed host

−Custom fact scripts must

− be executable

− have the extension ".fact”

− output in JSON

− be placed in /etc/ansible/facts.d directory on the managed host

>./local_script.fact

{"instances": [

{"name": "DBMS01CD","oracle_home": "…","version": "19c"

},{

"name": "DBMS02CD","oracle_home": "…","version": "18c"

}]

}

Dynamic Inventory


− Inventory scripts

− output in JSON

− mandatory arguments:

− --list

− --host <hostname>

− runs on the Ansible controller

− does not accept input parameters. But

you can work with environment

variables

> ./snb_invent.py --list

{“production": {

"hosts": ["prdserver1.snb.ch","prdserver2.snb.ch"

],"vars": {}

},“development": {

"hosts": ["devserver3.snb.ch"

]"vars": {}

}}

> ./snb_invent.py \--host prdserver2.snb.ch

{"instances": [

{"name": "DBMS01CD","oracle_home": "…“,"version": "19c"

},{

"name": "DBMS02CD","oracle_home": "…“,"version": "18c"

}]

}

Clone Git Repository to the Ansible Tower


># Ansible Tower>------------------

> ls -la tower_fs/oracle_distribution

.gitu01_app_oraclehome_oracle

- name: Clone Git project oracle_distribution

hosts: localhost

connection: local

gather_facts: false

tasks:

- name: Clone dev branch to Ansible Tower file system

git:

repo: 'http://ansitower:vk6z@gitlab/dbms/oracle_distribution.git'

dest: /tower_fs/

version: dev

force: yes

depth: 1


># Database Server>------------------

> ls -l /u01/app/oracle/snb

basenvbindbsetchistoryinstalllog -> /var/log/dbmsrcvsqltmp

- name: Oracle File Distribution

hosts: all

become: yes

become_user: oracle

gather_facts: true

gather_subset: min

tasks:

- name: Synchronize from the Tower file system to the managed host

synchronize:

src: "/tower_fs/oracle_distribution/u01_app_oracle/snb/{{ item }}"

dest: "/u01/app/oracle/snb/"

checksum: yes

delete: yes

times: yes

recursive: yes

rsync_opts: "--chown=oracle:dba"

loop:

- basenv

- bin

- install

- rcv

- sql

Distribute files from Ansible Tower to the Servers

Configuration files with Jinja2 Templates


− A template contains variables which are replaced by their values when the template is

rendered

− Ansible uses Jinja2 templating

<title>{% block title %}{% endblock %}</title><ul>{% for user in users %}

<li><a href="{{ user.url }}">{{ user.username }}</a></li>{% endfor %}</ul>

listener.ora with Jinja2 Templates


# Jinja2 Template # -----------------------------------LISTENER=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST={{ ansible_fqdn }})(PORT=1599)))

SID_LIST_LISTENER=(SID_LIST={% for instance in hostvars[inventory_hostname].instances %}(SID_DESC=(ORACLE_HOME={{ instance.oracle_home }})(SID_NAME={{ instance.name }})){% endfor %})

# Listener.ora# -----------------------------------------------------------------LISTENER=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dbserver.snb.ch)(PORT=1599)))

SID_LIST_LISTENER=(SID_LIST=(SID_DESC=(ORACLE_HOME=/u01/app/oracle/product/home19c)(SID_NAME=DBMS01CD))(SID_DESC=(ORACLE_HOME=/u01/app/oracle/product/home19c)(SID_NAME=DBMS02CD))(SID_DESC=(ORACLE_HOME=/u01/app/oracle/product/home18c)(SID_NAME=DBMS03CD)))

# Output of the Dynamic Inventory # Script# -------------------------------

> ./snb_invent.py --host dbserver.snb.ch{"instances": [{"name": "DBMS01CD","oracle_home": "…","version": "19c"

},{"name": "DBMS02CD","oracle_home": "…","version": "19c"

},{"name": "DBMS03CD","oracle_home": "…","version": "18c"

}]}

# Task in the playbook# ------------------------------------ name: creating listener.ora

template:src: "/tower_fs/oracle_distribution/db00_app_oracle/snb/templates/listener.j2"dest: "/u01/app/oracle/network/admin/listener.ora"owner: oraclegroup: dbamode: '0640'force: yes

Automation with Ansible

Modules


- name: httpd server installation

hosts: all

become: yes

become_user: root

tasks:

- name: httpd server is installed

yum:

name: httpd

state: present

- name: httpd is started and enabled

service:

name: httpd

state: started

enabled: true

># Ansible Tower

># -------------

> cat /usr/lib/python2.7/site-packages/ansible/modules/system/service.py

#!/usr/bin/python

...

options:

name:

description:

- Name of the service.

type: str

required: true

state:

description:

- C(started)/C(stopped) are idempotent actions that will not run

commands unless necessary.

- C(restarted) will always bounce the service.

- C(reloaded) will always reload.

- B(At least one of state and enabled are required.)

type: str

choices: [ reloaded, restarted, started, stopped ]

...

Module Support

− Four types of modules:

− Core: Maintained and supported by Ansible

engineering team

− Network: Maintained and supported by Ansible

Network team

− Certified: Maintained by Red Hat partners. First

level contact is Red Hat

− Community: Not supported under an Ansible

engine subscription


Documentation of the synchronize module

−Organizations requires regulations about using the community modules

Oracle Modules

−One of the most popular community module for Oracle is

Ansible-Oracle-Modules, developed by Mikael Sandström

− Can be downloaded from GitHub

−Oracle also provides Oracle Cloud Infrastructure Ansible

modules


> ls ansible-oracle-modules

oracle_asmdgoracle_asmvoloracle_awroracle_datapatchoracle_dboracle_factsoracle_gi_factsoracle_grantsoracle_joboracle_jobclassoracle_jobscheduleoracle_jobwindoworacle_ldapuseroracle_opatchoracle_parameteroracle_pdboracle_privsoracle_profileoracle_redooracle_roleoracle_rsrc_consgrouporacle_servicesoracle_sqloracle_stats_prefsoracle_tablespaceoracle_user

− Automating database tasks without logging in to each

server

− Database software installation

− Database creation

− Database configuration

− Database patching and upgrade

− Database cloning

− ...

− Base for a self-service platform


># Database Server># ---------------

> Creating User

># Database Server># ---------------

> Cloning PDB

># Database Server># ---------------

> Creating Database

Self Service Portal


curl -X GET …

Oracle Infrastructure Automation with Ansible

># Database Server># ---------------

> Creating User


curl -X GET …

Applying Release Update


- name: Start the instance and apply datapatch

shell: |

export ORACLE_HOME="{{ item.oracle_home }}"

export ORACLE_SID="{{ item.name }}"

$ORACLE_HOME/bin/sqlplus / as sysdba << EOF

startup

EOF

$ORACLE_HOME/OPatch/datapatch -verbose

register: shell_result

changed_when: >

( "'Installing patches...' in shell_result.stdout" ) or

( "'ORACLE instance started.' in shell_result.stdout" )

failed_when: "'ORA-' in shell_result.stdout"

with_items: "{{ hostvars[inventory_hostname]['instances'] }}"

when: item.oracle_home == "/db00/app/product/db19c/db00"

- name: Copy RU from the control host to the servers and unzip

unarchive:

copy: yes

src: "/ansible_fs/p30125133_190000_Linux-x86-64.zip"

dest: "/db00/app/snb/tmp_for_apply_psu"

owner: oracle

group: dba

- name: Apply RU with Ansible-Oracle-Modules

oracle_opatch:

oracle_home: "/db00/app/oracle/product/db19c/db00"

patch_base: "/db00/app/snb/tmp_for_apply_psu/30125133"

opatch_minversion: '12.2.0.1.17'

conflict_check: yes

stop_processes: no

state: present

Ansible Execution


># Ansible Tower># ---------------

> ansible-playbook db.yml -e "oracle_sid=orcl"

># Managed Host># ----------------

> 1. create directory in ~/.ansible/tmp/…> 2. copy module in the created directory> 3. execute> 4. send results in JSON format> 5. remove temporary directory

SSH

−Most of the modules requires python installed on the managed host. Modules can also

have additional requirements

− All playbooks are executed as awx user on the control node

># Managed Host># ----------------

> 1. create directory in ~/.ansible/tmp/…> 2. copy module in the created directory> 3. execute> 4. send results in JSON format> 5. remove temporary directory

Developing Ansible Modules

− You can use any language; however python is the most suitable

− Ansible modules strive for idempotency:

− You can run a playbook on the same host multiple times. When your systems are in the correct

state, the playbook should not make any changes

− Idempotency is not always possible especially when you are calling external programs, like

oracle setup or grid infrastructure root scripts


> mkdir /u01/app

> If (! -d /u01/app); then mkdir /u01/app ; fi

x

Simple Create/Remove Directory Module with bash and python

−Default custom module directory in Ansible Tower is

/usr/share/ansible/plugins/modules

− Ansible calls custom scripts with a parameter file

−Output should be a JSON object

− “failed” should be true in case of a failure; exit value is

not enough

− “changed” key is for idempotency


> ./my_bash_script.ksh /tmp/variables

{ "changed" : false,"msg": "ERROR:cannot create directory…","failed": true }

- name:create directory with a bash modulemy_bash_script:

dirname: "/tmp/new_dir"state: present

#!/usr/bin/ksh

source $1

echo ${dirname}echo ${state}



#!/usr/bin/bash

source $1

typeset -l state=${state:-present}

if !([ "${state}" = "absent" ] || [ "${state}" = "present" ]) then

echo "{ \"failed\": true,"

echo "\"msg\": \"ERROR: state must be absent or present\"}"

exit 1

fi

if [ "${dirname}" = "" ] ; then

echo "{ \"failed\": true, "

echo " \"msg\": \"ERROR: please provide dirname \" }"

exit 1

fi

#!/usr/bin/python

import os

def main():

module = AnsibleModule(

argument_spec = dict(

dirname = dict(required=True, type='str'),

state=dict(default='present',choices=['present','absent'])

)

)

dirname = module.params["dirname"]

state = module.params["state"]



if [ "${state}" = "absent" ] && [ -d "${dirname}" ] ; then

vRetMsg=$( rm -rf ${dirname} 2>&1 )

if [ $? -eq 0 ] ; then

echo "{ \"changed\": true }"

else


echo " \"msg\": \"ERROR: ${vRetMsg}\" }"

exit 1

fi

elif [ "${state}" = "present" ] && [ ! -d "${dirname}" ] then

vRetMsg=$( mkdir ${dirname} 2>&1 )

if [ $? -eq 0 ] ; then

echo "{ \"changed\": true }"

else


echo " \"msg\": \"ERROR: ${vRetMsg}\" }"

exit 1

fi

else

echo " { \"changed\": false }"

fi

exit 0

if state == 'absend' and os.path.exists(dirname):

try:

os.rmdir(dirname)

except OSError as err:

module.fail_json(msg="OS error: {0}".format(err))

else:

module.exit_json(changed=True)

elif state == 'present' and not os.path.exists(dirname):

try:

os.mkdir(dirname)

except OSError as err:

module.fail_json(msg="OS error: {0}".format(err))

else:

module.exit_json(changed=True)

else:

module.exit_json(changed=False)

from ansible.module_utils.basic import *

if __name__ == '__main__':

main()



TASK [Create directory] ************************************************

changed: [srv1.ubozkurt.com]

TASK [Create directory second time] ************************************************

ok: [srv1.ubozkurt.com]

TASK [Create directory without having permission on the fs] ************************************************

fatal: [srv1.ubozkurt.com]: FAILED! => {"changed": false, "msg": "ERROR: mkdir: cannot create directory ‘/umut_new_dir’: Permission denied"}

...ignoring

TASK [Invalid input parameter] ************************************************

fatal: [srv1.ubozkurt.com]: FAILED! => {"changed": false, "msg": "ERROR: state must be absent or present"}

...ignoring

tasks:

- name: Create directory

my_bash_script:

dirname: "/tmp/umut_new_dir"

state: present

- name: Create directory second time

my_bash_script :


state: present

- name: Create directory without having permission on the fs

my_bash_script:

dirname: "/umut_new_dir"

state: present

ignore_errors: true

- name: Invalid input parameter

my_bash_script:


state: "invalid_value"

ignore_errors: true

Summary

− Ansible is a simple, powerful automation engine

− There are many use cases for automating daily tasks

−Role based access control helps to integrate Ansible Tower in to the organizations

−Organization wide usage requires a good security concept


With Ansible and Git Umut Bozkurt, 20.11...Standardizing Oracle Environment With Ansible and Git...

Documents

Transcript of With Ansible and Git Umut Bozkurt, 20.11...Standardizing Oracle Environment With Ansible and Git...