Ansible is leveraged in order to help get the core services up and running on the required servers. These servers and services will be maintained with Ansible.
For newly provisioned servers there is a level of setup that needs to be done before these servers will be able to be targeted with the Ansible roles defined.
The public ssh key for the user running the Ansible playbooks needs to be present on the remote server in order for the playbook to successfully start and run. Generally a user account and password is given for the newly provisioned server that will have super user access, or alternatively the root password will be provided.
Once the key is uploaded to the server using the remote account Ansible will be able to SSH to that server without providing a password, since password-less certificate based authentication is being leveraged.
Specific playbooks have been written to setup these servers to get them into a standardized state that will allow the subsiquint playbooks to be executed in the manner in which they where intended. These initial playbooks are designed to setup the initial user accounts and change the behaviour of SSH, namely disallowing remote root access and only allowing password-less certificate based authentication.
There are the following environments to choose from when running the script.
Within each of these environments, you can choose the following host groups
sudo su - admin
ssh-copy-id root@<server ip/fqdn>
Now go to the new server and changes its password
sudo passwd root
Also make sure that the DNS servers to be used are accessible
vi /etc/resolv.conf
nameserver 8.8.8.8
nameserver 8.8.4.4
#nameserver 1xx.1xx.1xx.x
#nameserver 1xx.1xx.1xx.x
cd /home/admin/ansible
vim environments/setup/hosts
cd ansible
ansible-playbook -i environments/setup/hosts update_admin_key.yml
# The next updates all the packages. It is an optional step and isn't enforced, however it is always recommended to be running the latest patch level on a server. Unless there is a very specific reason this step is highly recommended.
ansible-playbook -i environments/setup/hosts update_host_software.yml
fatal: [192.10.10.3]: FAILED! => {"changed": true, "cmd": ["rpm", "--import", "https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux"], "delta": "0:00:00.782032", "end": "2025-03-19 11:53:21.356508", "msg": "non-zero return code", "rc": 1, "start": "2025-03-19 11:53:20.574476", "stderr": "warning: Signature not supported. Hash algorithm SHA1 not available.\nerror: https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux: key 1 import failed.", "stderr_lines": ["warning: Signature not supported. Hash algorithm SHA1 not available.", "error: https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux: key 1 import failed."], "stdout": "", "stdout_lines": []}
Do the following, you will need to manually ssh to each new server.
rpm --import https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux
dnf update ca-certificates
If you get an error 'Hash algorithm SHA1 not available' then re-activate it using the following
sudo update-crypto-policies --set DEFAULT:SHA1
#You can set this with
ansible-playbook -i environments/setup/hosts update-crypto-policy.yml
It might be needed to reboot the server.
Once this completes you can re-run step 5.
./playbook.sh <environment> basics <hosts in the environment> -e "restart_service=true clean_temp=false reboot=false"
7.1. Download from GIThub ( not working )
git clone https://github.com/ansible-lockdown/RHEL8-CIS.git
7.2. Install the collections needed in requirements.yml ( not working )
ansible-galaxy collection install community.general
ansible-galaxy collection install ansible.posix
ansible-galaxy collection install community.crypto
7.3 Run the playbook ( not working )
ansible-playbook -i hosts-setup rhel-cis/RHEL8-CIS/site.yml
This will run the "basic" Ansible role against a single host named "servername.domain.co.za" found under the prod environment.
sudo su - admin
cd ansible
./playbook.sh <environment> <role> <servergroup>
This will run the "basic" Ansible role against the host group labelled "test" found under the prod environment.
sudo su - admin
cd ansible
./playbook.sh prod basics test
With this method the script will accept an additional parameter for defining the state of environment variables at runtime. These values have the ability to override the ones set in the role and thus can be leveraged to do more advanced tasks that generally aren't wanted/expected during normal playbook execution. Each role will have it's
sudo su - admin
cd ansible
./playbook.sh prod basics test -e "restart_service=true clean_temp=false"
Set up a folder /home/admin/secrets, and create a yaml file for each role that you need with the external parameters
/home/admin/ansible/secrets/opensearch.yml
os_admin_password_hash: '$2y$12$bWOxxxxx....'
os_health_password: 'matching-admin-password'
os_bootstrap_cluster: true
os_security_enabled: true
os_generate_tls: true
os_defer_tls: false
Then call it with
./playbook.sh setup OPENSEARCH cluster -e "@/home/admin/ansible/secrets/opensearch.yml"
Replace opensearch.yml to the appropriate role context.
To access the roles that contain the various playbooks.
sudo su - admin
cd ansible/roles
| Name | Function | Tag options |
|---|---|---|
| BASICS | Basic initial server configuration. | restart_service=true (Restarts the services after running) clean_temp=false Clears the /tmp/ dir) |
| COUCHDB | Installs Couchdb Setup. Currently 3.3.1 | -e "couchdb_clustered=true administrator_password=xxx jwt_key=<jwt_key here> admins_lucene_password=xxx http_secret=" |
| COUCHDB_OPTIMIZE | Set up the optimisation parameters and file limits | ??? |
| ELASTIC | Installs Elastic stack | -e "elastic_setup=true elastic_set_passwords=true" when setting up a new instance. -e "elastic_upgrade_latest=true" when you want to run the upgrade after all the preparations for upgrade has been done |
| KIBANA | Installs Kibana | -e "kibana_set_passwords=true kibana_setup=true KIBANA_ELASTIC_password=xxxxxxxx" when setting up a new instance. -e= "kibana_upgrade_latest=true" when you want to run the upgrade after all the preparations for upgrade has been done |
| OPENSEARCH | Installs Opensearch | -e "@/home/admin/ansible/secrets/opensearch.yml" place the variables in this file |
| KIBANA | Installs Kibana | -e "os_dashboards_cluster_type=multi-node" when setting up a new instance. |
| MYSQL | Installs MySQL version 5.7.30 | -e "mysql_harden=true mysql_setup=true" Note if there is an issue that the server does not want to start, make sure that selinux is set to permissive mode (selinux) |
| MARIADB | Installs MARIADB version 10.6 | -e "mariadb_root_password= mariadb_user_password= mariadb_bind_address= |
| NGINX | Installs Nginx | -e "nginx_setup=true" |
| HAPROXY | Installs Haproxy | -e "haproxy_setup=true" |
| NODEJS | Installs node js and the core betterdata application | ??? |
| REDIS | Installs Redis | ?? |
| RHEL-CIS | Not sure | ??? |
Supports AlmaLinux 8 and 9, retaining the RedHat-family 7 path. Other
platforms are rejected before configuration starts. Run from a Linux Ansible
controller with an Ansible/Python combination that supports the target's Python
version. The role requires Ansible 2.10 or later and ansible.posix; install
collections with ansible-galaxy collection install -r BASICS/requirements.yml.
The version check is a minimum, not a guarantee that every newer controller
supports the Python shipped by every older target.
Packages use state: present, so an installed package is not upgraded on each
run. Ansible selects DNF on AlmaLinux and YUM on EL7. EPEL supplies packages such
as htop, ncdu and fail2ban. PowerTools (AlmaLinux 8) or CRB (AlmaLinux 9) is enabled
for package transactions through BASICS_extra_repositories, without rewriting
repository configuration. The standard AlmaLinux repositories must be available.
Includes net-tools (ifconfig, netstat, route), iproute (ip, ss), bind-utils (dig,
nslookup), traceroute, tcpdump, curl and lsof.
Services use started/enabled states. Changed configuration notifies restart
handlers; unchanged configuration does not restart services. SSH configuration
is validated with sshd -t before replacement. Fail2ban reads the system journal
and uses firewalld rich rules on both AlmaLinux releases. Firewalld is installed,
enabled and started before managing its public-zone SSH rule.
Unused dependency removal is disabled by default. Set BASICS_autoremove: true
to request it; Ansible reports real package removals and failures. Setting
basics: false also skips cleanup.
Use disposable AlmaLinux 8 and 9 VMs with systemd, repository access, and SSH
credentials for a non-root sudo user. The role disables root/password SSH login
and manages the user keys in tasks/users.yml. Ensure the test login remains
usable and any required network services are allowed before running the role.
From the parent roles directory, with inventory group basics_test containing
both VMs:
export ANSIBLE_ROLES_PATH="$PWD"
ansible-playbook -i inventory BASICS/tests/converge.yml --syntax-check
ansible-playbook -i inventory BASICS/tests/converge.yml
ansible-playbook -i inventory BASICS/tests/converge.yml
The second run must show changed=0, failed=0 and unreachable=0 for each VM
with default variables and no external state changes. Also verify ifconfig,
ip, ss, dig, traceroute, tcpdump, curl and lsof are available, and
sshd -t, chronyc tracking and fail2ban-client status sshd succeed.
Static validation does not replace these two real runs. No target hosts are
included with this role.
In environments/opensearch/hosts set up the following details for clustered installation of the servers. In this example three servers will complete all the roles of seed-host,data, ingest which is only suitable for low volume processing
[cluster]
opensearch1 ansible_host=<SSH-IP-1> os_cluster_address=<private-ip-1>
opensearch2 ansible_host=<SSH-IP-2> os_cluster_address=<private-ip-2>
opensearch3 ansible_host=<SSH-IP-3> os_cluster_address=<private-ip-3>
Create the admin user password hash
On your Linux controller, generate one:
sudo dnf install -y httpd-tools
htpasswd -nBC 12 admin
Enter your chosen OpenSearch admin password when prompted. The output looks like:
admin:$2y$12$...
In /home/admin/ansible/secrets/opensearch.yml, set:
os_admin_password_hash: '$2y$12$bWOxxxxx....'
os_health_password: '<password used for admin>'
os_bootstrap_cluster: true #only if you are setting it up for the first time
os_security_enabled: true
os_generate_tls: true
os_defer_tls: false
Set up the environments/setup/hosts with the following
[cluster]
opensearch1 ansible_host=20.107.178.245 os_cluster_address=10.20.3.4
opensearch2 ansible_host=52.158.32.79 os_cluster_address=10.20.3.5
opensearch3 ansible_host=20.67.234.232 os_cluster_address=10.20.3.6
and then call the playbook
./playbook.sh setup OPENSEARCH cluster -e "@/home/admin/ansible/secrets/opensearch.yml"
To check that everything has been set up correctly, run with this option
./playbook.sh setup OPENSEARCH cluster -e "os_bootstrap_cluster=false"
./playbook.sh setup OPENSEARCH <group> -e "cluster_type=single-node"
To install the dashboards on the server 4.231.232.46 to access the OPENSEARCH cluster on 10.20.3.4,10.20.3.5,10.20.3.6 what should the settings be
Use 4.231.232.46 as the Ansible SSH address. Dashboards should bind to that server’s private IP, which you’ll need to substitute below.
In your inventory:
[dashboards]
dashboards1 ansible_host=4.231.232.46
You need to pass the values of multi-node / single to
os_dashboards_cluster_type: multi-node
You need to run
./playbook.sh setup OPENSEARCH_DASHBOARDS dashboards -e "@secrets/dashboards.yml"
os_dashboards_version: '3.9.0'
os_dashboards_cluster_name: kwantu_cluster
os_dashboards_hosts:
- https://10.20.3.4:9200
- https://10.20.3.5:9200
- https://10.20.3.6:9200
os_dashboards_password: "{{ vault_opensearch_dashboards_password }}"
os_dashboards_bind_host: "{{ ansible_facts['default_ipv4']['address'] }}"
os_dashboards_internal_host: "<DASHBOARDS_PRIVATE_IP>"
os_dashboards_generate_tls: true
os_dashboards_pki_dir: "{{ playbook_dir }}/pki/opensearch/{{ env | default('default') }}/{{ os_dashboards_cluster_name }}"
os_dashboards_manage_firewall: false
os_dashboards_allowed_sources: []
Before running, ensure:
Then run:
./playbook.sh <environment> OPENSEARCH_DASHBOARDS dashboards
Open https://<DASHBOARDS_PRIVATE_IP>:5601. The current role supports private access; configuring 4.231.232.46 as the SSH address does not publish the Dashboards interface publicly.
Configures an existing CouchDB installation. Both local.ini and vm.args are
rendered for standalone and cluster modes. Changes notify one service restart.
Cluster membership must be established separately.
Edit secrets/couchdb.yml beside playbook.yml. Supply the
referenced vault variables or replace them with your values, then encrypt with
ansible-vault. The role loads this file automatically. Override
couchdb_ini_secrets_file for another controller-side path.
Set couchdb_ini_mode to standalone or cluster. Standalone defaults to
single_node=true, n=1, q=1 and a loopback Erlang node name. Cluster defaults to
single_node=false, n=3, q=8 and the host default IPv4 address.
Legacy couchdb_standalone/couchdb_clustered selectors work if mode is omitted;
do not enable both.
couchdb_ini_local accepts any section/key mapping, including new sections.
couchdb_ini_vm_args maps complete flags (e.g. '-kernel inet_dist_listen_min')
to values. Use native Erlang syntax strings for expressions. YAML booleans
render lowercase. Set any value to null to omit it; an empty string produces a
VM flag without arguments. Quote times such as '20:00'. Values must be single-line.
Defaults reflect the supplied reference files, including compression, CORS,
replication, compaction windows, Prometheus and VM tuning. Credentials, UUIDs,
JWT keys and cookies are not embedded. Supply a stable UUID per node and shared
cluster authentication secrets/cookies. Adjust the reference's 28 schedulers
for target hardware.
Run ansible-playbook playbook.yml --tags couchdb_ini --ask-vault-pass with your
normal inventory arguments. Files use mode 0640; secret operations suppress
output and diffs. Ownership, service name and destination paths are defaults
that can also be overridden in secrets/couchdb.yml.
./playbook.sh sendp COUCHDB_INI couchdb -e "@secrets/couchdb-sendp.yml"
Installs the latest clouseau 3.5.0. It expects coucdhb to be installed already.
./playbook.sh sendp CLOUSEAU couchdb -e "@secrets/clouseau-sendp.yml"
Added EXISTDB to playbook.yml—the project uses YAML, not XML—and updated playbook.sh to use its project folder and return Ansible failures correctly. YAML and Bash syntax checks passed.
To deploy:
Confirm the target hosts are listed in environments/<environment>/hosts.
Review secrets/existdb.yml:
EXISTDB_admin_current_password if admin already has a password.EXISTDB_update_passwords: false to preserve existing passwords; use true to change them.Confirm roles/EXISTDB/files/kwantuExistdb6.2.tar.gz is present.
From the Ansible project folder on your Linux controller, run:
./playbook.sh sendp EXISTDB apps -e "@secrets/existdb-sendp.yml"
Replace sendp and apps with your environment and inventory target.
Verify on the target server:
sudo systemctl status existdb
sudo journalctl -u existdb -n 100 --no-pager
Open http://<server>:8080/exist/ and test both database accounts.
The role installs Java 8, prepares directories and ownership, extracts eXist-db, enables and starts its service, and configures the accounts. Live deployment remains untested.
The installation will now follow these steps:
artemis service account with no interactive login./opt/artemis/apache-artemis-2.57.0./usr/local/data/artemis/broker, with message data in its data directory and a default maximum heap of 2G.<server-ip>:61616http://<server-ip>:8161/consoleartemis.Run with:
./playbook.sh <environment> ARTEMIS <host-group> -e "@secrets/artemis.yml"
Firewall rules and TLS remain separate configuration steps. Subsequent runs preserve existing broker configuration and messages; creation-time settings won’t automatically change an existing broker.
for ip in 10.20.3.4 10.20.3.5 10.20.3.6 172.16.0.4 172.16.0.5; do
sudo firewall-cmd --permanent --zone=public \
--add-rich-rule="rule family=\"ipv4\" source address=\"$ip/32\" port port=\"9300\" protocol=\"tcp\" accept"
done
sudo firewall-cmd --reload
ansible -m command -a "sudo firewall-cmd --permanent --add-rich-rule='rule family=""ipv4"" source address=""105.233.67.134/32"" port protocol=""tcp"" port=""5984"" accept'" cluster
ansible -m command -a "sudo firewall-cmd --reload" cluster
ansible -m command -a "sudo firewall-cmd --zone=trusted --add-source=41.76.208.248/32 --permanent" cluster
ansible -m command -a "sudo firewall-cmd --reload" cluster
Appserver: Redis, Mysql, Node, exist-db
kwantu02 197.242.147.53
Nginx:
kwantu00 154.0.170.230
DBCluster
User write: kwantu00 154.0.170.230
Read lucene: kwantu03 154.0.170.247
Node Services: kwantu05 154.0.174.153
Reports: existdb, kibana, elastic
kwantu06 154.0.175.164
Staging Server:
kwantu04 154.0.174.125
10.155.78.115