KYPO is a Cyber Range Platform (KYPO CRP) developed by Masaryk University since 2013. KYPO CRP is entirely based on state-of-the-art approaches such as containers, infrastructures as code, microservices, and open-source software, including cloud provider technology – OpenStack. (source).
KYPO CRP is now part of CONCORDIA consortium. CONCORDIA H2020 is a dedicated consortium of over 52 partners from academia, industry and public bodies. The main objective of the project is to lead the integration of Europe’s excellent cybersecurity competencies into the network of expertise to build the European secure, resilient and trusted ecosystem for the Digital Sovereignty of Europe.
The CONCORDIA project released KYPO CRP as open source in 2020. The release of an open-source cyber range is part of CONCORDIA strategy to build the European Trusted, Secure and Resilient Ecosystem for Digital Sovereignty of Europe.
KYPO Cyber Range Platform is the European Commission’s Innovation Radar Prize Winner in the ‘Disruptive Tech’ category.
In this article, I describe how to install KYPO Cyber Range Platform (CRP) on Openstack and Ubuntu Server running on the AWS cloud. I installed OpenStack on Ubuntu with DevStack.
What is a Cyber Range ?
Cyber Range is a platform for cyber security research and education – it is a simulated
representation of an organization’s network, system, tools, and applications connected
in an isolated environment.
Cyber Range (a sort of modelized network or a digital twin of a real network) allows Adversary Emulation, a type of ethical hacking engagement where the Red Team emulates how an adversary operates, leveraging the same tactics, techniques, and procedures (TTPs), against a target organization.
The goal of these engagements is to improve education but also technology and to do some cyber security research.
Adversary emulations are performed using a structured approach following industry methodologies and frameworks (such as MITRE ATT&CK) and leverage Cyber Threat Intelligence to emulate a malicious actor that has the opportunity, intent, and capability to attack the target organization.
What is DevStack ?
DevStack is a modular set of scripts that can be run to deploy a basic OpenStack cloud for use as a demo or test environment. The scripts can be run on a single node that is baremetal or a virtual machine. It can also be configured to deploy to multiple nodes. DevStack deployment takes care of tedious tasks like configuring the database and message queueing system, making it possible for developers to quickly and easily deploy an OpenStack cloud.
By default, the core services for OpenStack are installed but users can configure additional services to be deployed. All services are installed from source. DevStack will pull the services from git master unless configured to clone from a stable branch (i.e. stable/pike).
Devstack installed keystone, glance, nova, placement, cinder, neutron, and horizon. But DevStack doesn’t install heat, the orchestration service of Openstack which is required by KYPO CRP. So you have to configure DevStack to enable heat.
Unbun Server Installation on AWS Cloud
This is Ubuntu Server 20.04.3 LTS (HVM) with 4vCPU, 16 Go RAM and 55 Gb SSD Disk.
root# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.3 LTS
Release: 20.04
Codename: focal
DevStack installation
I followed this official tutorial but also this article. So let’s go step by step.
ubuntu$ sudo apt update
Fetched 20.6 MB in 4s (5862 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
31 packages can be upgraded. Run 'apt list --upgradable' to see them.
ubuntu$ sudo apt -y upgrade
Found linux image: /boot/vmlinuz-5.11.0-1021-aws
Found initrd image: /boot/microcode.cpio /boot/initrd.img-5.11.0-1021-aws
Found linux image: /boot/vmlinuz-5.11.0-1020-aws
Found initrd image: /boot/microcode.cpio /boot/initrd.img-5.11.0-1020-aws
Found Ubuntu 20.04.3 LTS (20.04) on /dev/xvda1
Done
ubuntu$ sudo apt -y dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
ubuntu$ sudo reboot
ubuntu$ sudo useradd -s /bin/bash -d /opt/stack -m stack
ubuntu$ echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack
ubuntu$ sudo su – stack
stack$ sudo su –
root$ su – stack
stack$ sudo apt -y install git
Reading package lists... Done
Building dependency tree
Reading state information... Done
git is already the newest version (1:2.25.1-1ubuntu3.2).
git set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
stack$ git clone https://git.openstack.org/openstack-dev/devstack
Cloning into 'devstack'...
warning: redirecting to https://opendev.org/openstack/devstack/
remote: Enumerating objects: 27621, done.
remote: Counting objects: 100% (27621/27621), done.
remote: Compressing objects: 100% (9258/9258), done.
remote: Total 47887 (delta 26959), reused 18363 (delta 18363), pack-reused 20266
Receiving objects: 100% (47887/47887), 10.19 MiB | 4.03 MiB/s, done.
Resolving deltas: 100% (33650/33650), done.
stack$ cd devstack
stack$ vi local.conf
Add:
[[local|localrc]]
# Password for KeyStone, Database, RabbitMQ and Service
ADMIN_PASSWORD=StrongAdminSecret
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD
Heat is configured by default on devstack for Icehouse and Juno releases. But as mentioned at the beginning, newer versions of OpenStack require enabling heat services in devstack local.conf. I followed this tutorial.
Add the following to [[local|localrc]] section of local.conf:
[[local|localrc]]
#Enable heat services
enable_service h-eng h-api h-api-cfn h-api-cw
Since Newton release, heat is available as a devstack plugin. To enable the plugin add the following to the [[local|localrc]] section of local.conf:
[[local|localrc]]
#Enable heat plugin
enable_plugin heat https://opendev.org/openstack/heat
I tried to add a stable branches by specifying the branch name to enable_plugin, but it doesn’t work for me so I didn’t add the following line.
enable_plugin heat https://opendev.org/openstack/heat stable/newton
It would also be useful to automatically download and register a VM image that heat can launch. To do that add the following to [[local|localrc]] section of local.conf:
IMAGE_URL_SITE="https://download.fedoraproject.org"
IMAGE_URL_PATH="/pub/fedora/linux/releases/33/Cloud/x86_64/images/"
IMAGE_URL_FILE="Fedora-Cloud-Base-33-1.2.x86_64.qcow2"
IMAGE_URLS+=","$IMAGE_URL_SITE$IMAGE_URL_PATH$IMAGE_URL_FILE
Disable the Ubuntu Firewall
stack$ sudo ufw disable
I then started the installation of Openstack.
stack$ ./stack.sh
This will take a 15 – 20 minutes, largely depending on the speed of the internet connection. At the end of the installation process, you should see output like this:
=========================
DevStack Component Timing
(times are in seconds)
=========================
wait_for_service 16
pip_install 232
apt-get 264
run_process 27
dbsync 15
git_timed 286
apt-get-update 1
test_with_retry 5
async_wait 72
osc 305
-------------------------
Unaccounted time 155
=========================
Total runtime 1378
=================
Async summary
=================
Time spent in the background minus waits: 367 sec
Elapsed time: 1378 sec
Time if we did everything serially: 1745 sec
Speedup: 1.26633
This is your host IP address: xxx.xxx.xxx.xxx
This is your host IPv6 address: ::1
Horizon is now available at http://xxx.xxx.xxx.xxx/dashboard
Keystone is serving at http://xxx.xxx.xxx.xxx/identity/
The default users are: admin and demo
The password: xxxxxxx
Services are running under systemd unit files.
For more information see:
https://docs.openstack.org/devstack/latest/systemd.html
DevStack Version: yoga
Change: f9a896c6e6afcf52e9a50613285940c26e353ba3 Rehome functions to enable Neutron's QoS service 2021-11-13 19:52:06 +0000
OS Version: Ubuntu 20.04 focal
2021-11-15 20:47:52.095 | stack.sh completed in 1378 seconds.
Copy the Horizon URL shown on the installation output and paste it into your web browser:
http://192.168.10.100/dashboard
Use the default users admin and configured password to login.
I have access to the Horizon web interface dashboard to manage vms, networks, volumes, and images.
Before you can start running client commands, OpenStack RC file must be downloaded from the Horizon dashboard and sourced in the current SHELL environment.
To download OpenStack RC file, log in to the Horizon dashboard. Check that you are in the good project (admin for me) and go to Project > API Access
On the API Access section, use the “Download OpenStack RC File” link to pull and save the « admin-openrc.sh » file on your desktop.
Copy the contents of the file on the server.
stack$ vi admin-openrc.sh
Source the file. As a security mechanism the file won’t contain the user password. You’ll be asked to set the password when sourcing the file.
source admin-openrc.sh
Please enter your OpenStack Password for project admin as user admin:
Test some OpenStack client commands just to confirm it is working. Check mainly that heat service is started.
stack$ openstack service list
+----------------------------------+-------------+----------------+
| ID | Name | Type |
+----------------------------------+-------------+----------------+
| 0b293dc58885450bad190bbfe3bacc40 | nova_legacy | compute_legacy |
| 1c05400514e341d09bd5a973136a9789 | cinderv3 | volumev3 |
| 3049ac1cc4a84b81a41d9fdb559ce922 | heat | orchestration |
| 775998becd0142579289a613a4313e1a | keystone | identity |
| 840023d4bc6f4e75a7fdb6e7d49ed28e | placement | placement |
| b9a2b39775a94d4f8a5fdfb25b9e4dc1 | neutron | network |
| c7b83375dafa428cbc21ceafb8611fbe | heat-cfn | cloudformation |
| e62bfc0c37774f8da910b3062df43d53 | cinder | block-storage |
| f2ffbf578599481295140dec77bcd549 | nova | compute |
| f619511aea824a59a76e66702de4e1c2 | glance | image |
+----------------------------------+-------------+----------------+
In order to avoid to set the password each time you source the RC file, you can optionally comment out the lines that prompts you the password and provide it statically:
stack$ vi admin-openrc.sh
# With Keystone you pass the keystone password.
#echo "Please enter your OpenStack Password for project $OS_PROJECT_NAME as user $OS_USERNAME: "
#read -sr OS_PASSWORD_INPUT
#export OS_PASSWORD=$OS_PASSWORD_INPUT
export OS_PASSWORD='xxxxxxxxxxx'
You can copy the RC file to keystonerc_admin
stack$ cp admin-openrc.sh keystonerc_admin
stack$ source keystonerc_admin
You can run some others OpenStack client commands to confirm that all is working properly:
stack$ openstack catalog list
+-------------+----------------+----------------------------------------------------------------------------+
| Name | Type | Endpoints |
+-------------+----------------+----------------------------------------------------------------------------+
| nova_legacy | compute_legacy | RegionOne |
| | | public: http://xxx.xxx.xxx.xxx/compute/v2/d81af43ddd074376a8e7fff88d61c905 |
| | | |
| cinderv3 | volumev3 | RegionOne |
| | | public: http://xxx.xxx.xxx.xxx/volume/v3/d81af43ddd074376a8e7fff88d61c905 |
| | | |
| heat | orchestration | RegionOne |
| | | public: http://xxx.xxx.xxx.xxx/heat-api/v1/d81af43ddd074376a8e7fff88d61c905 |
| | | |
| keystone | identity | RegionOne |
| | | public: http://xxx.xxx.xxx.xxx/identity |
| | | |
| placement | placement | RegionOne |
| | | public: http://xxx.xxx.xxx.xxx/placement |
| | | |
| neutron | network | RegionOne |
| | | public: http://xxx.xxx.xxx.xxx:9696/ |
| | | |
| heat-cfn | cloudformation | RegionOne |
| | | public: http://xxx.xxx.xxx.xxx/heat-api-cfn/v1 |
| | | |
| cinder | block-storage | RegionOne |
| | | public: http://xxx.xxx.xxx.xxx/volume/v3/d81af43ddd074376a8e7fff88d61c905 |
| | | |
| nova | compute | RegionOne |
| | | public: http://xxx.xxx.xxx.xxx/compute/v2.1 |
| | | |
| glance | image | RegionOne |
| | | public: http://xxx.xxx.xxx.xxx/image |
| | | |
+-------------+----------------+----------------------------------------------------------------------------+
I checked the endpoint list to find public endpoint for orchestration service (heat).
stack$ openstack endpoint list
+----------------------------------+-----------+--------------+----------------+---------+-----------+------------------------------------------------+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+----------------------------------+-----------+--------------+----------------+---------+-----------+------------------------------------------------+
| 064822424bfe4c4394951dce1832e316 | RegionOne | cinder | block-storage | True | public | http://xxx.xxx.xxx.xxx/volume/v3/$(project_id)s |
| 11fbdcab6dfe42cb82c3ac4c3f61296a | RegionOne | nova | compute | True | public | http://xxx.xxx.xxx.xxx/compute/v2.1 |
| 2cb9561aa98a4c079d0c7f35ba347647 | RegionOne | keystone | identity | True | public | http://xxx.xxx.xxx.xxx/identity |
| 52bcd8dde6fb4f7b82d976cf71a0d37e | RegionOne | cinderv3 | volumev3 | True | public | http://xxx.xxx.xxx.xxx/volume/v3/$(project_id)s |
| 82d29e1ceb464b7f831b84434ebb0be3 | RegionOne | glance | image | True | public | http://xxx.xxx.xxx.xxx/image |
| 98037666c0e74127ab713bd4865b062d | RegionOne | neutron | network | True | public | http://xxx.xxx.xxx.xxx:9696/ |
| 9bda08ed79fe4fc399f94f6274ceaca0 | RegionOne | placement | placement | True | public | http://xxx.xxx.xxx.xxx/placement |
| cc35f42f35304534b83301f4fc70e778 | RegionOne | nova_legacy | compute_legacy | True | public | http://xxx.xxx.xxx.xxx/compute/v2/$(project_id)s |
+----------------------------------+-----------+--------------+----------------+---------+-----------+------------------------------------------------+
stack$ openstack orchestration service list
+-----------------+-------------+--------------------------------------+-----------------+--------+----------------------------+--------+
| Hostname | Binary | Engine ID | Host | Topic | Updated At | Status |
+-----------------+-------------+--------------------------------------+-----------------+--------+----------------------------+--------+
| xxxxxxxxxxxxxxx | heat-engine | 5ffc9a0d-2756-462a-8da1-b5f2aeca165b | xxxxxxxxxxxxxxx | engine | 2021-11-15T21:40:55.000000 | up |
| xxxxxxxxxxxxxxx | heat-engine | 54387f8e-44d7-4749-853a-c06d6be92ace | xxxxxxxxxxxxxxx | engine | 2021-11-15T21:40:55.000000 | up |
+-----------------+-------------+--------------------------------------+-----------------+--------+----------------------------+--------+
KYPO CRP installation
OpenStack Requirements
Check before the OpenStack Requirements here
root# openstack flavor create --ram 2048 --disk 20 --vcpus 1 csirtmu.tiny1x2
+----------------------------+--------------------------------------+
| Field | Value |
+----------------------------+--------------------------------------+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| description | None |
| disk | 20 |
| id | 69fb4a25-d8f3-4a5b-afbd-8a4823210733 |
| name | csirtmu.tiny1x2 |
| os-flavor-access:is_public | True |
| properties | |
| ram | 2048 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 1 |
+----------------------------+--------------------------------------+
stack$ wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img -P /tmp/
Resolving cloud-images.ubuntu.com (cloud-images.ubuntu.com)... 91.189.88.248, 91.189.88.247, 2001:67c:1360:8001::33, ...
Connecting to cloud-images.ubuntu.com (cloud-images.ubuntu.com)|91.189.88.248|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 568131584 (542M) [application/octet-stream]
Saving to: ‘/tmp/focal-server-cloudimg-amd64.img.1’
focal-server-cloudimg-amd64.img.1 100%[===========================================================================>] 541.81M 87.5MB/s in 6.2s
2021-11-15 21:50:30 (88.1 MB/s) - ‘/tmp/focal-server-cloudimg-amd64.img.1’ saved [568131584/568131584]
stack$ openstack image create --disk-format qcow2 --container-format bare --public --property \
> os_type=linux --file /tmp/focal-server-cloudimg-amd64.img ubuntu-focal-x86_64
+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| container_format | bare |
| created_at | 2021-11-15T21:49:14Z |
| disk_format | qcow2 |
| file | /v2/images/f748c173-c9d7-4ded-92c2-d84d9d6bcd82/file |
| id | f748c173-c9d7-4ded-92c2-d84d9d6bcd82 |
| min_disk | 0 |
| min_ram | 0 |
| name | ubuntu-focal-x86_64 |
| owner | d81af43ddd074376a8e7fff88d61c905 |
| properties | os_hidden='False', os_type='linux', owner_specified.openstack.md5='', owner_specified.openstack.object='images/ubuntu-focal-x86_64', owner_specified.openstack.sha256='' |
| protected | False |
| schema | /v2/schemas/image |
| status | queued |
| tags | |
| updated_at | 2021-11-15T21:49:14Z |
| visibility | public |
+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Base Infrastructure
I then followed this tutorial that describes how to prepare the KYPO base infrastructure used by the KYPO Cyber Range Platform.
stack$ sudo apt install python3-pip openssh-client jq
Reading package lists... Done
Building dependency tree
Reading state information... Done
openssh-client is already the newest version (1:8.2p1-4ubuntu0.3).
openssh-client set to manually installed.
python3-pip is already the newest version (20.0.2-5ubuntu1.6).
The following NEW packages will be installed:
jq libjq1 libonig5
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 313 kB of archives.
After this operation, 1062 kB of additional disk space will be used.
Get:1 http://eu-west-3.ec2.archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB]
Get:2 http://eu-west-3.ec2.archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB]
Get:3 http://eu-west-3.ec2.archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB]
Fetched 313 kB in 0s (2145 kB/s)
Selecting previously unselected package libonig5:amd64.
(Reading database ... 140023 files and directories currently installed.)
Preparing to unpack .../libonig5_6.9.4-1_amd64.deb ...
Unpacking libonig5:amd64 (6.9.4-1) ...
Selecting previously unselected package libjq1:amd64.
Preparing to unpack .../libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ...
Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ...
Selecting previously unselected package jq.
Preparing to unpack .../jq_1.6-1ubuntu0.20.04.1_amd64.deb ...
Unpacking jq (1.6-1ubuntu0.20.04.1) ...
Setting up libonig5:amd64 (6.9.4-1) ...
Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ...
Setting up jq (1.6-1ubuntu0.20.04.1) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
stack$ sudo pip3 install pipenv
/usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
from cryptography.utils import int_from_bytes
/usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
from cryptography.utils import int_from_bytes
Collecting pipenv
Downloading pipenv-2021.11.15-py2.py3-none-any.whl (3.6 MB)
|████████████████████████████████| 3.6 MB 25.8 MB/s
Requirement already satisfied: virtualenv in /usr/local/lib/python3.8/dist-packages (from pipenv) (20.10.0)
Requirement already satisfied: setuptools>=36.2.1 in /usr/local/lib/python3.8/dist-packages (from pipenv) (59.1.0)
Collecting virtualenv-clone>=0.2.5
Downloading virtualenv_clone-0.5.7-py3-none-any.whl (6.6 kB)
Requirement already satisfied: certifi in /usr/lib/python3/dist-packages (from pipenv) (2019.11.28)
Requirement already satisfied: pip>=18.0 in /usr/local/lib/python3.8/dist-packages (from pipenv) (21.3.1)
Requirement already satisfied: filelock<4,>=3.2 in /usr/local/lib/python3.8/dist-packages (from virtualenv->pipenv) (3.3.2)
Requirement already satisfied: platformdirs<3,>=2 in /usr/local/lib/python3.8/dist-packages (from virtualenv->pipenv) (2.4.0)
Requirement already satisfied: distlib<1,>=0.3.1 in /usr/local/lib/python3.8/dist-packages (from virtualenv->pipenv) (0.3.3)
Requirement already satisfied: six<2,>=1.9.0 in /usr/local/lib/python3.8/dist-packages (from virtualenv->pipenv) (1.16.0)
Requirement already satisfied: backports.entry-points-selectable>=1.0.4 in /usr/local/lib/python3.8/dist-packages (from virtualenv->pipenv) (1.1.1)
Installing collected packages: virtualenv-clone, pipenv
Successfully installed pipenv-2021.11.15 virtualenv-clone-0.5.7
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Create application credentials with this video. Be sure to generate Application Credentials with the parameter unrestricted.
After application credentials created, download on your desktop the « app-cred-kypo-openrc.sh » file from the Horizon dashboard and copy/paste the contents of the file on the server.
stack$ vi app-cred-kypo-openrc.sh
Source the file
stack$ source app-cred-kypo-openrc.sh
stack$ git clone https://gitlab.ics.muni.cz/muni-kypo-crp/devops/kypo-crp-openstack-base.git
Cloning into 'kypo-crp-openstack-base'...
remote: Enumerating objects: 269, done.
remote: Counting objects: 100% (138/138), done.
remote: Compressing objects: 100% (80/80), done.
remote: Total 269 (delta 54), reused 129 (delta 47), pack-reused 131
Receiving objects: 100% (269/269), 78.56 KiB | 1.31 MiB/s, done.
Resolving deltas: 100% (111/111), done.
stack$ cd kypo-crp-openstack-base
stack$ pipenv install
Creating a virtualenv for this project...
Pipfile: /opt/stack/devstack/kypo-crp-openstack-base/Pipfile
Using /usr/bin/python3.8 (3.8.10) to create virtualenv...
⠴ Creating virtual environment...created virtual environment CPython3.8.10.final.0-64 in 235ms
creator CPython3Posix(dest=/opt/stack/.local/share/virtualenvs/kypo-crp-openstack-base-5QbM23-5, clear=False, no_vcs_ignore=False, global =False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/opt/stack/.local/share/virtualenv )
added seed packages: pip==21.3.1, setuptools==58.3.0, wheel==0.37.0
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
✔ Successfully created virtual environment!
Virtualenv location: /opt/stack/.local/share/virtualenvs/kypo-crp-openstack-base-5QbM23-5
Installing dependencies from Pipfile.lock (5ccba9)...
🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 61/61 — 00:01:39
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
stack$ pipenv shell
Creating a virtualenv for this project...
Pipfile: /opt/stack/Pipfile
Using /usr/bin/python3 (3.8.10) to create virtualenv...
⠋ Creating virtual environment...created virtual environment CPython3.8.10.final.0-64 in 619ms
creator CPython3Posix(dest=/opt/stack/.local/share/virtualenvs/stack-mJieuOd4, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/opt/stack/.local/share/virtualenv)
added seed packages: pip==21.3.1, setuptools==58.3.0, wheel==0.37.0
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
✔ Successfully created virtual environment!
Virtualenv location: /opt/stack/.local/share/virtualenvs/stack-mJieuOd4
Creating a Pipfile for this project...
Launching subshell in virtual environment...
stack$ . /opt/stack/.local/share/virtualenvs/stack-mJieuOd4/bin/activate
((kypo-crp-openstack-base) ) stack$ pipenv sync
Installing dependencies from Pipfile.lock (5ccba9)...
🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00
All dependencies are now up-to-date!
Before to go to the deployment, you need to obtain several configuration values that might be specific to your OpenStack instance.
stack$ openstack network list --external --column Name
+--------+
| Name |
+--------+
| public |
+--------+
stack$ openstack image list --column Name
+---------------------------------+
| Name |
+---------------------------------+
| Fedora-Cloud-Base-33-1.2.x86_64 |
| cirros-0.5.2-x86_64-disk |
| ubuntu-focal-x86_64 |
+---------------------------------+
stack$ openstack flavor list --column Name
+-----------+
| Name |
+-----------+
| m1.tiny |
| m1.small |
| m1.medium |
| m1.large |
| m1.nano |
| m1.xlarge |
| m1.micro |
| cirros256 |
| ds512M |
| ds1G |
| ds2G |
| ds4G |
+-----------+
Un-Source all variables from previous source command
stack$ unset "${!OS_@}"
Below is the default openstack-defaults.sh file.
stack$ cat openstack-defaults.sh
#!/usr/bin/env bash
export KYPO_HEAD_FLAVOR="standard.large"
export KYPO_HEAD_IMAGE="ubuntu-focal-x86_64"
export KYPO_HEAD_USER="ubuntu"
export KYPO_PROXY_FLAVOR="standard.medium"
export KYPO_PROXY_IMAGE="ubuntu-focal-x86_64"
export KYPO_PROXY_USER="ubuntu"
export DNS1="1.1.1.1"
export DNS2="1.0.0.1"
Modify this file and edit the desired values for images (<kypo_base_image>) and flavors (<kypo_base_flavor>). On my side, I remove « standard.large » flavor and « standard.medium » flavor. I replaced them with « m1.small » beause my config is not very strong.
stack$ cat openstack-defaults.sh
#!/usr/bin/env bash
export KYPO_HEAD_FLAVOR="m1.small"
export KYPO_HEAD_IMAGE="ubuntu-focal-x86_64"
export KYPO_HEAD_USER="ubuntu"
export KYPO_PROXY_FLAVOR="m1.small"
export KYPO_PROXY_IMAGE="ubuntu-focal-x86_64"
export KYPO_PROXY_USER="ubuntu"
export DNS1="1.1.1.1"
export DNS2="1.0.0.1"
stack$ source openstack-defaults.sh
I checked default Security Group Rules. I’m not sure if I have to delete them or not. The tutorial is not clear about this point.
stack$ openstack security group rule list default
+--------------------------------------+-------------+-----------+-----------+------------+--------------------------------------+
| ID | IP Protocol | Ethertype | IP Range | Port Range | Remote Security Group |
+--------------------------------------+-------------+-----------+-----------+------------+--------------------------------------+
| 38b8c48a-494c-49bd-bee3-51e2c415f30b | None | IPv4 | 0.0.0.0/0 | | d18b863a-0b8d-4110-878a-de40e24307a5 |
| 7e5ae97e-cb42-4024-aeef-ed631a2b567c | None | IPv6 | ::/0 | | None |
| d8808eb1-1edc-4e3f-8ddb-7061927fe9a3 | None | IPv6 | ::/0 | | d18b863a-0b8d-4110-878a-de40e24307a5 |
| de4d2cce-2fee-4e01-b966-5f7420c5d484 | None | IPv4 | 0.0.0.0/0 | | None |
+--------------------------------------+-------------+-----------+-----------+------------+--------------------------------------+
I then bootstraped Floating IPs and Keypair. The results will be saved into kypo-base-params.yml file. Private key of the keypair will be saved into <openstack-project>_kypo-base-key.key
stack$ ./bootstrap.sh public
Floating IP kypo-base-head for network public does not exist. Creating...
Floating IP kypo-base-proxy for network public does not exist. Creating...
No keypair with a name or ID of 'admin_kypo-base-key' exists.
Creating keypair admin_kypo-base-key.
fingerprint: 86:8f:ea:34:dc:4b:bc:77:a8:6d:d5:7b:42:3c:a4:e4
name: admin_kypo-base-key
user_id: 042e20a21d0f4cf2a8473daf72ca2193
Private key for user access does not exist. Creating...
Generating RSA private key, 2048 bit long modulus (2 primes)
......................+++++
..+++++
e is 65537 (0x010001)
stack$ ./create-base.sh
2021-11-15 23:57:31Z [kypo-base-networking-stack]: CREATE_IN_PROGRESS Stack CREATE started
2021-11-15 23:57:31Z [kypo-base-networking-stack.kypo-base-net]: CREATE_IN_PROGRESS state changed
2021-11-15 23:57:31Z [kypo-base-networking-stack.kypo-base-net]: CREATE_COMPLETE state changed
2021-11-15 23:57:31Z [kypo-base-networking-stack.kypo-base-subnet]: CREATE_IN_PROGRESS state changed
2021-11-15 23:57:32Z [kypo-base-networking-stack.kypo-base-router-public]: CREATE_IN_PROGRESS state changed
2021-11-15 23:57:32Z [kypo-base-networking-stack.kypo-base-subnet]: CREATE_COMPLETE state changed
2021-11-15 23:57:32Z [kypo-base-networking-stack.kypo-base-router-public-port]: CREATE_IN_PROGRESS state changed
2021-11-15 23:57:33Z [kypo-base-networking-stack.kypo-base-router-public-port]: CREATE_COMPLETE state changed
2021-11-15 23:57:34Z [kypo-base-networking-stack.kypo-base-router-public]: CREATE_COMPLETE state changed
2021-11-15 23:57:34Z [kypo-base-networking-stack.kypo-base-router-public-interface]: CREATE_IN_PROGRESS state changed
2021-11-15 23:57:36Z [kypo-base-networking-stack.kypo-base-router-public-interface]: CREATE_COMPLETE state changed
2021-11-15 23:57:36Z [kypo-base-networking-stack]: CREATE_COMPLETE Stack CREATE completed successfully
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| id | cb20a1c9-da98-4699-a14e-09b4d2ee78a4 |
| stack_name | kypo-base-networking-stack |
| description | KYPO base networking. |
| creation_time | 2021-11-15T23:57:30Z |
| updated_time | None |
| stack_status | CREATE_COMPLETE |
| stack_status_reason | Stack CREATE completed successfully |
+---------------------+--------------------------------------+
2021-11-15 23:57:42Z [kypo-base-security-groups-stack]: CREATE_IN_PROGRESS Stack CREATE started
2021-11-15 23:57:43Z [kypo-base-security-groups-stack.kypo-base-head-sg]: CREATE_IN_PROGRESS state changed
2021-11-15 23:57:43Z [kypo-base-security-groups-stack.kypo-base-head-sg]: CREATE_COMPLETE state changed
2021-11-15 23:57:44Z [kypo-base-security-groups-stack.kypo-global-ingress-icmp]: CREATE_IN_PROGRESS state changed
2021-11-15 23:57:44Z [kypo-base-security-groups-stack.kypo-global-ingress-icmp]: CREATE_COMPLETE state changed
2021-11-15 23:57:45Z [kypo-base-security-groups-stack.kypo-base-proxy-sg]: CREATE_IN_PROGRESS state changed
2021-11-15 23:57:45Z [kypo-base-security-groups-stack.kypo-base-proxy-sg]: CREATE_COMPLETE state changed
2021-11-15 23:57:45Z [kypo-base-security-groups-stack.kypo-global-remote-security-groups]: CREATE_IN_PROGRESS state changed
2021-11-15 23:57:57Z [kypo-base-security-groups-stack.kypo-global-remote-security-groups]: CREATE_COMPLETE state changed
2021-11-15 23:57:57Z [kypo-base-security-groups-stack]: CREATE_COMPLETE Stack CREATE completed successfully
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| id | 0b52e47b-45d8-47cb-aa06-c204feedb038 |
| stack_name | kypo-base-security-groups-stack |
| description | KYPO base security groups. |
| creation_time | 2021-11-15T23:57:42Z |
| updated_time | None |
| stack_status | CREATE_COMPLETE |
| stack_status_reason | Stack CREATE completed successfully |
+---------------------+--------------------------------------+
2021-11-15 23:58:00Z [kypo-head-stack]: CREATE_IN_PROGRESS Stack CREATE started
2021-11-15 23:58:00Z [kypo-head-stack.kypo-head-port]: CREATE_IN_PROGRESS state changed
2021-11-15 23:58:01Z [kypo-head-stack.kypo-head-port]: CREATE_COMPLETE state changed
2021-11-15 23:58:01Z [kypo-head-stack.kypo-head-floating-ip]: CREATE_IN_PROGRESS state changed
2021-11-15 23:58:01Z [kypo-head-stack.kypo-head]: CREATE_IN_PROGRESS state changed
2021-11-15 23:58:01Z [kypo-head-stack.kypo-head-floating-ip]: CREATE_COMPLETE state changed
2021-11-15 23:58:06Z [kypo-head-stack.kypo-head]: CREATE_COMPLETE state changed
2021-11-15 23:58:06Z [kypo-head-stack]: CREATE_COMPLETE Stack CREATE completed successfully
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| id | 043f10e3-a13d-4e87-9732-e85e13eb6e6c |
| stack_name | kypo-head-stack |
| description | KYPO Head server. |
| creation_time | 2021-11-15T23:58:00Z |
| updated_time | None |
| stack_status | CREATE_COMPLETE |
| stack_status_reason | Stack CREATE completed successfully |
+---------------------+--------------------------------------+
2021-11-15 23:58:12Z [kypo-proxy-jump-stack]: CREATE_IN_PROGRESS Stack CREATE started
2021-11-15 23:58:12Z [kypo-proxy-jump-stack.kypo-proxy-jump-port]: CREATE_IN_PROGRESS state changed
2021-11-15 23:58:13Z [kypo-proxy-jump-stack.kypo-proxy-jump-port]: CREATE_COMPLETE state changed
2021-11-15 23:58:13Z [kypo-proxy-jump-stack.kypo-proxy-jump-floating-ip]: CREATE_IN_PROGRESS state changed
2021-11-15 23:58:13Z [kypo-proxy-jump-stack.kypo-proxy-jump]: CREATE_IN_PROGRESS state changed
2021-11-15 23:58:14Z [kypo-proxy-jump-stack.kypo-proxy-jump-floating-ip]: CREATE_COMPLETE state changed
2021-11-15 23:58:19Z [kypo-proxy-jump-stack.kypo-proxy-jump]: CREATE_COMPLETE state changed
2021-11-15 23:58:19Z [kypo-proxy-jump-stack]: CREATE_COMPLETE Stack CREATE completed successfully
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| id | 479af43a-a6f8-4413-8291-5ce03eb56a4b |
| stack_name | kypo-proxy-jump-stack |
| description | KYPO Proxy Jump server. |
| creation_time | 2021-11-15T23:58:12Z |
| updated_time | None |
| stack_status | CREATE_COMPLETE |
| stack_status_reason | Stack CREATE completed successfully |
+---------------------+--------------------------------------+
I checked the stack list
stack$ openstack stack list
+--------------------------------------+---------------------------------+-----------------+----------------------+--------------+
| ID | Stack Name | Stack Status | Creation Time | Updated Time |
+--------------------------------------+---------------------------------+-----------------+----------------------+--------------+
| 479af43a-a6f8-4413-8291-5ce03eb56a4b | kypo-proxy-jump-stack | CREATE_COMPLETE | 2021-11-15T23:58:12Z | None |
| 043f10e3-a13d-4e87-9732-e85e13eb6e6c | kypo-head-stack | CREATE_COMPLETE | 2021-11-15T23:58:00Z | None |
| 0b52e47b-45d8-47cb-aa06-c204feedb038 | kypo-base-security-groups-stack | CREATE_COMPLETE | 2021-11-15T23:57:42Z | None |
| cb20a1c9-da98-4699-a14e-09b4d2ee78a4 | kypo-base-networking-stack | CREATE_COMPLETE | 2021-11-15T23:57:30Z | None |
+--------------------------------------+---------------------------------+-----------------+----------------------+--------------+
I checked all the installation on GUI Horizon dashboard
I launched Ansible scripts to test connectivity. PING and SSH are OK.
((kypo-crp-openstack-base) ) root@xxxxxxxx:~/kypo-crp-openstack-base# ./ansible-check-base.sh
PLAY [Check Base Stack] *********************************************************************************************************************************************************************************************************************
TASK [ping : Wait for ssh connection] *******************************************************************************************************************************************************************************************************
ok: [kypo-base-head]
ok: [kypo-base-proxy]
TASK [Try to reach the machine via ping] ****************************************************************************************************************************************************************************************************
ok: [kypo-base-head]
ok: [kypo-base-proxy]
PLAY RECAP **********************************************************************************************************************************************************************************************************************************
kypo-base-head : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
kypo-base-proxy : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
((kypo-crp-openstack-base) ) root@xxxx:~/kypo-crp-openstack-base# ./ansible-user-access.sh
PLAY [Create Access for KYPO User] **********************************************************************************************************************************************************************************************************
TASK [Gathering Facts] **********************************************************************************************************************************************************************************************************************
ok: [kypo-base-proxy]
TASK [user : Ensure group kypo] *************************************************************************************************************************************************************************************************************
changed: [kypo-base-proxy]
TASK [Ensure user kypo] *********************************************************************************************************************************************************************************************************************
changed: [kypo-base-proxy]
TASK [Set authorized key for kypo user] *****************************************************************************************************************************************************************************************************
changed: [kypo-base-proxy]
TASK [Add kypo user to sudoers] *************************************************************************************************************************************************************************************************************
changed: [kypo-base-proxy]
PLAY RECAP **********************************************************************************************************************************************************************************************************************************
kypo-base-proxy : ok=5 changed=4 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
All seem to be OK.
Now I have a problem. I rebooted my OpenStack server and I lost connectivity between Openstask host and instances VM. Ping and SSH was NOK.
After several researches, I found that a Devstack environment is not persistent across server reboots.
DevStack provides a set of scripts for automated installation of OpenStack on Ubuntu as well as Fedora Linux. It is a tool to help OpenStack developers to quickly set up an OpenStack environment using scripts. These scripts automatically download or clone the required packages and repositories from the OpenStack website that are necessary for setting up an OpenStack cloud. One drawback with this approach is that the environment is not persistent across server reboots.
Newer versions of DevStack runs it’s services as systemd unit files so, you can use systemctl to manage them. I checked Openstask services. All seem to be OK
((kypo-crp-openstack-base) ) root@ip-172-31-6-66:~/kypo-crp-openstack-base# sudo systemctl list-units devstack@*
UNIT LOAD ACTIVE SUB DESCRIPTION
devstack@c-api.service loaded active running Devstack devstack@c-api.service
devstack@c-sch.service loaded active running Devstack devstack@c-sch.service
devstack@c-vol.service loaded active running Devstack devstack@c-vol.service
devstack@dstat.service loaded active running Devstack devstack@dstat.service
devstack@etcd.service loaded active running Devstack devstack@etcd.service
devstack@g-api.service loaded active running Devstack devstack@g-api.service
devstack@h-api-cfn.service loaded active running Devstack devstack@h-api-cfn.service
devstack@h-api.service loaded active running Devstack devstack@h-api.service
devstack@h-eng.service loaded active running Devstack devstack@h-eng.service
devstack@keystone.service loaded active running Devstack devstack@keystone.service
devstack@n-api-meta.service loaded active running Devstack devstack@n-api-meta.service
devstack@n-api.service loaded active running Devstack devstack@n-api.service
devstack@n-cond-cell1.service loaded active running Devstack devstack@n-cond-cell1.service
devstack@n-cpu.service loaded active running Devstack devstack@n-cpu.service
devstack@n-novnc-cell1.service loaded active running Devstack devstack@n-novnc-cell1.service
devstack@n-sch.service loaded active running Devstack devstack@n-sch.service
devstack@n-super-cond.service loaded active running Devstack devstack@n-super-cond.service
devstack@placement-api.service loaded active running Devstack devstack@placement-api.service
devstack@q-ovn-metadata-agent.service loaded active running Devstack devstack@q-ovn-metadata-agent.service
devstack@q-svc.service loaded active running Devstack devstack@q-svc.service
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
20 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
I also restarted all services without change
sudo systemctl restart devstack@*
I checked the status of Open Vswitch
((kypo-crp-openstack-base) ) root@ip-172-31-6-66:~/kypo-crp-openstack-base# ovs-vsctl show
95bd0929-30f8-42d8-9a66-699036952e8c
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge br-ex
Port br-ex
Interface br-ex
type: internal
Port patch-provnet-b606a04f-2955-4f7d-807b-3677bb3cb4e3-to-br-int
Interface patch-provnet-b606a04f-2955-4f7d-807b-3677bb3cb4e3-to-br-int
type: patch
options: {peer=patch-br-int-to-provnet-b606a04f-2955-4f7d-807b-3677bb3cb4e3}
Bridge br-int
fail_mode: secure
datapath_type: system
Port br-int
Interface br-int
type: internal
Port tapf147ff8d-9b
Interface tapf147ff8d-9b
Port tapc6517e8f-b0
Interface tapc6517e8f-b0
Port tap7e86de1c-70
Interface tap7e86de1c-70
Port patch-br-int-to-provnet-b606a04f-2955-4f7d-807b-3677bb3cb4e3
Interface patch-br-int-to-provnet-b606a04f-2955-4f7d-807b-3677bb3cb4e3
type: patch
options: {peer=patch-provnet-b606a04f-2955-4f7d-807b-3677bb3cb4e3-to-br-int}
ovs_version: "2.13.3"
((kypo-crp-openstack-base) ) root@ip-172-31-6-66:~/kypo-crp-openstack-base# ovs-vsctl list-ports br-ex
patch-provnet-b606a04f-2955-4f7d-807b-3677bb3cb4e3-to-br-int
((kypo-crp-openstack-base) ) root@ip-172-31-6-66:~/kypo-crp-openstack-base# ovs-vsctl list-ports br-int
patch-br-int-to-provnet-b606a04f-2955-4f7d-807b-3677bb3cb4e3
tap7e86de1c-70
tapc6517e8f-b0
tapf147ff8d-9b
I found some error but I don’t if it’s important
((kypo-crp-openstack-base) ) root@ip-172-31-6-66:~/kypo-crp-openstack-base# ovs-ofctl dump-ports br-ex
2021-11-29T18:15:59Z|00001|vconn|WARN|unix:/var/run/openvswitch/br-ex.mgmt: version negotiation failed (we support version 0x01, peer supports versions 0x04, 0x06)
ovs-ofctl: br-ex: failed to connect to socket (Broken pipe)
((kypo-crp-openstack-base) ) root@ip-172-31-6-66:~/kypo-crp-openstack-base# ovs-dpctl show -s
system@ovs-system:
lookups: hit:8565 missed:364 lost:0
flows: 4
masks: hit:14131 total:2 hit/pkt:1.58
port 0: ovs-system (internal)
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 aborted:0 carrier:0
collisions:0
RX bytes:0 TX bytes:0
port 1: br-ex (internal)
RX packets:0 errors:0 dropped:4285 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 aborted:0 carrier:0
collisions:0
RX bytes:0 TX bytes:0
port 2: br-int (internal)
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 aborted:0 carrier:0
collisions:0
RX bytes:0 TX bytes:0
port 3: tap7e86de1c-70
RX packets:407 errors:0 dropped:0 overruns:0 frame:0
TX packets:38 errors:0 dropped:0 aborted:0 carrier:0
collisions:0
RX bytes:30048 (29.3 KiB) TX bytes:2668 (2.6 KiB)
port 4: tapc6517e8f-b0
RX packets:4 errors:0 dropped:0 overruns:0 frame:0
TX packets:16 errors:0 dropped:0 aborted:0 carrier:0
collisions:0
RX bytes:360 TX bytes:1216 (1.2 KiB)
port 5: tapf147ff8d-9b
RX packets:1592 errors:0 dropped:0 overruns:0 frame:0
TX packets:119 errors:0 dropped:0 aborted:0 carrier:0
collisions:0
RX bytes:115182 (112.5 KiB) TX bytes:6126 (6.0 KiB)
For the moment, I stopped the procedure at this level. I come back quickly. If you have some idea to help me to find connectivity between the OpenStack Host and instances VM, you are welcome !
Ressources
Website: https://www.kypo.cz/
Documentation: https://docs.crp.kypo.muni.cz/
Gitlab: https://gitlab.ics.muni.cz/muni-kypo-crp
Twitter: https://twitter.com/KYPOCRP
