The on-premises servers that host the Ayyeka solution require monitoring and ongoing maintenance to ensure reliability and availability of the services.
Warning: Failure to adequately maintain the server might result in downtime, lost data, and poor performance.
Contents
Server Architecture Overview
The server software is built of micro-services, each of which is in charge of a set of features and capabilities (for example, user interface, data processing, command processing, etc.). The micro-services are deployed and executed using Docker containers, a lightweight virtualization technology that improves software stability, scalability, and security.
The micro-services interact with each other to perform tasks and handle requests from users and devices. In addition, they use the following common infrastructure:
- MySQL Database for storing the data collected by the devices, as well as all the metadata required to process the data, manage users and devices, and in general, maintain the application's state.
- Redis™ for caching frequently accessed data in-memory and system-wide synchronization
- Rabbit MQ™ for communication between micro-services and also for receiving data from the devices
High availability of services
The on-premises software package is designed to run on a single server. If high availability is a requirement, appropriate architecture should be planned, based on the customer's requirements. The Support team is available to discuss and assist with planning, in collaboration with the system integrator and the customer’s IT staff.
Recommended solutions might include:
- Redundant internet connections
- Redundant servers
- High availability setup for MySQL, RabbitMQ and Redis
Maintenance Guidelines
Ayyeka recommends the following monitoring, maintenance, and resilience activities:
Monitoring of server and services
- Regular monitoring and cleaning of disk space to accommodate the data that is collected in the database. Disk usage depends on the number of devices, the amount of data collected per device, and other factors.
- There must always be sufficient free disk space available to allow for growth, backups, and software upgrades. The minimum amount depends on your specific setup, but at least 10GB should be available at all times.
- Disk space availability is especially critical during software upgrades.
- Disk usage needs to be monitored periodically to gauge the rate in which free space is consumed.
- Regular preventive maintenance and health checks on CPU and memory.
- Regular health checks on hypervisor when hosted as a virtual machine (VM).
- The following critical services should be monitored and running on their standard ports on the base Ubuntu operating system:
- MySQL
- RabbitMQ
- Redis
- DOCKER™
- Docker containers must be monitored to ensure they are running at all times. Containers that are being restarted frequently might indicate an ongoing or evolving issue with one of the microservices.
Ongoing maintenance of the server
- Regular operating system and security updates.
- Regular updates of the Ayyeka software on the on-premise server. Note that major updates are available about every two months, and minor updates are available approximately bi-weekly.
Caution: Installing updates might overwrite any changes that you made to configuration files. Therefore, it is highly recommended that you back up your system before doing any updates.
- Backup of the server that can be used to restore the server in the event of a disaster. Backups should be performed regularly and stored offline in a safe location. It is recommended to test backups at least quarterly.
Disk Space Monitoring and Planning
It is recommended to monitor the following directories regularly.
|
Directory Name |
Description |
| /var/ayyeka/ak-dumps |
What these files are: Backup of raw files received over HTTP Growth: It can grow over time, but a default retention period of 30 days is enforced. What you need to do: Make sure to have enough disk space to hold files for this retention period.
|
| /var/ayyeka/proto-dumps |
What these files are: Backup of raw files received over MQTT Growth: It can grow over time, but a default retention period of 30 days is enforced. What you need to do: Make sure to have enough disk space to hold files for this retention period.
|
| /var/ayyeka/backup |
What these files are: Backup of the database and configuration files Growth: Might grow over time, but a retention period of 7 days is enforced. What you need to do:
|
| /var/lib/mysql |
What these files are: DB storage Growth: Will grow as the number of devices in the system increases and as data is collected What you need to do: Make sure to have enough disk space and add more storage space as required. |
| /var/lib/rabbitmq |
What these files are: RabbitMQ storage Growth: Should not grow. What you need to do: Make sure it does not steadily increase. |
| /var/lib/docker |
What these files are: Docker container images and logs Growth: Should not grow regularly, might grow after software upgrades. What you need to do: Make sure it does not steadily increase. If it does, use the following commands:
docker inspect --format='{{.LogPath}}' $(docker ps -a -q) | sudo xargs -n 1 du -h
docker inspect --format='{{.LogPath}}' $(docker ps -a -q) | sudo xargs -n 1 truncate -s 0
docker image prune -a |
| /var/log/ayyeka |
What these files are: Application server log files Growth: Grows as a function of the server's activity. A retention period of 7 days is enforced. What you need to do: Make sure it does not steadily increase. Fast growth of logs and error logs might indicate a problem. |
| /var/lib/redis |
What these files are: Redis data Growth: Should not grow. What you need to do: Make sure it does not steadily increase. |
Troubleshooting
You can get information about the Docker containers by using the following CLI commands:
|
Information you need |
Command |
|---|---|
| Getting the size of a directory You can find an example of getting the size of DB in the command section. It is recommended to monitor the size of all directories that were mentioned above regularly. |
sudo du -sh /var/lib/mysql |
| List all running containers |
dps |
|
Show live resource usage statistics of all containers |
dstats |
|
List all images that are locally stored with the docker engine |
docker image ls |
| Stop a docker container |
dstop |
| Start a docker container |
dstart [container name as it appears in the dps command] |
| Restart a docker container |
drestart [container name as it appears in the dps command] |