As development teams continue to embrace microservices and distributed systems, observability is becoming increasingly important for service management, troubleshooting and monitoring production environments. With the rise of cloud computing, the cost of monitoring has never been lower. Yet observability solutions tend to be expensive and can be difficult to set up. OpenSearch offers an open-source alternative that promises to make observability easier than ever.
What is observability?
Observability is about how your application works from a technical perspective. It encompasses the tools and techniques that allow you to gain visibility into your systems, understand their behaviour and identify anomalies.
On one hand, observability is a new term for something that has been around for a long time. Logging has existed since computer systems have existed. But as technology has evolved from monolithic applications on physical hardware to distributed systems deployed as code on virtualized infrastructure, new challenges have emerged. Troubleshooting and debugging these modern systems require purpose-built tools.
Observability rests on three fundamental pillars: logs, metrics and traces. Logs tell you what happened. Metrics give you a quantitative view of your systems' state. Traces let you follow a request's journey through the different services in your architecture. It's the combination of these three elements that provides a complete picture of your infrastructure's health.
At Hidora, observability is at the core of our approach. Our managed services include setting up a complete observability stack for every client.
Why choose OpenSearch?
OpenSearch is a fully open, vendor-neutral standard that gives you complete observability and management of your data. It also works with your existing SIEM and analytics tools, making it ideal for large organizations that need a fast way to ingest data from multiple sources.
For developers, OpenSearch provides a standard interface for interacting with metadata on code dependencies without having to write additional code. This means they spend less time on instrumentation and more time building features. OpenSearch helps maximize developer productivity, making it an attractive prospect for any organization, regardless of size.
Key advantages of OpenSearch include:
- Open source and community-driven. No vendor lock-in, no expensive proprietary licences.
- Compatible with the Elasticsearch ecosystem. If you already use Elasticsearch, migration is simplified.
- Extensible. Plugins, integrations and customizations are possible without limits.
- Performant. Capable of handling massive data volumes in real time.
Configuring OpenSearch
If you're deploying a new application and want it to be observable, we recommend configuring OpenSearch from the start. Setting up OpenSearch will allow you to ingest your application logs using search filters. This is essential for understanding what's happening with your application in real time, so you can react quickly when things go wrong.
If you have many different microservices running on multiple hosts, sending data from each host independently may not be practical. Centralizing log shipping from a single source means all your data is in one place, saving you headaches later when you want to analyze it.
Logging services such as Beat agent, Logstash or Fluentd can also be used alongside OpenSearch. The Jelastic certified template is created for each open-source stack mentioned (OpenSearch, OpenSearch Dashboards, Logstash). These three components are combined into a single self-clustering solution, which significantly simplifies deployment.
Ingesting your data
You can ingest data into OpenSearch with many tools, including Logstash. Logstash is an event and log management application. Although it was originally created by Elasticsearch, it now supports other products such as Apache Kafka and Amazon Kinesis.
Logstash ingests data from almost any source using various methods, including TCP/UDP sockets and file system connectors (for example, S3 or FTP). Once your data is in Logstash, you can run simple or complex queries for better visibility into problems or trends in your application environment.
Logstash's flexibility is one of its greatest strengths. Whether your data comes from Docker containers, physical servers, cloud applications or IoT devices, Logstash can ingest and transform it before sending it to OpenSearch.
Creating your first dashboard
Create your first real-time dashboard by ingesting data from OpenSearch. Start by logging into your OpenSearch account and selecting a collection where you want to view metrics. Then add a search to a new or existing application you're interested in, such as Kubernetes.
In Kubernetes, add labels for each key metric collected, such as CPU and memory usage. OpenSearch Dashboards lets you create custom visualizations, trend charts and comprehensive dashboards that provide an instant overview of your infrastructure's health.
Adding alerts
You can easily add alerts to OpenSearch, allowing Ops teams to create notifications based on specific events. For example, if an application fails to start, an alert can be generated and sent via email or Slack.
OpenSearch comes with a set of simple rules for quickly finding failing instances. But to go further, you can configure OpenSearch so that if an instance fails multiple times within a given period (for example, three failures in 15 minutes), it automatically triggers a corrective action. This way, your application continues to run efficiently while reducing costs.
Deploying OpenSearch with Hidora
At Hidora, we've developed a PaaS template that lets you get up and running quickly. In just a few minutes, you can deploy your own fully functional OpenSearch instance. Our consulting team can also help you implement a comprehensive observability strategy tailored to your specific needs.
Observability is not a luxury, it's a necessity for any organization operating production systems. With OpenSearch and Hidora's support, you can set up a robust, scalable and cost-effective solution.
