ivaap

When doing demos of IVAAP, the wow factor is undeniably its user interface, built on top of GeoToolkit.JS. What users of IVAAP typically don’t see is the part accessing the data itself, the IVAAP backend. When we designed the IVAAP backend, we wanted our customers to be able to extend its functionalities. This is one of the reasons we chose Java for its programming language—customers typically have access to Java programmers.

Java is the programming language; it is a well-known, generic-purpose language, but the IVAAP Backend Software Development Kit (SDK) is typically only discovered during an IVAAP evaluation. In previous articles, I described the Lookup API (How to Empower Developers with à la Carte Deployment in IVAAP Upstream Data Visualization Platform) and the use of scopes (Using Scopes in IVAAP: Smart Caching and Other Benefits for Developers). As the SDK has grown, I thought it would be a good time to review what else this SDK provides.

One Optimized Use Case: Plugging Your Own Data

The most common question that I get is: “I see that you can access a WITSML datasource, a PPDM database. I have my own proprietary store for geoscience data, what do I need to do to make IVAAP visualize the data for my data store?” This is where the SDK comes into play. You do not need to modify IVAAP backend’s code to add your own data. In a nutshell, you just need to write a few Java classes, compile them, and add them to your IVAAP deployment.

The Java classes you write need to meet the Application Programming Interface (API) that the SDK defines. If you are a developer, this answer is not enough, this is the textbook definition of a SDK. What makes the IVAAP Backend SDK efficient for our use case is that you only need to write the API for the data you have. Since IVAAP’s built-in data model allows the visualization of maybe 30 different aspects of a well (log curves, deviations, tubing sets, mud logs, raster logs, etc), you only need to write classes for the data you have. For example, to visualize log curves, regardless of how these curves are stored, you only need to write about a dozen classes for a complete implementation.

The next question I get at this point is: “How do I know what to write?”. There is a large amount of documentation available. During the evaluation process, you are granted access to our developers site. This site is a reference used by all INT developers working on the IVAAP backend, whether they are developing IVAAP itself, or creating plugins for customers. It’s a Wiki and gets updated regularly. When I get support questions about the SDK, I typically will write an article in that Wiki and share the link. This is not the only piece of documentation available. There is a classic JavaDoc documentation that details the API in a formal manner. And there is also sample code. We created a sample connector to a SQL database storing well curves, trajectories, well locations and schematics as a practical example on how to use the SDK.

An Extensive Geoscience Data Model to Leverage

Lots of work has been done in IVAAP to facilitate workflows associated with wells, whether they are drilling workflows, production monitoring workflows, or just to manage an inventory. Specifically, IVAAP has a data model to expose the location of wells, log curves, deviation curves, mud logs, schematics, fracking, core images, raster logs, tops and any type of well documentation. Wells are not the only data models that IVAAP includes. Other models exist for seismic data and reservoirs. Several types of surfaces are also supported such as faults, grid surfaces, triangle meshes and seismic horizons.

These data models were built over-time based upon the common denominator between models coming from different systems. For example, if you are familiar with WITSML, you will find that the definition of a well log resembles what WITSML provides, but is flexible enough to also support LAS and DLIS files. From a developer perspective, the data model is exposed through the SDK’s API, without making any assumption on how this data is stored. The data model works for data stored in the cloud, on a file system, in a SQL database, and even data exposed only through a web service. While most of IVAAP’s connectors access one form of data store at a time, some connectors mix storages to combine data from web services and cloud storages. IVAAP’s data model is storage-agnostic, and the services to expose this data model to the HTML5 client are storage-agnostic as well.

IVAAP covers the most common data types found in geoscience. It provides the services to access this data, and the UI to visualize it. When starting an IVAAP development project, most developers should only have to focus on plugging their data, expressing through the SDK’s API on how to retrieve this data.

An API to Customize Entitlements

There is one more way that the IVAAP SDK makes the developer experience seamless when plugging a proprietary datastore. Not only does no code have to be written to expose this data to the viewer, but no code has to be written to control who has access to which data. Both aspects are built-in into the code that will call your implementation. You only have to write the data access layer, and not worry about entitlements or web services. By default, entitlements are based upon the information entered in the IVAAP Administration application.

This separation between data access and entitlements saves development time, but there are cases when a data store controls both data and access to this data. When IVAAP needs to access such an integrated system, the entitlement checks layer needs to be performed by the data access code. The entitlement API allows these checks to be performed at the data level.

The entitlement API is actually very fine-grained. You can customize the behavior of each service to limit access to specific data points. For example, the default behavior of IVAAP is to grant access to all curves of a well when you have been granted access to that well. Depending on your business rules, you might elect to restrict access to specific log curves. The SDK doesn’t force you into an “all or nothing” decision.

An API to Implement Your Own REST Services

Another typical use case is when you need to give access to data that doesn’t belong to the IVAAP built-in data model. In this particular situation, you need to extend IVAAP by adding custom widgets, and ad-hoc web services are needed to expose the relevant data to this widget. There is of course an API for this. External developers use the same API as INT developers to implement web services. INT has developed more than 500 REST services using this API, and external developers benefit from this experience.

Most services are JSON-based, and IVAAP uses the jackson libraries to create JSON content. To advertise capabilities to the HTML5 client, the IVAAP backend uses HATEOAS links. For example, if the JSON description of a well has a link to the mud logs services, then this well has mud logs. If this link is not present, the HTML5 client understands that this well doesn’t contain mud logs, and will adapt its UI accordingly. If you were to add your own service exposing more data associated with a well, you would typically want to add your own HATEOAS to the description of wells. Adding HATEOAS links to existing services is possible by plugging so-called Entity classes. You do not need to modify the code of this service to modify its behavior.

IVAAP’s REST services follow the OpenAPI specifications. There is actually a built-in web service whose only purpose is to expose the available services in the classic Swagger format. IVAAP’s SDK uses annotations similar to the Swagger Annotations API. If you are familiar with this API, documenting your own REST services should be a breeze.

Most of the REST services are JSON-based, but sometimes binary streams are used instead for performance reasons. Binary streams are typically used in IVAAP to expose seismic data, but also surfaces. The SDK uses events to implement such streaming services.

An API to Implement Your Own Real Time Feeds

The service API is not limited to REST services. An API is also available to communicate with the IVAAP HTML5 client through websockets. The WebSockets API is typically used to implement real time communications between the client and the server. For example, when a user opens a well, the user interface uses websockets to send a subscription message to the backend, requesting to be notified if this well changes. This enables a whole set of capabilities, such as real time monitoring. This is the API we use to monitor wells from WITSML datasources. The SDK includes an entire set of hooks so that customers can write their own feeds, including subscription, unsubscription and broadcast of messages.

When you write REST services, the container details are abstracted away and you only need to worry about implementing domain-related code. A REST service working in a Tomcat based development environment will work without any modification in a Play cluster. Likewise, feeds developed with the SDK work seamlessly in both Tomcat and Play. On a developer station, the SDK will use end points from the Servlet API to carry messages. In a Play cluster, the SDK will use ActiveMQ. ActiveMQ allows scalability and reliability features that servlets miss, such as high-rate of messages, and reliable delivery of messages. The use of ActiveMQ is transparent to the developers of feeds.

Utilitarian APIs

There is more to the IVAAP SDK than its APIs to access data, write services or customize entitlements. There are a few other APIs worth mentioning. One of them is the API to perform CRS conversions. Its default implementation uses Apache SIS, but the API itself is generic in nature. CRS conversions are often needed in geoscience, for example to visualize datasets on a map, on top of satellite imagery. Years of work has been built into the Apache SIS library, and virtually no work is needed by IVAAP developers to leverage this library when the SDK is used.

There are also APIs to execute code at startup and to query the environment that IVAAP is running on. The Lookup API gives access to the features that are plugged. The DataSource API indicates which data sources are configured to run in the JVM. The Hosted Services API provides an inventory of the external services that an IVAAP instance needs to interact with. A hosted service could be the REST service that evaluates formulas, or the machine learning system that IVAAP feeds its data to.

A “Developer-Friendly” Development Environment

We made lots of efforts to make sure the development process would be as simple as possible. Developers with experience with Java Servlets will be at ease with their IVAAP development environment. They will use tools they are familiar with such as Eclipse and Tomcat. A production instance of IVAAP doesn’t use servlets, it uses the Play framework. By following the SDK’s API, it is virtually transparent to developers that their code will be deployed in a cluster.

There are a few instances where awareness of the cluster environment is needed. For example, when caching is involved, you want to make sure that all caches are cleared across all JVMs when data gets updated. The IVAAP SDK includes an API to send and receive cluster events, and to create your own events. Since events are serialized from/to JSON, instances in the cluster do not need to share the same build version to interact with each other. This was a deliberate design choice so that you can upgrade your cluster while it’s running, without service interruption.

Caching is a large topic, outside of the scope of this article. IVAAP’s SDK proposes a “DistributedStore” API that hides the complexity of sharing state across JVMs. As long as you use this API, code that caches data will work without any modification in a single-JVM development environment and a multiple-JVMs production environment.

Finally, the SDK’s API is designed to allow fast iterative development. For example, once you have implemented the two classes that define how to list wells in your datastore, you can test them right away with Postman. Earlier I wrote that plugging your own log curves requires about a dozen classes. There is no need to write all twelve to start seeing results. Actually, you do not need to launch Postman to test your web services. You can test services using JUnit. A REST service written with the SDK can be tested with JUnit. This saves time by eliminating the need to launch Tomcat.

When you evaluate IVAAP, you might not have enough time to grasp the depth of the IVAAP SDK. Hopefully, this guide will help you get started.