INT

Empowering Visualization

COMMUNITY BLOG
CONTACT US SUPPORT

ivaap

Using Scopes in IVAAP: Smart Caching and Other Benefits for Developers

With the release of IVAAP 2.2 coming up, there are lots of new features to explore. Since more data sources have being added, the Software Development Kit (SDK) of IVAAP’s backend has also grown. As developers get acquainted with IVAAP’s Application Programming Interface (API), one often-asked question concerns the presence of a scopeUUID parameter in several method declarations: What is this scope, and why is it useful?

Smart Caching

The purpose of IVAAP’s backend is to access the data from many data sources and present this data in a unified manner to the IVAAP HTML5 client. This backend is written in Java and accesses SQL databases, web services, basically any type of data store. Performance is key, and some data stores are faster to access than others. For example, web services tend to have much higher latency than SQL databases. In a web service configuration, a smart caching strategy is required so that the same web/HTTP calls are not made to the same data store twice while a service request is being fulfilled, regardless of its complexity. This is where the concept of scope comes in.

A scope is defined as a short-lived execution, and within this short lifetime, it’s generally OK to assume that two identical calls to the same data store will return the same result. The scopeUUID is a unique identifier of a scope. A scope is created automatically when an HTTP request is sent to a backend service, and disposed of when that entire request has been fulfilled. Depending of the performance characteristics of a data source, developers working on a data connector have the option to reuse cached information across the time span of a scope.

Scopes are needed because of the asynchronous nature of IVAAP’s execution architecture. Asynchronous code is not as common as synchronous code; it’s more complex to write, but tends to withstand higher workloads, which is a key requirement for IVAAP. If service executions were performed synchronously, the scope would be strongly associated with the current thread, and developers would typically cache the same information in a ThreadLocal variable instead.

Identifying the Source of a Method Call

While the most common use of scopes is for caching, being able to identify the source of a method call can be quite useful when running your application. In a synchronous world, the Java call stack provides this information. But in an asynchronous world where messages are being passed between actors, this call stack provides very little useful information. The IVAAP API allows you to build a useful call stack while developing. For example, if your backend code makes a call to an external web service, but you’re not sure why, you can plug your own AbstractScopesController class to control when this scope is created, then plug your own AbstractActorsController class to track how actors spawn other actors within that scope.

Managing Transactions

A third typical usage of scopes is to help with the management of transactions. When you retrieve a connection from a SQL connection pool, knowing the scope allows you to decide whether to reuse the same connection instead of the first one from the pool. Transactions are tightly associated with connections, and, without a scope, you’d need to pass connections around while a transaction is in play. Passing a scopeUUID instead avoids this code clutter, allowing you to easily implement APIs that are truly data-source agnostic.

Scope: Just More Code Clutter?

This brings me to the main objection to scopes: “Isn’t passing a scopeUUID a code clutter of its own?” The answer lies in how a developer plans to use the IVAAP backend SDK. If you use this API to write synchronous code, the scopeUUID will appear to get in the way. If you use the same API to write asynchronous code, you’ll find that the internal maintenance of scopes doesn’t affect the implementation of your actors. The scopeUUID of a service request is passed automatically from one actor to the next, without the developer’s intervention. If your data source is fast enough that it doesn’t require caching, if you do not make use of transactions, your asynchronous code will not need to be aware of scopes. It’s only when you will have to troubleshoot the relationships between actors at runtime that you might come to appreciate its usefulness.

Visit our products page for more information about IVAAP or contact us for a demo.


My Experience at INT with IVAAP: A First Look as a Developer

I started at INT a few weeks ago and my first task as a new INT developer was to add a data connector to IVAAP, INT’s HTML5 visualization framework for upstream E&P solutions.

As a new member of the software development team, I had no prior experience with development on this platform. To gain knowledge of IVAAP and to understand more about the IVAAP software development kit, I used the IVAAP developer’s guide. I found this guide quite useful as it made the key points behind IVAAP easily understandable.

With only a few years of experience with Java, I was surprised by the lookup system. IVAAP has a microservices REST architecture and is very modular in nature, and the lookup system ties all these modules together. It’s quite powerful, but this is something I had never encountered before.

Coding with IVAAP uses a simple model where each entity implementation consists of implementing a POJO (Plain Old Java Object) class and its finder. This paradigm is consistent throughout the entire code. Essentially, for this project, I plugged only a few classes:

  • A data source type class
  • A data source class
  • A log curve class and its finder
  • A log curve data series class and its finder
  • A log curve data frame class and its finder

Implementing these classes essentially consists of following templates, where the public API provides hooks and the developer adds the implementation specific to their project. The public API is documented, making it clear what each method or class is meant to perform.

Even though this project is to be deployed on Linux, my development environment was on Windows. It consisted of an Integrated Development Environment (the NetBeans IDE) and the Postman tool for testing individual services. This project accessed a SQL server database, so I used dbVisualizer to browse the data.

Since this project only included adding a data connector to IVAAP, I didn’t try to add new services, only extend the data sources it supports. Building a connector on top of the existing web services allowed me to validate my work as it progressed. For example, after plugging a new data source type, I could immediately verify that it worked as intended using Postman and following the HATEOAS links. This remained true when I plugged a data source and each finder. No need to wait until all classes are plugged to verify that the logic works. I also found that the error management built into IVAAP helped me be efficient since the error report made it easy to trace the actual issue.

The learning curve of the IVAAP software development kit is gradual. The API guides you. Unlike some of the frameworks I have worked with, there is no prior knowledge necessary to get started. You can be effective from day one with just basic Java knowledge.

Visit our products page for more information about IVAAP or contact us for a demo.


What Cloud Data Lakes Mean for Geoscience

With the explosion of storage capacity, cloud computing, and bandwidth availability, a trend has emerged in the oil and gas industry over the last few years. Data that was previously aggregated and discarded is now maintained and stored, creating an opportunity for the industry. Coupled with new advancements in machine learning and Al, this data availability is poised to drive more data driven decision making in well planning, drilling & completions, and well operations.

After our conversations with major operators over the last few months, we realized that the concept of a data lake is still pretty new and can be perceived differently. We thought this would be a great opportunity to explain what this technology approach is and its benefits and how major operators can leverage INT’s enterprise data visualization platform to help garner insights from geoscience data in the cloud.

What is a Data Lake?

The idea behind a data lake is that as businesses gather more data, this data cannot be used the same way it has been used in the past. Databases are not a good fit for big data, not just because of the size of that data but also because this data cannot be normalized. A Data Lake is a large storage of raw data where each data point is stored in its native format, along with relevant metadata.

This approach solves a problem that the oil and gas industry has faced for a long time. Despite industry efforts to standardize data formats, these formats are loosely followed and too limited to contain all information related to a particular seismic survey or an oil field. Keeping the data as is makes your job of extracting valuable information difficult. And if you opt to normalize that data, you inevitably lose information. The concept of Data Lake opens the possibility of both keeping your original data while allowing its exploitation.

How do you exploit data from a Data Lake?

When you add files to your Lake, you carry along these files’ relevant metadata. When the right set of metadata is provided, you can search your data from this set. For example, if each document is geo-referenced, you can search for all documents relative to a geographical area.

How is the concept of Data Lake different from a search engine?

Search engines can’t really exploit files in SEG-Y or LAS formats. And even for well-known formats such as PDF, search engines have no awareness of which attributes of a PDF file are important to you. For example, if your metadata indicates that a file documents the characteristics of a well at a particular location, a Data Lake will allow you to find this file just by selecting the right region of interest.

How are cloud providers helping with Data Lakes?

Microsoft Azure, Amazon Web Services, and Google Cloud have developed a wide range of products and components to facilitate the creation and use of data lakes, such as nearly infinite storage, artificial intelligence, and machine learning to provide a seamless way to ingest and consume your data. The technology behind such advanced indexing and analytics cannot be reproduced in-house—you need a world-class partner.

How is INT helping with Data Lakes?

INT helps in two ways: We have unique experience in the industry. Working with so many actors, we have acquired the knowledge required to read multiple data formats, even when these formats are not strictly followed. Our tools facilitate the extraction of the metadata required for the Lake to function as intended, and not as a “swamp.”

And, of course, our visualization technology is what makes it all possible, all from the comfort of your browser. Our IVAAP Enterprise Cloud Viewer allows you to visualize the datasets and documents stored remotely in your Data Lake. You’ll see how you can start from a map and drill down all the way down to the log curves of a well found on that map. In the same screen, you’ll be able to review PDF reports for that well and navigate through the slices of the matching seismic survey as if it was stored locally.

For more information about our enterprise data visualization solutions, visit the IVAAP product page, or contact us.


Bridging the Gap Between Business and IT: Visualization Architecture in the Digital Oilfield

A Closer Look at IVAAP

Thierry Danard, VP of Core Platform Technologies

In our latest Tech Talk, E&P Visualization in the Cloud, we featured IVAAP, our cloud-enabled visualization and analytics development platform. We showed how it can be used to monitor and analyze well data as a critical part of your digital transformation.

Thierry Danard, our VP of Core Platform Technologies, presented some of the technical aspects of IVAAP, so we asked him a few questions after the talk to dig a bit deeper:

> Hi, Thierry! We already know that you are the brains behind INTViewer, so which part of IVAAP are you responsible for?

I mostly work on the “P” part of IVAAP, the “platform.” IVAAP can be customized fully, both on the browser side and on the backend side. I focus on the backend side, meaning the microservices on the data side.

> What makes the IVAAP platform unique?

IVAAP comes with a Software Development Kit (SDK) so geoscience developers can tailor our solution to their needs. Developing solutions for the cloud is hard. We want to facilitate the work of these developers. The SDK is designed to ease the challenges developers face when developing distributed solutions.

But developers are not the only customers of IVAAP within IT. Deploying cloud solutions is also hard, and infrastructure folks want options when it comes to deployment. We made the IVAAP platform container-agnostic so that it can be deployed in a highly distributed environment or using standalone servers without changes: This is the same microservice code running.

IVAAP is unique because it bridges the gap between the business and IT: It provides a common platform that both sides can embrace, not just end-users.

 

SDK-architecture-ivaap
SDK Architecture

 

> Can you give us examples of containers that IVAAP works on?

The most widely used container for the IVAAP backend is Play. This is a high-velocity web framework designed to run on multiple machines, in a distributed fashion.

Another one is Apache Tomcat, the most widely used standalone Java application server. Other well-known JavaEE application servers are Oracle Glassfish and WebLogic.

> Why might a developer choose Tomcat over Play?

Not every customer has a network of machines to dedicate to well monitoring or analysis. Depending on what you use IVAAP for, you might not need distributed processing.

But developers also benefit. Developers can use the Integrated Development Environment (IDE) that they already use; it already works with Tomcat. No need to use a special environment, no need to install special plugins or to configure several servers. Developers can be productive from day one. The promise of IVAAP is to accelerate the delivery of geoscience, drilling and production cloud-enabled solutions. You can’t accelerate these deliveries unless your developers are productive.

The IVAAP Approach

 

> How does the SDK help developers create distributed microservices?

The IVAAP backend API makes a large use of the Akka library. Akka is a toolkit for building highly concurrent, distributed applications. The core Java programming model makes it very difficult for cloud developers to implement distributed processing. The Akka library addresses this concern with its simple model based on actors and messages.

Akka and Play are designed to work together. When Akka code is deployed in Play, you can sustain heavy loads. For example, the Akka actor system might decide to delegate individual processing units to one or several machines. This is virtually transparent to the developer as this is a behavior that depends on the state of each server.

> How does the SDK help developers create efficient microservices?

The API of the SDK is designed from the ground up to favor asynchronous execution over synchronous execution.

Synchronous code tends to reserve lots of resources just to wait for an answer. Asynchronous code doesn’t reserve these resources while a long processing task is being performed. Less CPU and less memory usage means more processing power for each deployed server, allowing your solution to perform under heavy loads.

> What’s coming next for the IVAAP backend?

Now that we made it easy to add new data sources and new microservices, we are adding connectivity to even more data repositories, such as OSISoft PI, Procount, or Peloton. This is a typical use case of the backend API. We have cleanly separated the microservices and data access parts. Now it’s just a matter of plugging additional data sources.

Stay tuned for more interviews with our developers! In the meantime, click here to learn more about IVAAP.