Products

Nov 20 2020

A New Era in O&G: Critical Components of Bringing Subsurface Data to the Cloud

The oil and gas industry is historically one of the first industries generating actionable data in the modern sense. For example, the first seismic imaging was done in 1932 by John Karcher.

first-seismic — Seismic dataset in 1932.

Since that first primitive image, seismic data has been digitized and has grown exponentially in size. It is usually represented in monolith data sets which may span in size from a couple of gigabytes to petabytes if pre-stack.

seismic-faults — Seismic datasets today.

The long history, large amount of data, and the nature of the data pose unique challenges that often make it difficult to take advantage of advancing cloud technology. Here is a high-level overview of the challenges of working with oil and gas data and some possible solutions to help companies take advantage of the latest cloud technologies.

Problems with Current Data Management Systems

Oil and Gas companies are truly global companies, and the data is often distributed among multiple disconnected systems in multiple locations. This not only makes it difficult to find and retrieve data when necessary but also makes it difficult to know what data is available and how useful it is. This often requires person-to-person communication, and some data may even be in offline systems or on someone’s desk.

The glue between those systems is data managers who are amazing at what they do but still introduce a human factor to the process. They have to understand which dataset is being requested, then search for it on various systems, and finally deliver it to the original requester. How much does this process take? You guessed it—way too much! And in the end, the requester may realize that it’s not the data they were hoping to get, and the whole process is back to square one.

After the interpretation and exploration process, decisions are usually made on the basis of data screenshots and cherry-picked views, which limit the ability of specialists to make informed decisions. Making bad decisions based on incomplete or limited data can be very expensive. This problem would not exist if the data was easily accessible in real-time.

And that doesn’t even factor in collaboration between teams and countries.

How can O&G and service companies manage
their massive subsurface datasets better
by leveraging modern cloud technologies?

3 Key Components of Subsurface Data Lake Implementation

There are three critical components of a successful subsurface data lake implementation: a strong cloud infrastructure, a common data standard, and robust analysis and visualization capabilities.

AWS: Massive Cloud Architecture

While IVAAP is compatible with any cloud provider—along with on-premise and hybrid installations—AWS offers a strong distributed cloud infrastructure, reliable storage, compute, and more than 150 other services to empower cloud workflows.

OSDU: Standardizing Data for the Cloud

The OSDU Forum is an Energy Industry Forum formed to establish an open subsurface Reference Architecture, including a cloud-native subsurface data platform reference architecture, with usable implementations for major cloud providers. It includes Application Standards (APIs) to ensure that all applications (microservices), developed by various parties, can run on any OSDU data platform, and it leverages Industry Data Standards for frictionless integration and data access. The goal of OSDU is to bring all existing formats and standards under one umbrella which can be used by everyone, while still supporting legacy applications and workflows.

IVAAP: Empowering Data Visualization

A data visualization and analysis platform such as IVAAP, which is the third key component to a successful data lake implementation, provides industry-leading tools for data discovery, visualization, and collaboration. IVAAP also offers integrations with various Machine Learning and artificial intelligence workflows, enabling novel ways of working with data in the cloud.

Modern Visualization — The Front End to Your Data

To visualize seismic data, as well as other types of data, in the cloud, INT has developed a native web visualization platform called IVAAP. IVAAP consists of a front-end client application as well as a backend. The backend takes care of accessing, reading, and preparing data for visualization. The client application provides a set of widgets and UI components empowering search, visualization, and collaboration for its users. The data reading and other low-level functions are abstracted from the client by a Domain API, and work through connector microservices on the backend. To provide support for a new data type, you only need to create a new connector. Both parts provide an SDK for developers, and some other perks as well.

Compute Close to Your Data

Once the data is in the cloud, a variety of services become available. For example, one of them is ElasticSearch from AWS, which helps index the data and provides a search interface. Another service that becomes available is AWS EC2, which provides compute resources that are as distributed as the data is. That’s where IVAAP gets installed.

One of the cloud computing principles is that data has a lot of gravity and all the computing parts tend to get closer to it. This means that it is better to place the processing computer as close to the data as possible. With AWS EC2, we at INT can place our back end very close to the data, regardless of where it is in the world, minimizing latency for the user and enabling on-demand access. Elastic compute resources also enable us to scale up when the usage increases and down when fewer users are active.

All of this works together to make your data on-demand—when the data needs to be presented, all the tools and technologies mentioned above come into play, visualizing the necessary data in minutes, or even seconds, with IVAAP dashboards and templates. And of course, the entire setup is secure on every level.

Empower Search and Discovery

The next step is to make use of this data. And to do so, we need to provide users a way to discover it. What should be made searchable, how to set up a search, and how to expose the search to the users?

Since searching through numerical values of the data won’t provide a lot of discovery potential, we need some additional metadata. This metadata is extracted along with the data and also uploaded to the cloud. All of it or a subset of metadata is then indexed using AWS Elasticsearch. IVAAP uses an Elasticsearch connector to the search, as well as tools to invoke the search through an interactive map interface or filter forms presented to the user.

How can you optimize web performance of massive domain datasets?

Visualizing Seismic Datasets on the Web

There are two very different approaches to visualizing data. One is to do it on the server and send rendered images to the client. This process lacks interactivity, which limits the decisions that can be made from those views. The other option is to send data to the client and visualize it on the user’s machine. IVAAP implements either approach.

While the preferred method—sending data to the client’s machine—provides limitless interactivity and responsiveness of the visuals, it also poses a special challenge: the data is just too big. Transferring terabytes of data from the server to the user would mean serious problems. So how do we solve this challenge?

First, it is important to understand that not all the data is always visible. We can calculate which part of the data is visible on the user’s screen at any given moment and only request that part. Some of the newer data formats are designed to operate with such reads and provide ways to do chunk reads out of the box. A lot of legacy data formats—for example, SEG-Y—are often unstructured. To properly calculate and read the location of the desired chunk, we need to first have a map—called an Index—that is used to calculate the offset and the size of chunks to be read. Even then, the data might still be too large.

Luckily, we don’t always need the whole resolution. If a user’s screen is 3,000 pixels wide, they won’t be able to display all 6,000 traces, so we can then adaptively decrease the number of traces to provide for optimal performance.

Often the chunks which we read are in different places in the file, making it necessary to do multiple reads at the same time. Luckily, both S3 storage and IVAAP support such behavior. We can fire off thousands of requests in parallel, maximizing the efficiency of the network. Live it to the full, as some people like to say. And even then, once the traces are picked and ready to ship, we do some vectorized compression before shipping the data to the client.

We were talking about legacy file formats here, but it’s worth mentioning that GPU compression is also available for newer file formats like VDS/OpenVDS and ZGY/OpenZGY. It’s worth mentioning that the newer formats provide perks like brick storage format, random access patterns, adaptive level of detail, and more.

Once the data reaches the client, JavaScript and Web Assembly technologies come together to decompress the data. The data is then presented to the user using the same technologies through some beautiful widgets, providing interactivity and a lot of control. From there, building a dashboard—drilling, production monitoring, exploration, etc.—with live data takes minutes.

All the mentioned processes are automated and require minimal human management. With all the work mentioned above, we enable a user to search for the data of interest, add it to desired visualization widgets (multiple are available for each type of data), and display on their screen with a set of interactive tools to manipulate the visuals. All within minutes, and while being in their home office.

That’s not all—a user can save the visualizations and data states into a dashboard and share it with their colleagues sitting on a different continent, who can then open the exact same view in a matter of minutes. With more teams working remotely, this seamless collaboration helps facilitate collaboration and reduce data redundancy and errors.

Data Security

How do we keep this data secure? There are two layers of authentication and authorization implemented in such a system. First, AWS S3 has identity-based access guarantees that data can be visible to only authorized requests. IVAAP uses OAuth2 integrated with AWS Cognito to authenticate the user and authorize the requests. The user logs into the application and gets a couple of tokens that allow them to communicate with IVAAP services. The client passes tokens back to the IVAAP server. In the back end, IVAAP validates the same tokens with AWS Cognito whenever data reads need to happen. When validated, a new, temporary signed access token is issued by S3, which IVAAP uses to make the read from the file in a bucket.

Takeaways

Moving to the cloud isn’t a very simple task and poses a lot of challenges. By using technology provided by AWS and INT’s IVAAP and underlined by OSDU data standardization, we can create a low-latency data QC and visualization system which puts all the data into one place, provides tools to search for data of interest, enables real-time on-demand access to the data from any location with the Internet, and does all that in a secure manner.

For more information on IVAAP, please visit int.com/ivaap/ or to learn more about how INT works with AWS to facilitate subsurface data visualization, check out our webinar, “A New Era in O&G: Critical Components of Bringing Subsurface Data to the Cloud.”

Nov 17 2020

INT’s GeoToolkit.JS 2020.1 Release Brings Major Features in 2D/3D Visualization and Optimized Rendering in Web Browser

The release also includes many new tutorials and demos for ES6, new features around WebGL, WebAssembly, and more.

Houston, TX —INT is pleased to announce the newest release of GeoToolkit.JS data visualization libraries and toolkit. This major release includes new 3D visualization capabilities—contour 3D, 3D shape highlighting, and better 3D seismic—and new rendering options.

“Visualizing seismic data in the cloud is a challenge for many oil and gas companies. We have been working closely with our developers and partners to find ways to optimize seismic visualization and workflows for our users,” said Dr. Dmitriy Lukyanenko, Director of Software Development at INT. “We’ve added server-side rendering and improved our decompression and compression techniques to speed up performance and have optimized our seismic reader. We think these improvements will result in significant performance gains for our clients.”

GeoToolkit.JS 2020.1 also includes improved capabilities such as combined Vertical and Deviated Schematics, more robust BHA editor, new heatmap layer, and seismic decompression and filters with WebAssembly.

RELEASE HIGHLIGHTS:

Server-Side Rendering Capabilities — To offer a complete set of options for great performance for interacting with seismic in the cloud, we added support for server-side rendering in addition to the existing widget for client-side rendering by just integrating new services
Improved ES6 Support — Full set of all-new tutorials and demos for ES6 with code snippets. Supports Angular, React, and Vue frameworks.
Redesigned the Graphical Annotation — A new version of annotation tool with layer support to add text and graphical annotations, plus a demo to demonstrate this functionality to facilitate adoption
New 3D Features and Faster 3D Rendering — New contour 3D (in addition to existing 2D), optimized 3D seismic visualization (same API but faster rendering), and new 3D shape highlighting, and multilateral trajectory
MultiWell Performance Enhancements — Improved performance of MultiWellWidget if the same template is used for all wells
WebGL — Optimized deviated image and gradients and array log (log2d) with WebGL.

Read the press release on PRWeb >

For more information about GeoToolkit.JS or INT’s other data visualization products, please visit www.int.com.

Nov 09 2020

Human Friendly Error Handling in the IVAAP Data Backend

As the use cases of IVAAP grow, the implementation of the data backend evolves. Past releases of IVAAP have been focused on providing data portals to our customers. Since then, a new use case has appeared where IVAAP is used to validate the injection of data in the cloud. Both use cases have a lot in common, but they differ in the way errors should be handled.

In a portal, when a dataset fails to load, the reason why needs to stay “hidden” to end-users. The inner workings of the portal and its data storage mechanisms should not be exposed as they are irrelevant to the user trying to open a new dataset. When IVAAP is used to validate the results of an injection workflow, many more details about where the data is and how it failed to load need to be communicated. And these details should be expressed in a human friendly way.

To illustrate the difference between a human-friendly message and a non-human friendly message, let’s take the hypothetical case where a fault file should have been posted as an object in Amazon S3,… but the upload part of the ingestion workflow failed for some reason. When trying to open that dataset, the Amazon SDK would report this low-level error: “The specified key does not exist. (Service S3, Status Code: 404, Request ID: XXXXXX)”. In the context of an ingestion workflow, a more human friendly message would be “This fault dataset is backed by a file that is either missing or inaccessible.”

The IVAAP Data Backend is written in Java. This language has a built-in way to handle errors, so a developer’s first instinct is to use this mechanism to pass human friendly messages back to end users. However, this approach is not as practical as it seems. The Java language doesn’t make a distinction between human-friendly error messages and low-level error messages such as the one sent by the Amazon SDK, meant to be read only by developers. Essentially, to differentiate them, we would need to create a HumanFriendlyException class, and use this class in all places where an error with a human-friendly explanation is available.

This approach is difficult to scale to a large body of code like IVAAP’s. And the IVAAP Data Backend is not just code, it also comes with a large set of third-party libraries that have their own idea of how to communicate errors. To make matters worse, It’s very common for developers to do this:

    try {

             // do something here

        } catch (Exception ex) {

            throw new RuntimeException(ex);

        }

This handling wraps the exception, making it difficult to catch by the caller. A “better” implementation would be:

try {

             // do something here

       } catch (HumanFriendlyException ex) {

            throw ex;

        } catch (Exception ex) {

            throw new RuntimeException(ex);

        }

While this is possible to enforce this style for the entirety of IVAAP’s code, you can’t do this for third party libraries calling IVAAP’s code.

Another issue with Java exceptions is that they tend to occur at a low-level, where very little context is known. If a service needs to read a local file, a message “Can’t read file abc.txt” will only be relevant to end users if the primary function of the service call was to read that file. If reading this file was only accessory to the service completion, bubbling up an exception about this file all the way to the end-user will not help.

To provide human-friendly error messages, IVAAP uses a layered approach instead:

High level code that catches exceptions reports these exceptions with a human friendly message to a specific logging system
When exceptions are thrown in low level code, issues that can expressed in a human friendly way are also reported to that same logging system

With this layered approach where there is a high-level “catch all”, IVAAP is likely to return relevant human friendly errors for most service calls. And the quality of the message improves as more low-level logging is added. This continuous improvement effort is more practical than a pure exception-based architecture because it can be done without having to refactor how/when Java exceptions are thrown or caught.

To summarize, the architecture of IVAAP avoids using Java exceptions when human-friendly error messages can be communicated. But this is not just an architecture where human-friendly errors use an alternate path to bubble up all the way to the user. It has some reactive elements to it.

For example, if a user calls a backend service to access a dataset, and this dataset fails to load, a 404/Not Found HTTP status code is sent by default with no further details. However, if a human friendly error was issued during the execution of this service, the status code changes to 500/Internal Server Error, and the content of the human friendly message is included in the JSON output of this service. This content is then picked up by the HTML5 client to show to the user. I call this approach “reactive” because unlike a classic logging system, the presence of logs modifies the visible behavior of the service.

With the 2.7 release of IVAAP, we created two categories of human friendly logs. One is connectivity. When a human friendly connectivity log is present, 404/Not Found errors and empty collections are reported with a 500/Internal Server Error HTTP status code. The other is entitlement. When a human friendly entitlement log is present, 404/Not Found errors and empty collections are reported with a 403/Forbidden HTTP status code.

The overall decision on which error message to show to users belongs to the front-end. Only the front-end knows the full context of the task a user is performing. The error handling in the IVAAP Data Backend provides a sane default that the viewer can elect to use, depending on context. OSDU is one of the deployments where the error handling of the data backend is key to the user experience. The OSDU platform has ingestion workflows outside of IVAAP, and with the error reporting capabilities introduced in 2.7, IVAAP becomes a much more effective tool to QA the results of these workflows.

For more information on INT’s newest platform, IVAAP, please visit www.int.com/products/ivaap/

Nov 06 2020

How to Get the Best Performance out of Your Seismic Web Applications

One of the most challenging data management problems faced in the industry is with seismic files. Some oil and gas companies estimate that they acquire a petabyte of data per day or more. Domain knowledge and specific approaches are required to move, access, and visualize that data.

In this blog post, we will dive deep into the details of modern technology that can be useful to achieve speed up. We will also cover: common challenges around seismic visualization, how INT helps solve these challenges with advanced compression and decompression techniques, how INT uses vectorization to speed up compression, and more.

What Is IVAAP?

IVAAP is a data visualization platform that accelerates the delivery of cloud-enabled geoscience, drilling, and production solutions.

IVAAP Client offers flexible dashboards, 2D & 3D widgets, sessions, and templates
IVAAP Server side connects to multiple data sources, integrates with your workflows, and offers real-time services
IVAAP Admin client manages user access and projects

Server – Client Interaction

Interaction occurs when the client requests a file to display from the server, the server returns the file lists, the user chooses a file to display, and then the server starts sending chunks of data while it displays this data.

Some issues encountered with this scheme include:

Seismic data files are huge in size — they can be hundreds of gigabytes or even terabytes.
Because of the file size, it takes too much time to transfer files via network.
The network can have too much bandwidth.

The goals of this scheme are to:

Speed up file transfer time
Reduce data size for transfer
Add user controls for different network bandwidth

And the solution:

We decided to implement server-side compression and client-side decompression. We also decided to provide the client parameter that we call acceptable error level after the seismic data file compression/decompression process.

By taking a closer look at compression and decompression data, we can see that the original seismic data goes through a set of five transformations — AGC, Normalization, Hear Wavelets, Quantization, and Huffman. As a result of this transformation, we get a compressed file that can be sent to clients via network. And on the client’s side, there is a decompression process that goes in different directions — from inverse Huffman to inverse AGC. This is the way that clients get original data. It does not get precise, original data. But it gets data after the compression and decompression process. That’s why we added an acceptable error level after the compression and decompression process. This is because we have different scenarios where clients don’t always require the full original data with the full level of precision. For example, sometimes the client only needs to review the seismic data. So using this acceptable error level, they can control how much data will be passed by a network and, of course, speed up this process.

The resulting scheme looks like this:

The client requests a file list from the server, the user chooses a file to display, and then the server starts sending the data and compresses it. The server then sends it to the client, the client decompresses, and finally, it displays the data. This is repeated for each tile to display.

So why not use any other existing compression, like GZIP, LZ Deflate, etc.? We tried these compressions, but we found out that this type of compression is not as effective as we’d like it to be on our seismic data.

Server-Side Interaction

The primary objective was to speed up the current implementation of compression and decompression on both the server and client side.

The proposal:

Server-side compression is implemented in Java, so we decided to create C++ implementation of compression sequence and use JNI layer to call native methods. For the client-side decompression, we implemented in JavaScript to create C++ implementation of decompression and use WebAssembly (WASM) proposal for integrating C++ code into JS.
We implemented both compression and decompression algorithms in C++, but after comparing the results and performance of C++ and Java, we discovered that C++ was just 1.5 times faster than “warmed up JVM”. That’s why we decided to move on and apply SIMD instructions for further speedup.

Single Instruction Multiple Data (SIMD)

SIMD architecture performs the same operation on multiple data elements in parallel. For Scalar operation, you have to perform four separate calculations to get the right result. For SIMD operations, you apply one vector value calculation to get the correct result.

SIMD benefits:

Allows processing of several data values with one single instruction.
Much faster computation on predefined computation patterns.

SIMD drawbacks:

SIMD operations cannot be used to process multiple data in different ways.
SIMD operations can only be applied to predefined processing patterns with independent data handling.

Normalization: C++ scalar implementation

Normalization: C++ SIMD SSE implementation

Server-Side Speedup Results

There are different types of speedup for different algorithms:

Normalization is 9 times faster than the scalar C++ version
Haar Wavelets is 6 times faster than the scalar C++ version
Huffman has no performance increase (not vectorizable algorithm)

Overall, the server-side compression performance improvement is around 3 times faster than the Java version. This is applying SIMD C++ code. This was good for us, so we decided to move on to the client-side speedup.

Client-Side Speedup

For the client-side speedup, we implemented decompression algorithms in C++ and used WASM to integrate the C++ code in JavaScript.

WebAssembly

WASM is:

A binary executable format that can run in browsers
A low-level virtual machine
A high-level language compile result

WASM is not:

A programming language
Connected to the web and cannot be run outside the web

Steps to get WASM working:

Compile C/C++ code with Emscripten to obtain a WASM binary
Bind WASM binary to the page using a JavaScript “glue code”
Run app and let the browser instantiate the WASM module, the memory, and the table of references. Once that is done, the WebApp is fully operative.

C++ Code to Integrate (TaperFilter.h/cpp)

Emscripten Bindings

WebAssembly Integration Example

Client-Side Speedup Takeaways:

Emscripten supports the WebAssembly SIMD proposal
Vectorized code will be executed by browsers
The results of vectorization for decompression algorithm are:
- Inv Normalization: 6 times speedup
- Inv Haar Wavelets: 10 times speedup
- Inv Huffman: no performance improvement (not vectorizable)

Overall, the client-side decompression performance improvement with vectorized C++ code was around 6 times faster than the JavaScript version.

For more information on GeoToolkit, please visit int.com/geotoolkit/ or check out our webinar, “How to Get the Best Performance of Your Seismic Web Applications.”

Oct 07 2020

Search, Access, and Visualize OSDU Data with IVAAP

INT and OSDU Collaboration

INT has been working with the Open Subsurface Data Universe™ (OSDU) consortium since the beginning of the OSDU forum in 2019. OSDU is a standard data platform for the oil and gas industry which will reduce silos and put data at the center of the subsurface community. INT made IVAAP available as part of the Demo release of the OSDU platform.

IVAAP: The Premier HTML5 Data Visualization Platform for OSDU

At INT, we worked closely with the OpenSDU team to assist with validating the platform and creating a robust framework.

Developers can deliver cloud-based digital solutions quicker than ever.
Using the powerful OSDU search and delivery API, IVAAP allows users to search, find, display, and analyze data on the fly, including log, trajectories, tops, seismic, horizon, and fault data.
Creating a common environment hosted in the cloud enables oil and gas companies to efficiently access massive amounts of data, reduce data silos, collaborate remotely, implement machine learning (ML), and lower the cost of operations.

Search and interact with different data types.

Access your data in the cloud from anywhere.

Visualize data aggregated from various data sources in one place.

With IVAAP on OSDU R2, you can:

Combine data on the fly from wells, seismic, and many other sources.
Use a microservices architecture, IVAAP supports industry data standards, including OSDU, PPDM, WITSML, and several commercial databases.
Enable user co-visualization.
Combine data types like real-time, historical, 2D/3D seismic, reservoir, and more in your web browser with IVAAP’s one-of-a-kind dashboard capabilities.

Request a demo for IVAAP today

What Is OSDU?

The OSDU Forum was created with the objective of enabling new cloud-native data-driven applications with seamless access to the full range of subsurface and wells data as well as supporting existing applications and data frameworks.

Separating data from applications is the core principle of the OSDU solution. The plan is to achieve this by developing a common data platform with standard public APIs and to involve global cloud hosting vendors to build working implementations that will:

Allow companies to focus on the truly differentiating part of business activities
Reduce the resource and cost burden of independently designing and validating a subsurface data platform
Influence and accelerate key reference components that will be the foundation of the upstream digital transformation.

The OSDU data platform will:

Enable secure, reliable, global, and performant access to all subsurface and wells data
Reduce current data silos to enable transformational workflows
Accelerate the deployment of emerging digital solutions for better decision-making
Create an open, standards-based ecosystem that drives innovation

View the official OSDU brochure.

An Industry Solution

The OSDU Forum will:

Publish a Reference Architecture: a cloud-native subsurface and wells data platform reference architecture with initial implementations by Microsoft^® Azure, Amazon Web Services^® and Google^® GCP, with others expected.
Define Application Standards to ensure applications (microservices), developed by various parties, can run on the same OSDU data platform
Leverage Industry Data Standards for frictionless integration and data access

Business Impact

Reducing data silos: subsurface and wells data integrated and accessible in one data platform
Workflows in the cloud: user workflows seamlessly executed across the OSDU data platform, taking advantage of powerful cloud-native solutions
Access to all metadata: users will have access to powerful search capabilities
Competition and cooperation: software vendors, oil companies, academia, and open community projects can develop software on the OSDU data platform

Scope

Current Scope

Coverage: any organization with a need to store, manage or analyze subsurface and wells data
Portfolios: subsurface including exploration, development, and wells
Support for Data Driven Applications: All data in one Data Platform will enable AI (such as Machine Learning) based applications to access this data

Current Problem Statement

Data is linked to applications: we are not data-driven but applications-driven and, therefore, we will have silos
Data is stored in silos: therefore, there is no lineage between data sources or from wells back to exploration
Metadata is not stored with the data: metadata is information about the data
Limited/no search capabilities: since we have no metadata, we have no real search capabilities
Data set-up: not suitable for data-driven applications

OSDU Scope

All public cloud-based: provides unlimited scalability
All data for the data platform services loaded via the data ingestion services: extract metadata, data quality, etc.
Well-defined (RestFUL) APIs defining the access to the data platform services
Applications: (micro) services-based
Information security: covering all elements with focus on data security
In-country solution: for those countries where we are unable to move data out of the country and where there are no cloud services
Support for legacy applications based on Microsoft^® Windows or Linux^® desktop

What Is the OSDU Value?