Summary
This document describes the motivation behind forming the Open Ag Data Alliance (OADA), data-related challenges that farmer’s face today, how OADA will address those challenges, and how farmers will benefit from OADA. OADA will create a secure data ecosystem that enables data security, privacy and interoperability for the entire agriculture industry. OADA will achieve this through an open standards software effort to establish secure data exchange protocols. This open software development approach is how Internet, network and web standards have succeeded in providing secure, scalable solutions for businesses and consumers alike.
Starting with a common, secure, and interoperable Application Program Interface (API) specification, an environment can be created in which farmers have complete freedom to choose best-in-class products from precision ag vendors with confidence in the data security and privacy and no danger of vendor data lock-in. Farmer participation in the newly-enabled OADA-compliant precision ag services market will drive innovation across the agriculture industry to enable the next generation of sustainable agriculture.
Background
Modern production agriculture has the potential to dramatically improve crop yields and reduce environmental impacts by enabling farmers to properly evaluate past, current and future farm management decisions through analysis of agronomic data generated in the field.
However farmers are currently overwhelmed with walled gardens of incompatible data generated by their existing systems (geodata images, logs, reports, charts). Farmer’s want the hardware and software systems they use to interoperate – that is, to share information and be able to adequately rely on each other to help support decision-making.
The most scalable solution for farmers is to enable their hardware and software systems to communicate automatically through secure cloud services or tools. Any system needs to incorporate data from a multitude of sources in order to be useful. However, existing solutions suffer from:
- an inability to gather data from all aspects of any given farm since no one manufacturer accesses all the operational data on a farm,
- farmers’ concern about what will happen to their data,
- and questions about data ownership and intellectual property.
In order to enable this next stage in production agriculture productivity, these three issues have to be resolved. Past attempts at solving data sharing and compatibility have revolved around creating monolithic standards that are licensed commercially and selectively. Farmers require an open solution that works with existing standards, adheres to clear privacy and security policies, and doesn’t require farmers to pay to access their own data. This is not a unique set of challenges. Other industries have faced a similar stage in their evolution, including financial services, healthcare, and the Internet. In all cases, a distributed, rather than centralized, network model emerged as best for the end users.
What OADA is
With the mission to ensure farmers have full data access, security and privacy, OADA:
- will operate with a farmer-focused approach through a central guiding principle that each farmer owns data generated or entered by the farmer, their employees or by machines performing activities on their farm,
- will develop open reference implementations of data storage and transfer mechanisms with security and privacy protocols,
- will provide a forum for technical community discussion,
- will be led according to the processes of open standards projects that have built much of the Internet’s networking, security, web and data standards with multiple university, individual and corporations participating (often while competing in the marketplace). Examples include the Internet Engineering Taskforce (IETF), World Wide Web Consortium (W3C) and The Apache Software Foundation, (which supports over 100 projects).
- will direct any financial contributions to a not-for-profit foundation whose purpose will be to enable open source projects in agriculture in service of the OADA mission.
What OADA is not
It is important to also specify what OADA does not intend to do. OADA:
- will not produce or sell commercial products,
- will not be a provider of cloud storage or services,
- will not be a political lobbying organization,
- and will not endorse or oppose products or services beyond providing software tests that can be used to validate OADA functional support.
The OADA Approach to Interoperability
The OADA approach to interoperability is to avoid implementation frameworks that inherently dictate winners and losers, focusing instead on providing developers with the useful libraries they need in order to communicate and access data without spending tedious hours memorizing several monolithic standards. In short, the problem of interoperability is largely solved simply by making API’s and libraries for data access publicly available rather than requiring everyone to use a predefined standard.
Existing efforts at enabling interoperability are focused on closed, licensed standards that attempt to enforce uniformity of data formats. While uniform representations of data are a laudable goal, organic growth of standards out of open implementations have proven far more effective in other industries than top-down control. For example, images can be stored in many “standard” formats (jpeg, png, gif, etc.), yet there is little developer pain around using images today because simple libraries grew out of communities that enable a developer to use all of them. Software exists today that can handle wide varieties of ag data formats. The real problem is getting secure access to data without tedious human intervention and repeatedly reinventing the wheel.
While each commercial vendor can implement their own OADA-compliant tools and services using technologies familiar to their organization, the existence of an open reference implementation that contains a verification test suite will provide a framework by which the industry can work together.
How OADA Supports Security & Farmer Privacy
OADA will develop authentication and security protocols that will support third-party verification of provider security and privacy practices. OADA security and privacy working groups will collaborate with stakeholder groups in defining the technical capabilities that can represent the range of business security and privacy practices that are desired by the broad community. The farmer will be able to use OADA services with a full understanding of different service provider security and privacy levels which are clearly and simply represented through their OADA-compliant tools.
Example Use Case
The OADA vision is best explained through an example. Consider the common use case of prescription maps. Frank is a farmer with a corn planter capable of varying seed population based on GPS location using a prescription map: i.e., under the irrigator seeds should be closer together and outside the irrigator they should be farther apart because there is less water to go around.
Andy, a local agronomist, has worked with Frank for years. Andy offers to create a custom prescription map for some of Frank’s fields. To do this, Andy says he needs the past three years of yield data, recent soil tests, as-applied fertilizer maps, outlines of Frank’s irrigator pivot coverage area, and info on the type of corn variety that Frank intends to plant in each field. Andy loads Frank’s data into an existing software package, generates a prescription map that works with Frank’s planter monitor, and gets the map to Frank before planting begins.
Use Case with Current Industry State
Figure 1: Current state of the ag data industry from a farmer's point of view. Note that some important data (in red) never makes it back to the farmer at all.
The way this use case generally occurs today begins with Frank telling Andy those dreams of having all that data aren’t going to happen. Frank then spends a painstaking amount of time simply gathering whatever information he can. Exporting yield data for every field from his desktop computer or OEM-specific cloud service, looking up past PDF files that have soil test information, zipping all these files up, and sending them to Andy. Frank doesn’t have the as-applied maps for fertilizer, because the fertilizer co-op never sent them. Frank looks back at his seed order receipts, and types up a spreadsheet with field names and seed varieties.
Andy then calls up the local seed salesman for some of the unfamiliar varieties to get seeding recommendations. Frank doesn’t have the outlines for his irrigators saved anywhere, so Andy pulls up Google Earth, draws the approximate circles, and saves them to a KML file on his personal computer. After some significant data massaging, Andy gets all this imported into his existing software, makes a seed prescription, exports a file formatted for Frank’s planter monitor, and emails it to Frank. Frank saves the file on his computer with prescriptions from other people for other fields, and eventually puts them on a USB stick to import into his planter monitor.
Use Case with OADA Interoperability
Figure 2: The farmer's clouds can work together through the common OADA REST API to enable trusted agents to manage the farmer's data for him. Should the farmer become unhappy with any given cloud provider, he can easily transfer his data to another cloud provider.
The OADA example workflow for the prescription map is one where Frank has an account with an OADA-compliant cloud provider. His precision ag data is there, his third-party applicator’s precision ag records for his fields are there, and the KML for his irrigator pivot outline is there. Some of Frank’s precision ag data first goes into a different OADA-compliant cloud due to device compatibility on one of his tractors, but Frank has configured his main cloud to just sync up those files automatically via those service’s OADA-compliant capabilities (provided by the OADA REST API framework).
Frank likes this, because he finally has a one-and-done place to keep his data, and he can switch to another cloud seamlessly someday if he has reason to do so. Frank logs in to the service and gives Andy permission to access the files he needs. The permission provisioning implementation is provided by the cloud service but must adhere to the community guidelines for privacy.
Andy’s latest version of prescription-building software has support for retrieving files from OADA-compliant data stores. The ag-based file metadata can tell it everything needed to parse the files properly: manufacturer, data type, version, shapefile columns, etc. Andy points it to the cloud, loads in the files, and builds the prescription. He exports a file for Frank’s monitor and puts it directly into Frank’s account. Frank’s new OADA-compliant telematics device has been configured to point to Frank’s cloud and retrieve prescription files directly, and upload as-applied planting maps automatically.
It is the ag-based metadata that distinguishes this approach from a standard cloud-based file store. That basic layer will allow OADA to extend and support a wide array of ag-specific features that simply are not possible with standard file storage, such as opening files directly in client software without the need to manually specify any information about them.
Technical Summary of the OADA Approach
The following sections contain technical details and are written for architects and engineers.
To address the farmer’s data interoperability and security concerns, a set of common, extensible REST API specifications will be defined. Precision ag systems that want to take advantage of this data interoperability and want to securely access farmer data will build their systems to match the OADA API specification. While a precision ag system will create an implementation of the open source OADA API specification, the implementation itself is the IP of the entity implementing the OADA API. Building an implementation that adheres to the OADA API specification in no way requires the implementation to be open sourced. The open source OADA API specification is merely a vehicle to facilitate interoperability between both open and closed precision ag systems.
The REST API specification will focus on data exchange that allows data from any precision ag platform in any format to be easily transferred between platforms. The API will not be concerned with the details of the data format of the files being exchanged beyond including information about which kind of format they are. This will allow APIs to easily evolve over time with new features and still remain backwards compatible. Combined with the industry standard OAuth 2.0 authorization protocol and successor versions, the API specification will have common data sharing semantics that ensure only farmer approved trusted agents or vendors can have access to the farmer’s shared data. The API will have operations that allow data to be shared both permanently and temporarily as the farmer sees fit.
A verification suite will be created that can automatically be run against vendor implementations of the OADA API specification. Results of the verification suite will be used to measure a platform’s compliance with the OADA API specification and ensure interoperability across the industry. Development of the above will occur openly in the community for all to see. An open source agile development approach will be used. Code will be placed in publicly available source code repositories released under the MIT License for everyone to freely contribute and use.
The community will drive the ultimate roadmap and specific features of the API specification. Through this open, community-driven effort, farmers will get precision ag platforms that are able to easily integrate despite the diversity of data formats. Support for future, yet to be defined industry data standards will be enabled. An environment that facilitates the creation of innovative services that benefit the farmer will be created by lowering the cost of entry to integrate among precision ag platforms and by having common security mechanics.
OADA will naturally facilitate a discussion in the industry of what proper privacy and security practices are for ag data. Community guidelines will be developed that third party auditors can use to evaluate varying levels of OADA compliance, and the industry as a whole can use as a common language to properly and easily relate the ramifications of complex technical details. For example, OADA “level 1” and “level 2” compliance will have a definite meaning, and the industry can begin with that definition as a means to communicate and compare business practices.
Technical Roadmap
The features described here are a starting point and will eventually be finalized through involvement of the broader OADA technical community. A first draft of the API can be found on GitHub at https://github.com/OADA/oada-api-spec/blob/master/oada-api.raml
Building a common, extensible, interoperable REST API that allows data to be stored and transformed in any format requires defining a standard mechanism from which farmer data can be referenced. To simplify integration the API will expose the data through a file system paradigm. Farmers and external systems will reference farmer data through file paths and unique file identifiers. The underlying implementation of the API does not need to store files and paths: it only needs to provide the interface. A given implementation can use a distributed file system, database, or any other storage mechanism that works best for their needs. No detailed knowledge of the files’ underlying storage format will be required to store or transfer data. The API would initially have the following key capabilities:
- Sharing that allows farmers to authorize permanent or temporary access to their data to an external precision ag system or trusted advisors through OAuth 2.0.
- File Create, Read, Update, Delete - operations to read, write, and delete farmer data files
- File Transformation - transformation operations to convert data between different precision ag data formats. It would be up to each OADA API implementation to report what format transformations it supports. A basic set of transformation drivers for known public data formats (shape, GeoJSON, KML, etc.) will be developed, and contributions from industry participants who want to provide drivers to their own file formats will be accepted.
- File Metadata - the ability to associate metadata with the data file. Some lightweight OADA-specific metadata about file type and manufacturer will be required if available, but custom metadata will also be supported to enable individual implementations to add features as they see fit.
- System Features – A set of operations to communicate which data formats and transformations the OADA API implementation supports.
An early draft of the API specification using the API specification language RAML can be found on Github
Potential future API capabilities will include searching, synchronization, alerting, and streaming:
- Alerting of events such as the arrival of new data so that a farmer’s prescription can be updated with the most recent data.
- Synchronization among clouds.
- Streaming to support capabilities such as writing and retrieving data in chunks as needed.
- Search functionality would range from basic searches across metadata to spatial queries to identify data layers that fall with in a field boundary. Searching will support use cases that require identifying key data layers that need to be transferred from a pure data management provider to an ag analytics provider that is providing prescription services.
A reference implementation of the API specification in a yet to be identified programming language will be developed and maintained by the OADA technical community and hosted on GitHub.
Accompanying the reference implementation will be a verification suite. The verification suite will contain a set of functional tests to validate each feature of the API. The verification suite will be automated and can be run on a repeated basis to measure the compliance of any implementation with the OADA API specification.
Other potential community driven technical artifacts include transformation libraries that convert data between the different proprietary industry data formats and existing open standard formats such as shape, GeoJSON, etc.
Project Structure
Please refer to the Developer Guidelines for information about contributing to the project and the process by which code will be approved for inclusion in the project.