Introducing the AGS data transfer format

The AGS data transfer format is a well-established and well-used method of sharing or transferring geotechnical and geoenvironmental data. In this article I will explain what it is, where it comes from, what it is used for and how it works. This is primarily intended for newcomers to the AGS format, but experienced users of AGS data may also benefit, especially in relation to the training courses I mention towards the end.

What is the AGS format?

The AGS data transfer format is a standard file format for the transfer of geotechnical and geoenvironmental data. Its scope is generally limited to factual data from ground investigations (GI), i.e. what is typically included in the logs and test reports that make up a GI factual report. However, it is also capable of handling certain types of construction monitoring data.

‘AGS’ stands for the Association of Geotechnical and Geoenvironmental Specialists, a not-for-profit trade association operating in the United Kingdom for the benefit of companies working in the ground engineering sector.

The AGS format is well-established having been around for over 30 years, pre-dating the recent advances in BIM and digital engineering by some distance. It is routinely used in the UK on projects of all shapes and sizes. It has also been adopted by some other countries, including Australia, Hong Kong, Singapore and New Zealand.

It was created by the industry, for the industry, and is widely regarded. It is considered by many to be the world’s most successful transfer format for ground data.

Who uses it, and for what?

The main use of the AGS format is the transfer of data associated with a GI factual report from GI contractor to design consultant. The consultant will then typically import the data into their own system, typically specialist geotechnical database software (discussed below), which they then use to create outputs to inform their interpretation and design. Being able to import the data saves having to manually key it in from hard copy reports. As well as saving a lot of time, this also helps to reduce re-keying errors. This is all summed up nicely by an oft-repeated quote that has been around many years…

Input data only once – and preferably get someone else to do it!

However, GI contractors and consultants are not the only people using AGS data, and nor should they be. For example, AGS data is often used:

• Within the GI contractor’s supply chain for the collation of source data from field logging or laboratories.
• By other designers or contractors in addition to the design consultant originally receiving the data.
• By clients, as part of their asset management system.
• By the British Geological Survey (BGS), who now accept AGS data in addition to conventional reports, and make this data available to interested third parties.

The diagram below illustrates some of the above along with the potential data flows.

How does it work?

The AGS format essentially comprises two elements:

• The data structure, i.e. how the data must be organised.
• The file format, i.e. how to convert the structured data into an .ags file deliverable.

The above are defined and underpinned by a set of ‘Rules’.

All the above are described in the documentation published on the AGS website. The documentation does not have the status of a formal BS or ISO standard, but this is by choice as AGS prefer to retain full control over the format and its documentation. This also means that it is able to make it available to the industry for free. This has helped the AGS format to become the de facto standard for ground data transfer in the UK.

The basic concepts behind the data structure are covered by the Rules but full details are provided in the Data Dictionary part of the documentation (example shown below).

AGS data is arranged into pre-defined Groups which are divided up by subject area, such as:

• LOCA, for defining each exploratory hole including location and metadata.
• SAMP, for data associated with the samples and sampling.
• Groups for each type of in situ test.
• Groups for each type of laboratory test.

The Groups are divided up into Fields, alternatively known as Headings (technically a Heading is the name of a Field). The Groups are organised into a logical hierarchical structure with the relationships between them formed using the concept of Key Fields, i.e. fields common to both ‘parent’ and ‘child’ Groups.

This structure reflects that found in many relational databases (Groups = Tables and Fields/Headings = Fields/Columns) although the AGS format is not intended to be a specification for a database.

As of AGS version 4.1.1 (current at time of writing) there were 148 Groups comprising a total of 2895 Headings. The AGS format is big!

The file format is text-based, which means that any .ags file can be opened and edited in a normal text editor. The text file comprises blocks of CSV-like¹ data. Each block corresponds to the data for a Group and comprises a set of header rows followed by rows of data, as shown in the small snippet below.

1. CSV = comma separated variable (file format) ↩︎

Whilst the Data Dictionary is very comprehensive it is also possible to add custom or user-defined Groups and/or Fields, provided this is done using the methodology prescribed in the Rules. However, such customisation should be used with care and only when necessary because it can cause problems for both data producers and downstream data users.

AGS data normally comprises a single file carrying all data for a GI project, but it is also possible to provide other files alongside the data, e.g. photographs and drawings, using the concept of Associated Files as described in the Rules.

Can anyone use it?

The main AGS format document published on the AGS website can be freely accessed by anyone, anywhere at any time. There is no fee to pay and no login is required.

However, there are some additional resources and documents on the website, including standard code lists and guidance, that can only be accessed by ‘registered’ users of the data format.

In addition, there is a requirement (within the document – section 1.3) that organisations that “submit or receive” AGS data shall register with AGS, and there is a small annual fee to pay for this. Registration also applies to software vendors (or service providers) working with AGS data.

For organisations that are already members of AGS this fee is normally bundled with the main AGS membership fee. For non-AGS members, including overseas users, it is possible to register without joining the AGS itself. Refer to the AGS website for details.

The intent of the fee, which is modest, is to generate some income for AGS so that it can continue to maintain and develop the format. It is intended to target those that are actively using AGS data, not organisations that are only passing AGS data on to others.

How to use it

Heavy users of AGS data typically use commercial geotechnical database software. GI contractors use it for input and assembly of data from fieldwork and laboratory testing and subsequent production of both the report and AGS data. Design consultants use it to ingest AGS data, sometimes merging it with data from other sources, followed by output of data plots and drawings to inform interpretation, reporting and subsequent design.

Some users, typically those operating at the smaller end of the market, may use spreadsheets as their main tool for handling AGS data. This is possible, but it is not ideal as there are several potential pitfalls to work around. However, the rising popularity of scripting tools such as Python is encouraging new ways of working, such as scripting used in conjunction with spreadsheets which can potentially address some of the problems.

Validation of AGS data, i.e. a check to see whether an AGS file complies with the Rules and Data Dictionary, is a vital part of the process. Historically, validation has been carried out using commercial software. However, there has been a recent move to establish a ‘single source of truth’ validation tool, managed by the AGS. This is known as the AGS Validator and it is available as a Python library, desktop application or web app (e.g. BGS or Digital Geotechnical). It also includes a function that allows AGS data to be converted to Excel, and vice versa.

The topic of software and tools used for handling AGS data, and geotechnical data generally, is one that is close to my heart. I plan to take a closer look at this is in future blogs (spoiler alert: I have a special interest in one of the software solutions – a new entry to the UK market!)

Further education and training

This article is a brief high-level introduction to AGS data. If you wish to find out more then the obvious starting place is the AGS documentation itself. However, whilst the AGS documentation is relatively readable and understandable compared to many other data standards, you may still find yourself struggling a little.

Don’t worry – help is at hand in the form of a pair of training courses dedicated to the topic of AGS data. These are commercially-run courses so they are not free of charge, but in my opinion they are well worth paying for. Then again, I would say that – because I am the curator and presenter of these courses!

Introducing the AGS data format

This is an entry level introduction to the AGS data format, covering background and context before going on to describe the data structure and file format in detail. Specification and validation of AGS data are also introduced. The AGS format documentation is fully referenced throughout.

This is an online on-demand self-guided course. The content is divided up into modules each comprising a short presentation video along with suggestions for further reading and study. There are also some quiz questions to test your developing knowledge of the subject. The total length of the course should be 2-3 hours, depending on how much additional reading/study is undertaken.

Mastering the AGS data format

This provides a deeper dive into the AGS data transfer format. The course starts with a recap on fundamentals such as the data structure and file format rules, focussing on some of the less well-known aspects. It then goes on to study specific areas of the AGS data schema, identifying some of the common problems that can arise and strategies that can be adopted to avoid them. Specification and validation of AGS data are also covered.

This is held as a one-day trainer-led course, with both classroom and online sessions available. Some prior knowledge and experience of the AGS format is beneficial, but this does not need to be extensive. Alternatively, this course is a natural follow on from the ‘Introducing’ course.

Both of the above courses are available via Equipe Training – see the Equipe website for further details, costs and booking. The classroom courses are generally held at the Equipe offices near Banbury. However, we can also provide bespoke ‘on site’ training where we can bring the course to your own organisation, should you have enough attendees to justify it.