Digital Geotechnical have just launched a set of web apps for carrying out some simple tasks with AGS data, such as:
- Obtaining a quick overview of the contents of an AGS file without having to import it to a database or Excel, or wade through a text file
- Check/validate an AGS file using the ‘official’ AGS validator (beta)
- Editing out unwanted parts of an AGS file, or splitting up a large file into more manageable chunks
Interested? Then please check them out on our website.
If you would like to know more about what they are, some background, nerdy technical info and future plans, then please read on…
What are they and what do they do?
Digital Geotechnical Ltd is my business name, just in case you were wondering. I have done all the coding so I will use the first-person singular from now on.
The apps just released (in February 2022) are as follows:
- DG AGS4 validation tool checks whether an AGS file conforms with the AGS format rules and data dictionary. This is a front end to the ‘official’ AGS validation tool created and maintained by the AGS validator open source project, which I will talk more about below.
- DG AGS4 summary tool provides a quick summary of the contents of an AGS file. Includes a list of all groups with a count of the records in each, a summary of exploratory holes (LOCA) by type and a list of all holes.
- DG AGS4 reducer tool starts off the same as the summary tool, but you can then select groups and/or holes to be deleted, reducing the size of the file. With a little bit of imagination you could use this to split a large into more manageable chunks.
- DG AGS4 record count by LOCA tool also provides a summary but this time presented as a count of records in each Group for each hole. This could be useful for checking whether the file content is broadly as expected.
The apps are easy to use and follow a similar pattern where you select a file to process and the results are then presented on screen, with an option to download a copy of the results, or the output AGS file in the case of the reducer. I hope that you like the output format. I’m particularly proud of the sticky row and column headers on the record count by LOCA tool! All feedback, good or bad, is welcome.
AGS validator open source project
The DG AGS4 validation tool app is a front end to the ‘official’ AGS validation tool created and maintained by the AGS validator open source project (useful links given in the box below).
Commercial software for checking/validating AGS data has been around for many years. Unfortunately, the software belonging to different vendors often gives different results. As a result, specifications normally ask for checks to be run using more than one tool. This is far from ideal. This project aims to sort out this mess by creating a definitive AGS validation tool. By making it open source AGS aims to encourage community involvement and buy in.
The source code for the validator is written in Python and a Python library is available for use in your own python scripts. In addition, a Desktop App (executable file), which is also part of the open source project, is available. Links for both are given in the box below.
If you do try using the validator, there are a few things that you need to be aware of:
- During testing, it has been found that the AGS validator is more rigorous than the existing commercial software on some of the rule checks, leading to errors being identified on files that were previously thought to be ‘clean’.
- The validator only considers the AGS format rules. It does not do any of the additional ‘FYI’ checks that are included in some commercial software. The possibility of adding such checks is on the radar, but the question of what additional checks should and should not be done is a bit of a rabbit hole. No decision has been made yet.
- Only AGS4 files are supported. There are no current plans to support (legacy) AGS3 files.
Whilst this is now an open source project managed by the AGS Data Management Working Group, most of the credit must go to Asitha Senanayake of Fugro, who had already written most of the code before bringing it to the attention of AGS. Big shout-outs also go to Roger Chandler (Bentley) and Tony Daly (Amageo) who have helped to develop the project to where it is today. I am involved with the project but my contributions to date have been modest. I hope to be able to contribute more going forward.
A reminder that the AGS validator tool is still in the beta stage (at time of writing, February 2022). AGS state that this version must not be used in or specified for use in a production environment.
AGS validator home page (includes link for downloading Desktop App):
https://www.ags.org.uk/data-format/ags-validator/
AGS news article (good background, but slightly out of date as a new version has since been released): https://www.ags.org.uk/2021/09/new-ags-validator-launched-by-ags/
Project Gitlab page (Python library):
https://gitlab.com/ags-data-format-wg/ags-python-library
Python package (on PyPi): https://pypi.org/project/python-ags4/
Background and technical details
My original motivation for creating these apps was to learn about the relevant technologies, and then try them out. After having some early success, I then realised that sharing them with the world may be a good way of bringing some attention to my business. If you have read this far then that seems to be working!
The catalyst for getting started was the AGS validator project. I saw an opportunity to provide an application that people may be genuinely interested in. It also meant that I could provide some useful feedback to that project.
The other apps were a natural follow on, providing functionality that I have found useful in the past.
For the technical nerds out there, here is a summary of what is going on under the bonnet:
- Hosted on Google App Engine (Standard Environment)
- Backend (server side) code written in Python, making use of the Flask web framework and Jinja templates
- Bootstrap used for HTML formatting and widgets
- Client side (browser) code generally limited to Bootstrap Javascript
Conditions and limitations
The apps are primarily intended as technology demonstrators. There are a few rough edges and possibly some bugs still to be discovered.
You can use them commercially* if you wish, but at your own risk. They are provided ‘as is’. However, if you like what you see then please get in touch. There are several commercial options that could be explored.
I have put limits on the upload file sizes. This has been driven by several factors including constraints arising from Google App Engine (used to host the apps), the desire to keep run times sensible and to discourage commercial usage on major projects (they can afford to pay!) If the limits are a problem, then please contact us as there are several options that we could discuss.
* AGS currently prohibit use of the AGS validator for production (it is still in beta)
Future plans
I am well aware that there are some things that could be done better. A progress bar or spinner whilst you wait would be good start. However, I figured that it would be better to these apps out as they are, and see what the reaction is before getting carried away with refinements
I have plenty of ideas for enhancements and new apps. Again, I want to see what the reaction is before going further. I may decide to hold back major enhancements for a commercial version, should there ever be demand for such a thing.
For example, one application am I playing around with right now is an AGS3 to AGS4 converter. If you think you may be interested in this then please let me know.
I will now tease you by saying that I have started work on what I consider to be a ‘major’ app that I believe will be of much wider interest and application. All I will say is that it aims to address a particular bugbear of mine. Watch this space!
If you have any comments on what I have done, or suggestions for the future, then feel free to either use the blog comments below, or alternatively contact me via the contact form on my website.
