Export formats - JSON

[Wookey]

On 2016-06-29 18:19 -0400, Philip Balister wrote:
> 
> So I think we are missing the underlying reason Philip is interested in
> a standard format for storing survey data.
> 
> Whenever someone comes along and says "I can make a better cave survey
> program", they start by reinventing wheels. Now, survex has pretty much
> solved the data processing problem and is used by cavewhere and therion.
> (on my wishlist, have subsurface export an API so people do not take the
> code and hack at it)
> 
> But we still have the problems of data from things like smaps, compass,
> walls etc. (I'm assuming everyone on this list understands the evils of
> binary only programs)
> 
> What need is to start moving to a standard extensible format that is
> easily usable without everyone having to write yet another parser for
> processing cave data. 

This has been desirable for a very long time. Every attempt to create such
a format so far has failed to achieve significant adoption, and there
have been quite a few since I took an interest in this subject:
* HTO: http://www.chaos.org.uk/survex/cp/CP02/CPoint02.htm#Art6
* Rosetta/CDI: http://www.resurgentsoftware.com/rosettastal.htm 
* Cavescript: http://www.speleonics.com.au/cavescript/
* caveXML: http://www.cavexml.uis-speleo.org/
* and now cavemeta: 

Did I miss any? (I think there was a toporobot-based effort too?)
Notably most of the people involved with those that are still alive
are on this list :-)

The only formats with any longevity are the native ones of programs
and despite its unsatisfying nature, writing tools like caveconverter
and Rosettastal to convert between them has worked better than efforts
to define an interchange format.

Now that doesn't stop this being a nice idea, and maybe its time has
finally come, but history is not on your side here :-)

What Martin and Graham say about the survex and therion formats is
true, but importantly, it is very hard to parse those data formats
without using the survex/therion parsers themselves, and those only
come and C and C++ respectively, and not as libraries. Re-implementing
those parsers in other languages (as Tunnel has done in java and
Troggle has done in Python and caveconverter and rosettastal must have
done too) is error-prone, tends to only implement the commonly-used
parts of the spec, and goes wrong every time Survex or Therion add new
features.

It has been clear to me (and Ol) for a long time that having a
separate survex parser library that supports C, C++, python and java
(at least) would be a very useful contribution to the general problem
for survey data interchange. However no-one has got round to doing
that work. SWIG is a good way of providing the one true, tested
C-library, with APIs for python, perl, ruby, java, C#, javascript, etc.

If anyone got enthused about actually doing that it would be a service
to the world.

> If we can get this format in place and hopefully
> adopted, people can process data with the program that makes them happy,
> without annoying other people who like other programs.

I've not looked at cavemeta yet, to see what might be missing from it,
but I think you should take a long hard look at all the other attempts
in this area and try to understand why they were not adopted. 
 
> Ideally, someone would write something translate svx files into metacave
> for people that like the survex format. Possibly even build this into
> survex, since it already knows how to parse svx, and then use a library
> to write out the data as metacave.

Adding this functionality to caveconverter would make sense. RosettaStal
is nice but Windows-only and thus not very general (totally useless to
me for example)
 
> Now that we have a standard format to store data in, people interested
> in processing it do not need to sit down and come up with a storage
> format when then have some crazy idea to do with cave data.

New 'standard' formats for storing survey data in is not the
problem...conversion (and fidelity of that conversion) and adoption is.

Wookey
-- 
Principal hats:  Linaro, Debian, Wookware, ARM
http://wookware.org/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.survex.com/pipermail/survex/attachments/20160712/85f3d1a5/attachment.sig>