Data Formats Working Group: Difference between revisions

From canSAS
(add the current working definition)
 
(81 intermediate revisions by 7 users not shown)
Line 1: Line 1:
[http://smallangles.net/pipermail/cansas-1dwg_smallangles.net/ Mailing List Archive]


===Timeline===
= Overview =
* 2007-12-31 agree on v1.0 format
 
* 2008-01-01 start implementing v1 at facilities
As of early 2017, for new work, it is recommended to use the NXcanSAS (multi-dimensional)
* 2008-06 representative sampling of data available for inter-facility comparison
standard for both 1-D and n-D data to maximize the chance that other
* 2008-10 presentation of results at NOBUGS2008 meeting (date TBA)
software will be able to read your data files.
 
We strongly encourage all authors of small-angle scattering related software to add support for NXcanSAS.
 
= Latest News =
 
[[User:Smk|Steve King]]: 2024-09 : Proposals for NXcanSAS 1.1 in development
 
[[User:Butler|Paul Butler]]: 2019-07 : Lots of discussion at [[canSAS-XI]] about metadata. Discussion and actions from the meeting are available [[canSAS-XI/DataFormats]]


=  Links =


=Current Definition=
=== nD: NXcanSAS ===
==XML Schema==
to be defined


==XML template==
standard for reduced small-angle scattering data of any dimension, using NeXus (HDF5)
<pre>
<SASroot version="0.1a">
  <SASentry>
    <Title></Title>
    <Run></Run>
    <SASdata>
      <Idata><Q></Q><I></I><Idev></Idev><Qdev></Qdev></Idata>
    </SASdata>
    <SASsample></SASsample>
    <SASinstrument>
      <SASsource></SASsource>
      <SAScollimation></SAScollimation>
      <SASdetector></SASdetector>
    </SASinstrument>
    <SASprocess></SASprocess>
    <SASnote></SASnote>
  </SASentry>
</SASroot>
</pre>


==XML example==
* '''v1.1''': Proposals in development
* '''v1.0''': 2017-01-18
* '''Announced''': 2017-06-06 (at [[canSAS-IX]])
* '''Documentation''': [http://download.nexusformat.org/doc/html/classes/applications/NXcanSAS.html http://download.nexusformat.org/doc/html/classes/applications/NXcanSAS.html]
* '''Examples''': [https://github.com/canSAS-org/NXcanSAS_examples https://github.com/canSAS-org/NXcanSAS_examples]
* '''GitHub (NeXus)''': [https://github.com/nexusformat/definitions https://github.com/nexusformat/definitions]
* '''Poster''': [[Media:2017-05-09_NXcanSAS_2D_format.pdf|PDF]]
* '''CanSAS 2012 Discussion''': 2012-07-31 [[2012_Data_Discussion|Data formats discussion page]]


=Discussion=
=== 1D: cansas1d ===
===Considerations===
* a key point of what we discussed at NIST: 
namely that our goal is to agree a format which that whilst using as much best XML practice as is reasonable, leaves the file instantly human-readable, editable in the simplest of editors, and importable by simple text import filters in programs that don't recognise the XML.
* document what we decide
**1DWG will take care of documenting the format it defines.
*** make that definition with a schema (for absolute validation of any proposed XML file against the standard)
*** instructions on how to use that schema
*** XSL style sheets to present the XML contents in various forms (also serves as examples)
*** a couple of examples
*** maybe also some words.
** move some of this discussion to
*** discussion page
*** other wiki pages
*** /dev/null after its usefulness has been exhausted
* coordinate with other communities
** [http://www.nexusformat.org NeXus (http://www.nexusformat.org)]
** reflectivity
** powder diffraction
*** [http://www.xrdml.com XRDML (http://www.xrdml.com)]
* should we consider a file naming convention?
* should we consider a SAS scan naming convention?
** sequential run number from facility
** convention set by the detector software provider
* XML representation of the I vs. Q data
** tabular format
** vector format
* general XML coding style
** readability by humans
*** with lots of computer skills
*** with rudimentary computer skills
** readability by computers
*** standard XML libraries
*** generic visualization tools
*** common software such as MS Excel  or Open Office  or [ftp://ftp.ill.fr/pub/cs/rxml XMLPLO source code for windows/linux/OSX86]
*** A plugin for IGOR has been written, [http://www.igorexchange.com/project/XMLutils XMLutils.xop], that can handle XML data.  The [[sasXML and IGOR]] page gives details on the IGOR code required to write the prototype sasXML file.
** availability of style sheets
* scalability of XML format to 2D data?
* What is required?
* What is optional?
* Use the same tags again in similar contexts
** X,Y pairs for example, whether detector position, beam center, sample position
{|  width="100%" border="1" style="border:1px solid navy; border-collapse: collapse"
|-
! width="50%" | inconsistent
! width="50%" | consistent
|-
| <pre>
<beam_size axis="x" units="mm">12.00</beam_size>
<beam_size axis="y" units="mm">12.00</beam_size>
<x0 units="mm">322.64</x0>
<y0 units="mm">327.68</y0>
<pixel_x units="mm">5.00</pixel_x>
<pixel_y units="mm">5.00</pixel_y>
</pre>
| <pre>
<beam_size axis="x" units="mm">12.00</beam_size>
<beam_size axis="y" units="mm">12.00</beam_size>
<beam_center axis="x" units="mm">322.64</beam_center>
<beam_center axis="y" units="mm">327.68</beam_center>
<pixel_size  axis="x" units="mm">5.00</pixel_size>
<pixel_size  axis="y" units="mm">5.00</pixel_size>
</pre>
|}


===Points for Discussion===
standard for 1-D reduced small-angle scattering data, using XML (text)


*Do we want to advocate/recommend particular names for particular tags; eg, SASdata, SASsample, Idata, etc.?
* v1.1: 2013-03-29
** which ones?
* v1.0: 2009-05-12
* provide for (optional) inclusion of sample prep details
* provide for (optional) inclusion of other (non-SAS) data in the XML
* Need to allow for more than a single SAS data set in one .xml file


===Other Points===
* '''Documentation''': [http://www.cansas.org/formats/canSAS1d/1.1/doc/ http://www.cansas.org/formats/canSAS1d/1.1/doc/]
* It's not clear how to specify that multiple runs were reduced together
* '''Examples''':      [https://github.com/canSAS-org/1dwg/tree/master/examples https://github.com/canSAS-org/1dwg/tree/master/examples]
**(AJJ) Assuming that those multiple runs were first stored as XML then referencing the individual files would give all that back information (a la Ghosh suggestion). At NIST we take absolute I vs Q files and combine them to produce an absolute I vs Q file thus that is reasonable here. What about elsewhere?
* '''GitHub''':      [https://github.com/canSAS-org/1dwg  https://github.com/canSAS-org/1dwg]
* How does one include the instrument information of the many runs that we used to make up the composite file
* If we have reduction information, then everything needs to be in there, i.e. the run numbers for the can, the standard, the uniform field, etc.
* Information on the averaging, is it radial, sector, rectangular, etc.


=Members=
= Members =
* Andrew Jackson (NIST)
* Andrew Jackson (ESS)- Chair
* Jeff Krzywon (NIST)
* Pete Jemian (APS)
* Pete Jemian (APS)
* Steve King (ISIS)
* Steve King (ISIS)
* Ken Littrell (ORNL)
* Andrew Nelson (ANSTO)
* Andy Nelson (ANSTO)
* Ron Ghosh (ILL)
* Jan Ilavsky (APS)
* Jan Ilavsky (APS)
* Lisa Debeer-Schmitt (ORNL)


=News/Status=
= Timeline =
--[[User:Jemian|Jemian]] 12:00, 14 December 2007 (EST)draft status updated
* 2007-12-31 agree on v1.0 format
* 2008-01-01 start implementing v1 at facilities
* 2008-06    representative sampling of data available for inter-facility comparison
* 2008-11-3  presentation of results at NOBUGS2008 meeting ([http://www.nbi.ansto.gov.au/nobugs2008 http://www.nbi.ansto.gov.au/nobugs2008])
* 2009-05-12 tagged SVN changeset 67 as v1.0 since it has been unmodified since January
* 2009-09-17 agreed that v1.1 revision should be forthcoming with corrections/additions to be identified in TRAC tickets
* 2017-01    NXcanSAS 1.0 released
* 2015-12    multidim moved to github a while ago
 
= Status =
[[User:Jemian|Jemian]] 11:10, 06 September 2008 (CDT) IgorPro GUI tool included in Irena [http://usaxs.xor.aps.anl.gov/staff/ilavsky/irena.html Irena] analysis suite<br />
[[User:Jemian|Jemian]] 13:08, 14 March 2008 (EDT) IgorPro tool to read cansas1d/1.0 XML files available<br />
[[User:Jemian|Jemian]] 13:38, 25 February 2008 (EST) starting to add case studies <br />
[[User:Jemian|Jemian]] 00:39, 21 December 2007 (EST) complete first draft (v0.1a) ready for comments from 1DWG <br />
[[User:Jemian|Jemian]] 12:00, 14 December 2007 (EST) draft status updated <br />
[[User:Jemian|Jemian]] 17:07, 12 May 2009 (EDT) changeset 67 tagged as v1.0
 
= Examples and Usage =
The following are example v1.1 compliant 1D data files containing [besides essential metadata] - right-click & save as to view, stylesheet is on the TRAC site:<br /><br />
SANS data only [[media:ISIS_SASXML_v1_1_SANS_Example.XML | ISIS_SASXML_v1_1_SANS_Example.XML]]<br />
Wavelength-dependent transmission [[media:ISIS_SASXML_v1_1_Transmission_Example.XML | ISIS_SASXML_v1_1_Transmission_Example.XML]]<br />
Wavelength-dependent monitor spectrum [[media:ISIS_SASXML_v1_1_Monitor_Spectrum_Example.XML | ISIS_SASXML_v1_1_Monitor_Spectrum_Example.XML]]<br />
Wavelength-dependent detector efficiency ratio [[media:ISIS_SASXML_v1_1_Det_Efficiency_Example.XML | ISIS_SASXML_v1_1_Det_Efficency_Example.XML]]<br />


The files can be usefully displayed in many web browsers: [[Rendering_canSAS1D_in_Web_Browsers]]


[[Category:Working Groups]]
[[Category:Working Groups]]
= Mailing List=
The mailing list is now maintained by subscribing to this Google group:
* [https://groups.google.com/forum/#!forum/cansas-dfwg https://groups.google.com/forum/#!forum/cansas-dfwg]
Please address all questions and answers to this Google group.

Latest revision as of 11:41, 25 October 2024

Overview

As of early 2017, for new work, it is recommended to use the NXcanSAS (multi-dimensional) standard for both 1-D and n-D data to maximize the chance that other software will be able to read your data files.

We strongly encourage all authors of small-angle scattering related software to add support for NXcanSAS.

Latest News

Steve King: 2024-09 : Proposals for NXcanSAS 1.1 in development

Paul Butler: 2019-07 : Lots of discussion at canSAS-XI about metadata. Discussion and actions from the meeting are available canSAS-XI/DataFormats

Links

nD: NXcanSAS

standard for reduced small-angle scattering data of any dimension, using NeXus (HDF5)

1D: cansas1d

standard for 1-D reduced small-angle scattering data, using XML (text)

  • v1.1: 2013-03-29
  • v1.0: 2009-05-12

Members

  • Andrew Jackson (ESS)- Chair
  • Jeff Krzywon (NIST)
  • Pete Jemian (APS)
  • Steve King (ISIS)
  • Andrew Nelson (ANSTO)
  • Jan Ilavsky (APS)
  • Lisa Debeer-Schmitt (ORNL)

Timeline

  • 2007-12-31 agree on v1.0 format
  • 2008-01-01 start implementing v1 at facilities
  • 2008-06 representative sampling of data available for inter-facility comparison
  • 2008-11-3 presentation of results at NOBUGS2008 meeting (http://www.nbi.ansto.gov.au/nobugs2008)
  • 2009-05-12 tagged SVN changeset 67 as v1.0 since it has been unmodified since January
  • 2009-09-17 agreed that v1.1 revision should be forthcoming with corrections/additions to be identified in TRAC tickets
  • 2017-01 NXcanSAS 1.0 released
  • 2015-12 multidim moved to github a while ago

Status

Jemian 11:10, 06 September 2008 (CDT) IgorPro GUI tool included in Irena Irena analysis suite
Jemian 13:08, 14 March 2008 (EDT) IgorPro tool to read cansas1d/1.0 XML files available
Jemian 13:38, 25 February 2008 (EST) starting to add case studies
Jemian 00:39, 21 December 2007 (EST) complete first draft (v0.1a) ready for comments from 1DWG
Jemian 12:00, 14 December 2007 (EST) draft status updated
Jemian 17:07, 12 May 2009 (EDT) changeset 67 tagged as v1.0

Examples and Usage

The following are example v1.1 compliant 1D data files containing [besides essential metadata] - right-click & save as to view, stylesheet is on the TRAC site:

SANS data only ISIS_SASXML_v1_1_SANS_Example.XML
Wavelength-dependent transmission ISIS_SASXML_v1_1_Transmission_Example.XML
Wavelength-dependent monitor spectrum ISIS_SASXML_v1_1_Monitor_Spectrum_Example.XML
Wavelength-dependent detector efficiency ratio ISIS_SASXML_v1_1_Det_Efficency_Example.XML

The files can be usefully displayed in many web browsers: Rendering_canSAS1D_in_Web_Browsers

Mailing List

The mailing list is now maintained by subscribing to this Google group:

Please address all questions and answers to this Google group.

Retrieved from "https://wiki.cansas.org/index.php?title=Data_Formats_Working_Group&oldid=2681"