Data Formats Working Group: Difference between revisions

From canSAS
No edit summary
 
(78 intermediate revisions by 7 users not shown)
Line 1: Line 1:
[http://smallangles.net/pipermail/cansas-1dwg_smallangles.net/ Mailing List Archive]


===Timeline===
= Overview =
* 2007-12-31 agree on v1.0 format
 
* 2008-01-01 start implementing v1 at facilities
As of early 2017, for new work, it is recommended to use the NXcanSAS (multi-dimensional)
* 2008-06 representative sampling of data available for inter-facility comparison
standard for both 1-D and n-D data to maximize the chance that other
* 2008-10 presentation of results at NOBUGS2008 meeting (date TBA)
software will be able to read your data files.
 
We strongly encourage all authors of small-angle scattering related software to add support for NXcanSAS.


= Latest News =


=Current Definition=
[[User:Smk|Steve King]]: 2024-09 : Proposals for [[NXcanSAS v1.1|NXcanSAS 1.1]] in development
==XML template: cansas1d.xml==
<pre>
<?xml version="1.0"?>
<SASroot xmlns="http://www.smallangles.net/cansas1d"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.smallangles.net/cansas1d cansas1d.xsd"
version="0.1a">
<SASentry>
<Title></Title>
<Run></Run>
<SASdata>
<Idata>
<Q units="1/A"></Q>
<I units="1/cm"></I>
<Qdev units="1/A"></Qdev>
<Idev units="1/cm"></Idev>
<Qfwhm units="1/A"><!-- Qfwhm is optional --></Qfwhm>
<Qmean units="1/A"><!-- Qmean is optional --></Qmean>
<ShadowFactor units="1/A"><!-- ShadowFactor is optional --></ShadowFactor>
</Idata>
</SASdata>
<SASsample></SASsample>
<SASinstrument>
<SASsource></SASsource>
<SAScollimation></SAScollimation>
<SASdetector></SASdetector>
</SASinstrument>
<SASprocess></SASprocess>
<SASnote></SASnote>
</SASentry>
</SASroot>
</pre>


==XML Schema: cansas1d.xsd==
[[User:Butler|Paul Butler]]: 2019-07 : Lots of discussion at [[canSAS-XI]] about metadata. Discussion and actions from the meeting are available [[canSAS-XI/DataFormats]]
<pre>
<?xml version="1.0" encoding="UTF-8"?>
<schema xmlns="http://www.w3.org/2001/XMLSchema"
targetNamespace="http://www.smallangles.net/cansas1d"
xmlns:tns="http://www.smallangles.net/cansas1d"
elementFormDefault="qualified">


<complexType name="SASrootType">
= Links =
<sequence>
<element name="SASentry" minOccurs="1" maxOccurs="unbounded"
type="tns:SASentryType">
</element>
</sequence>
<attribute name="version" type="string" />
</complexType>


<complexType name="SASentryType">
=== nD: NXcanSAS ===
<sequence>
<element name="Title" minOccurs="0" maxOccurs="1"
type="string">
</element>
<element name="Run" minOccurs="0" maxOccurs="1"
type="string">
</element>
<element name="SASdata" type="tns:SASdataType"></element>
<element name="SASsample" />
<element name="SASinstrument" />
<element name="SASprocess" />
<element name="SASnote" />
</sequence>
<attribute name="name" type="string" />
</complexType>


<complexType name="SASdataType">
standard for reduced small-angle scattering data of any dimension, using NeXus (HDF5)
<sequence>
<element name="Idata" minOccurs="1" maxOccurs="unbounded"
type="tns:IdataType" />
</sequence>
<attribute name="name" type="string" />
</complexType>


<complexType name="IdataType">
* '''v1.1''': Proposals in development
<sequence>
* '''v1.0''': 2017-01-18
<element name="Q" minOccurs="1" maxOccurs="1"
* '''Announced''': 2017-06-06 (at [[canSAS-IX]])
type="float">
* '''Documentation''': [http://download.nexusformat.org/doc/html/classes/applications/NXcanSAS.html http://download.nexusformat.org/doc/html/classes/applications/NXcanSAS.html]
</element>
* '''Examples''': [https://github.com/canSAS-org/NXcanSAS_examples https://github.com/canSAS-org/NXcanSAS_examples]
<element name="I" minOccurs="1" maxOccurs="1"
* '''GitHub (NeXus)''': [https://github.com/nexusformat/definitions https://github.com/nexusformat/definitions]
type="float">
* '''Poster''': [[Media:2017-05-09_NXcanSAS_2D_format.pdf|PDF]]
</element>
* '''CanSAS 2012 Discussion''': 2012-07-31 [[2012_Data_Discussion|Data formats discussion page]]
<element name="Qdev" minOccurs="0" maxOccurs="1"
type="float">
</element>
<element name="Idev" minOccurs="0" maxOccurs="1"
type="float">
</element>
<element name="Qfwhm" minOccurs="0" maxOccurs="1"
type="float">
</element>
<element name="Qmean" minOccurs="0" maxOccurs="1"
type="float">
</element>
<element name="ShadowFactor" minOccurs="0" maxOccurs="1"
type="float">
</element>
</sequence>
</complexType>


=== 1D: cansas1d ===


<element name="SASroot" type="tns:SASrootType" />
standard for 1-D reduced small-angle scattering data, using XML (text)


</schema>
* v1.1: 2013-03-29
</pre>
* v1.0: 2009-05-12


==XML example==
* '''Documentation''': [http://www.cansas.org/formats/canSAS1d/1.1/doc/ http://www.cansas.org/formats/canSAS1d/1.1/doc/]
to be defined
* '''Examples''':      [https://github.com/canSAS-org/1dwg/tree/master/examples https://github.com/canSAS-org/1dwg/tree/master/examples]
* '''GitHub''':      [https://github.com/canSAS-org/1dwg  https://github.com/canSAS-org/1dwg]


=Members=
= Members =
* Andrew Jackson (NIST)
* Andrew Jackson (ESS)- Chair
* Jeff Krzywon (NIST)
* Pete Jemian (APS)
* Pete Jemian (APS)
* Steve King (ISIS)
* Steve King (ISIS)
* Ken Littrell (ORNL)
* Andrew Nelson (ANSTO)
* Andy Nelson (ANSTO)
* Ron Ghosh (ILL)
* Jan Ilavsky (APS)
* Jan Ilavsky (APS)
* Lisa Debeer-Schmitt (ORNL)


=News/Status=
= Timeline =
--[[User:Jemian|Jemian]] 12:00, 14 December 2007 (EST)draft status updated
* 2007-12-31 agree on v1.0 format
* 2008-01-01 start implementing v1 at facilities
* 2008-06    representative sampling of data available for inter-facility comparison
* 2008-11-3  presentation of results at NOBUGS2008 meeting ([http://www.nbi.ansto.gov.au/nobugs2008 http://www.nbi.ansto.gov.au/nobugs2008])
* 2009-05-12 tagged SVN changeset 67 as v1.0 since it has been unmodified since January
* 2009-09-17 agreed that v1.1 revision should be forthcoming with corrections/additions to be identified in TRAC tickets
* 2017-01    NXcanSAS 1.0 released
* 2015-12    multidim moved to github a while ago
 
= Status =
[[User:Jemian|Jemian]] 11:10, 06 September 2008 (CDT) IgorPro GUI tool included in Irena [http://usaxs.xor.aps.anl.gov/staff/ilavsky/irena.html Irena] analysis suite<br />
[[User:Jemian|Jemian]] 13:08, 14 March 2008 (EDT) IgorPro tool to read cansas1d/1.0 XML files available<br />
[[User:Jemian|Jemian]] 13:38, 25 February 2008 (EST) starting to add case studies <br />
[[User:Jemian|Jemian]] 00:39, 21 December 2007 (EST) complete first draft (v0.1a) ready for comments from 1DWG <br />
[[User:Jemian|Jemian]] 12:00, 14 December 2007 (EST) draft status updated <br />
[[User:Jemian|Jemian]] 17:07, 12 May 2009 (EDT) changeset 67 tagged as v1.0
 
= Examples and Usage =
The following are example v1.1 compliant 1D data files containing [besides essential metadata] - right-click & save as to view, stylesheet is on the TRAC site:<br /><br />
SANS data only [[media:ISIS_SASXML_v1_1_SANS_Example.XML | ISIS_SASXML_v1_1_SANS_Example.XML]]<br />
Wavelength-dependent transmission [[media:ISIS_SASXML_v1_1_Transmission_Example.XML | ISIS_SASXML_v1_1_Transmission_Example.XML]]<br />
Wavelength-dependent monitor spectrum [[media:ISIS_SASXML_v1_1_Monitor_Spectrum_Example.XML | ISIS_SASXML_v1_1_Monitor_Spectrum_Example.XML]]<br />
Wavelength-dependent detector efficiency ratio [[media:ISIS_SASXML_v1_1_Det_Efficiency_Example.XML | ISIS_SASXML_v1_1_Det_Efficency_Example.XML]]<br />


The files can be usefully displayed in many web browsers: [[Rendering_canSAS1D_in_Web_Browsers]]


[[Category:Working Groups]]
[[Category:Working Groups]]
= Mailing List=
The mailing list is now maintained by subscribing to this Google group:
* [https://groups.google.com/forum/#!forum/cansas-dfwg https://groups.google.com/forum/#!forum/cansas-dfwg]
Please address all questions and answers to this Google group.
= Session Notes SAS2024 - Sun 2024-11-03 =
=== Topic 1: Magnetic/ Polarized SANS ===
* A new proposal for the .nxs format has been published on the wiki (see section "Latest News"), e.g. including the combination of all four polarization spin channels, as well as the necessary storage of the polarization efficiency information, the metadata, sample history, etc.
=== Topic 2: Complementary data ===
* It has to be addressed how multimodal experiments can be taken into account, i.e., we will need a data format for the storage, reading, and processing of multi-dimensional data.
=== Topic 3: Summary & To-Do ===
* Further extending the magnetic/ polarized SANS data format description?
* Complementary data description
* Advanced resolution descriptions of 1D,2D,3D resolution functions - here the question was raised if this topic should form a separate working group!
* Which techniques all have to be taken into account for multimodal experiments?
* Should a hackathon be organized? If yes, at the European Spallation Source?
=== Notes from the discussion round: ===
==== Related to magnetic/ polarized SANS ====
* Polarized beam issues - who can be contacted for this topic at which source? (This hasnt been discussed thoroughly)
* This formulism should not only be inserted for polarized neutrons, but also for polarized X-ray measurements! Note1: The proposed format is specialised on the polarization of neutrons, the proposed data format should be checked. Note2: Anyhow also a proposition for the data format of polarized X-ray measurements should be done! Should for this purpose the community of polarized neutrons and polarized X-rays work together?
* Also depolarization effects from the sample have to additionally be taken into accound! Note: Shouldnt this be part of the data analysis, at it is not per se clearly distinguishable from the polarization efficiency?
==== Related to resolution functions ====
* Important raised question: When speaking about a 3D Q-resolution function: what do we define as Qx,Qy,Qz? Do the Q-components in the standard definition have to be corrected for a projection issues due to flat detectors? Note1: Typically the measurement of Qx,Qy,Qz on a flat plane (flat detector) has to be corrected to correspond to the definition of Qx,Qy,Qz on Ewald sphere. Only these corrected Q-values should be exported for users. Note2: Actually this is not a question of correcting Q, but correcting the angles. Note3: This should be done for both, neutron and X-ray measurements! Note4: For a reliable correction one should measure and correct for a reference sample.
* In reference measurements the beam profile should be logged, as it goes hand in hand with defining the resolution function
* The question has been raised how to store/record the information on the resolution function? How can we keep the description of data formats in general easily extendable?
* The action has been raised to have a working meeting for this topic next year (2025)
* In general, the formation of a seperate working group on this topic was favoured by the audience! The relevant people should come together, and importantly, from different facilities/fields

Latest revision as of 08:52, 5 November 2024

Overview

As of early 2017, for new work, it is recommended to use the NXcanSAS (multi-dimensional) standard for both 1-D and n-D data to maximize the chance that other software will be able to read your data files.

We strongly encourage all authors of small-angle scattering related software to add support for NXcanSAS.

Latest News

Steve King: 2024-09 : Proposals for NXcanSAS 1.1 in development

Paul Butler: 2019-07 : Lots of discussion at canSAS-XI about metadata. Discussion and actions from the meeting are available canSAS-XI/DataFormats

Links

nD: NXcanSAS

standard for reduced small-angle scattering data of any dimension, using NeXus (HDF5)

1D: cansas1d

standard for 1-D reduced small-angle scattering data, using XML (text)

  • v1.1: 2013-03-29
  • v1.0: 2009-05-12

Members

  • Andrew Jackson (ESS)- Chair
  • Jeff Krzywon (NIST)
  • Pete Jemian (APS)
  • Steve King (ISIS)
  • Andrew Nelson (ANSTO)
  • Jan Ilavsky (APS)
  • Lisa Debeer-Schmitt (ORNL)

Timeline

  • 2007-12-31 agree on v1.0 format
  • 2008-01-01 start implementing v1 at facilities
  • 2008-06 representative sampling of data available for inter-facility comparison
  • 2008-11-3 presentation of results at NOBUGS2008 meeting (http://www.nbi.ansto.gov.au/nobugs2008)
  • 2009-05-12 tagged SVN changeset 67 as v1.0 since it has been unmodified since January
  • 2009-09-17 agreed that v1.1 revision should be forthcoming with corrections/additions to be identified in TRAC tickets
  • 2017-01 NXcanSAS 1.0 released
  • 2015-12 multidim moved to github a while ago

Status

Jemian 11:10, 06 September 2008 (CDT) IgorPro GUI tool included in Irena Irena analysis suite
Jemian 13:08, 14 March 2008 (EDT) IgorPro tool to read cansas1d/1.0 XML files available
Jemian 13:38, 25 February 2008 (EST) starting to add case studies
Jemian 00:39, 21 December 2007 (EST) complete first draft (v0.1a) ready for comments from 1DWG
Jemian 12:00, 14 December 2007 (EST) draft status updated
Jemian 17:07, 12 May 2009 (EDT) changeset 67 tagged as v1.0

Examples and Usage

The following are example v1.1 compliant 1D data files containing [besides essential metadata] - right-click & save as to view, stylesheet is on the TRAC site:

SANS data only ISIS_SASXML_v1_1_SANS_Example.XML
Wavelength-dependent transmission ISIS_SASXML_v1_1_Transmission_Example.XML
Wavelength-dependent monitor spectrum ISIS_SASXML_v1_1_Monitor_Spectrum_Example.XML
Wavelength-dependent detector efficiency ratio ISIS_SASXML_v1_1_Det_Efficency_Example.XML

The files can be usefully displayed in many web browsers: Rendering_canSAS1D_in_Web_Browsers

Mailing List

The mailing list is now maintained by subscribing to this Google group:

Please address all questions and answers to this Google group.

Session Notes SAS2024 - Sun 2024-11-03

Topic 1: Magnetic/ Polarized SANS

  • A new proposal for the .nxs format has been published on the wiki (see section "Latest News"), e.g. including the combination of all four polarization spin channels, as well as the necessary storage of the polarization efficiency information, the metadata, sample history, etc.

Topic 2: Complementary data

  • It has to be addressed how multimodal experiments can be taken into account, i.e., we will need a data format for the storage, reading, and processing of multi-dimensional data.

Topic 3: Summary & To-Do

  • Further extending the magnetic/ polarized SANS data format description?
  • Complementary data description
  • Advanced resolution descriptions of 1D,2D,3D resolution functions - here the question was raised if this topic should form a separate working group!
  • Which techniques all have to be taken into account for multimodal experiments?
  • Should a hackathon be organized? If yes, at the European Spallation Source?

Notes from the discussion round:

Related to magnetic/ polarized SANS

  • Polarized beam issues - who can be contacted for this topic at which source? (This hasnt been discussed thoroughly)
  • This formulism should not only be inserted for polarized neutrons, but also for polarized X-ray measurements! Note1: The proposed format is specialised on the polarization of neutrons, the proposed data format should be checked. Note2: Anyhow also a proposition for the data format of polarized X-ray measurements should be done! Should for this purpose the community of polarized neutrons and polarized X-rays work together?
  • Also depolarization effects from the sample have to additionally be taken into accound! Note: Shouldnt this be part of the data analysis, at it is not per se clearly distinguishable from the polarization efficiency?

Related to resolution functions

  • Important raised question: When speaking about a 3D Q-resolution function: what do we define as Qx,Qy,Qz? Do the Q-components in the standard definition have to be corrected for a projection issues due to flat detectors? Note1: Typically the measurement of Qx,Qy,Qz on a flat plane (flat detector) has to be corrected to correspond to the definition of Qx,Qy,Qz on Ewald sphere. Only these corrected Q-values should be exported for users. Note2: Actually this is not a question of correcting Q, but correcting the angles. Note3: This should be done for both, neutron and X-ray measurements! Note4: For a reliable correction one should measure and correct for a reference sample.
  • In reference measurements the beam profile should be logged, as it goes hand in hand with defining the resolution function
  • The question has been raised how to store/record the information on the resolution function? How can we keep the description of data formats in general easily extendable?
  • The action has been raised to have a working meeting for this topic next year (2025)
  • In general, the formation of a seperate working group on this topic was favoured by the audience! The relevant people should come together, and importantly, from different facilities/fields
Retrieved from "https://wiki.cansas.org/index.php?title=Data_Formats_Working_Group&oldid=2718"