2012 Standards Discussion

Discussion on Standardisation

Comments Made Prior to canSAS2012

• The following is the agenda of work posted under business for canSAS-2012. Please add comments and expand on details here:
  • Purpose and goals: Intercomparison of data measured on the same sample with different instruments and different techniques (SAXS, SANS, light scattering etc.) can prove valuable in a number of ways. In particular it aids understanding of details of the experimental methods and it can help assess reliability. In a similar way, looking at results of data reduction or analysis generated with different software can provide valuable information about performance and verification of methodology. Specifically these activities should:
    • Provide Quality Assurance/Quality Control,
    • Improve (reduce) uncertainties of SAS measurements in general,
    • Help each facility continuously improve performance and quality of data.
  • We will discuss what types of tests are interesting/important:
    • Beam intensity standards - there are several different ways to quantify this
    • Standards to test resolution
    • Absolute intensity calibrations,
    • Materials for Q calibration,
    • etc. etc,
  • Standards are not just measurements:
    • Software comparison - do we derive the same results from different computer programs?
    • Analysis methods may be similar or different (e.g. modelling versus transforms versus calculation of invariants)
    • Different procedures use different approximations - are these documented?
    • Approximations rather than the most elaborate calculations may be useful? Under what circumstances?
    • How do analysis programs interpret data? What do they assume if data (such as uncertainty or resolution) is missing?
  • Some other related issues:
    • Inelastic,
    • Multiple scattering,
    • Wavelength contamination,
    • Grazing incidence scattering - standards,
    • Detector efficiencies at different wavelengths,
    • limits in signal to noise - how weak a signal can be reliably extracted,
    • etc.
  • Outcomes needed are:
    • A written plan to sustain long term effort in this area
    • This should describe how to seed, co-ordinate and publicise “ad-hoc” projects,
    • Assess how frequently exercises can be undertaken?
    • Define good ways to disseminate/share results. This will including “advertising” projects and using them as input for other activities.
  • We should aim to define a list of 2 or 3 projects for work in the near term. This should include a plan of action and participants for each.
  • We shoud have a plan for presentation at SAS 2012. (This might just be an announcement of the plan and see who wants to participate?)

• ARR suggests: that we might discuss how people will be able to meet the ideas in the article by Jacques et al that describes guidelines for publication of SAS data from biological macromolecules. There is an accompanying editorial. Are there ideas for modifications to these guidelines? (D. A. Jacques, J. M. Guss, D. I. Svergun and J. Trewhella 'Publication guidelines for structural modelling of small-angle scattering data from biomolecules in solution' Acta Cryst. (2012), D68, 620-626. doi:10.1107/S0907444912012073)

• ARR: What have we learnt and what more can we gain from theGlassy Carbon Round Robin and polystyrene Latex Round Robin exercises?

   PDB  I think these have shown that the current agreement is as expected withing 10 or 20% in most cases (which is all 
        the technique really claims to be good to if you read the old papers)  I think the real opportunity now is to see if we
        can go beyond that and figure out how to get agreement regularly at the 5% level.  That probably means the community
        will have to understand a lot of the subtler issues that have been shoved under the rugs to date and can come in from
        instrument hardware improvement to the analysis software improvements -- my 2c worth:-)

Discussion 28 July 2012

Agenda

The 'Standardisation' sub-group started by having a general discussion with the following agenda items:

• Suggest what new standardisation is needed
• Identify how best to organise activities
• Think about ways to document results
• What more can we learn? How can one disseminate more from previous activities

What new work is needed?

It is necessary to find a broader range of materials and samples that can be used as secondary standards so as to allow measurements with a broad range of instruments and configurations. There are boundariues imposed by count-rate, q-range etc. Users may often prefer to use standards that are related to their field of science (better understanding of any observed anomalies). Samples that are robust (physical handling, temperature, beam damage, etc.) and that can be made in adequately large quantities are desirable.

Work needs to extend beyond traditional calibration of intensity and momentum transfer/wavelength. For example it is useful to determine resolution, detector efficiency etc.

Standards for emerging techniques such as grazing incidence scattering are needed. At the moment there is little intercomparison of data in this field and relatively poor modelling of absolute intensity. The field would benefit from stable, 'standard' samples that might give calculable scattering patterns and could be compared at different facilities. It would be helpful to have samples that were appropriate to both GiSAXS and GiSANS.

Discussion and improvement of publication standards is important. There are some challenges in providing standardised deposit of data with documentation of how the data are reduced. Good practice would have appropriate metadata maintained in processed data files. Procedures for data reduction and data analysis need considerably more documentation than just program names as input parameters such as transmission, scaling, and calibration procedures/data are required to reproduce the analysis.

Organisation of Activities

• Small-groups can work efficiently. These may need to be formed 'ad-hoc' from people that have a particular interest in a problem or field. The expansion of a particular comparison or study to larger groups can follow in a staged process.

• The benefit of standardisation is achieved when facilities and instruments can ensure that results are exploited. Faciltating application of new calibration methods and procedures is important

Documentation of Results

Documentation is crucial. Sharing knowledge and understanding of the technique that has been gained is the perhaps the most important part of 'standardisation'. Desciptions that appear in published papers will have most impact.

Exploiting Previous Studies

(a) The round-robin activities with glassy carbon displayed interesting SANS results that should be analysed further and described. The publication on USAXS/SAXS from this material is helpful but the broader application, particularly for SANS would require interpretation of the data as regards contrast, wavelength dependence etc.

Some further work on inelastic scattering may be needed on the same samples that have been measured in the round-robin.

(b) Work on interpretation of the results of polystyrene latex round-robin is under way and an extended abstract for SAS2012 is available. A fuller paper would be useful.

Discussion 29 July 2012

Round Table 29 July 2012 AM

• Questions were raised regarding long-term reproducibility of measurements from standards
  • Here are some fitted transmission values from the ISIS 'TK49' dPS/hPS polymer blend standards & their random copolymer backgrounds measured with the same detectors and calculated by the same algorithm:
  • BLEND: June 97: 0.8429 @ 0 Ang, 0.6348 @ 10 Ang
  • BLEND: July 05: 0.8648 @ 0 Ang, 0.6650 @ 10 Ang
  • COPOL: June 97: 0.8419 @ 0 Ang, 0.6518 @ 10 Ang
  • COPOL: July 05: 0.8406 @ 0 Ang, 0.6878 @ 10 Ang

(Data provided by SMK)

Group Discussion

The group discussion focussed on some main themes:

How to disseminate better results from previous projects?

• It is appropriate to publish papers on the round robin studies that have been established. Separate papers for submission to e.g. J. Appl. Cryst. on the glassy carbon (with a focus on comparison of SAXS (and USAXS) with SANS and wavelength variation observed in SANS studies) and on the polystyrene latex (expanding on the submission to SAS2012) should be prepared.

• Sarah Rogers will look at the measured data from ISIS to see how different wavelengths compare and wheter similar differences to those in the NCNR data are apparent. We will enquire of Elliot Gilbert whether the C4 and G9 samples used for the round robin are still at are still at ANSTO and could be returned for inelastic measurements on at either ISIS or ILL at the start of the next cycle. Authorship and work on the article should be discussed with the facilities involved.

• Conclusions from the PS3 latex comparisons were summarised for submission to SAS2012:
  • On time-of-flight SANS instruments, data can be measured with comparatively low dQ/Q over a wide range of Q in a single configuration. This has drawn attention to the significance of multiple scattering that smears measurements in different ways to polydispersity and instrument resolution.
  • Measurements on samples with different but low concentrations on monochromatic SANS instruments have verified the effect of multiple scattering. This has prompted development of a simple algorithm that can be included in fitting software to include most of the effect of multiple scattering.
  • The well-known form factor of spherical particles has allowed specific problems with procedures to correct for detector uniformity and intensity calibration to be identified.
  • Calculations of instrumental resolution have been improved with good data that allowed test of models and input parameters. The need to have accurate values of wavelength distribution that can vary with collimation has been identified if precise data modelling is to be undertaken.
  • Use of samples on a range of instruments and treating data with a variety of software has highlighted a few deficiencies in metadata (e.g. provision of resolution or the parameters to calculate resolution or data to determine absolute intensity). Facilities are working to rectify these difficulties.
  • The uncertainty in fitted parameters, such as radius and polydispersity, is limited less by statistical uncertainty in data and fitting but more by knowledge of systematic errors in calibration and modelling. For a given data set and analysis procedure, the uncertainty in radius, for example, may be just 0.5 to 1% but the spread of values may be about 2% and this depends mostly on the choice of analysis and interpretation of instrumental factors.

• A number of other facilities and users (FRMII, Diamond, APS, Spring-8, etc.) have expressed recent interest in measuring samples and these will be encouraged. It was felt that publication of the results should not wait on further data. These can be incorporated in future publications if they are not yet available.

New Projects

Given the likely available effort and the time required to complete projects only a limited number of studies should be initiated although a number of ideas for requirement have been identified. It was thought that for efficiency just two or three facilities or groups would be optimal to initiate work on any give project (prepare samples, initial tests) and that these should then seek broader participation as soon as possible. From the initial ideas listed above the following measurement projects were proposed:

• Preparation and measurement of one or more 'standards' for grazing incidence scattering (both GiSAXS and GiSANS). This would probably be a nanofabricated, large two dimensional array with anisotopic pattern on the scale of hundreds of nm. Coating or making the sample of material that did not change under normal atmospheric conditions would be important. Some details need to be discussed: what depths of features are desirable to test scattering intensity? Sizes? anisotropy. Gratings and Fresnel lenses are readily available. It is necessary to find out where best access to nanofab facilities is available: ISIS has co-operation with Leeds University and this could be explored. Some labs in USA also have good fabrication facilities. ISIS and Diamond are likey to want to co-operate in initial measurements.

• Green fluorescent protein is available in large quantities at ISIS and is stable when frozen and gives reproducable solutions when suitable protocols are followed. Some SANS data is probably already measured Sarah Rogers will talk with Luke Clifton to identify what data has already been collected and possibility of further maesurements will then be explored. It would be good to invite EMBL, Hamburg and Australian groups to join these activities early as they have strong interests in this area and expertise.

• A longer term activity that we will need to discuss further will be to look at reproducibility of calibrations and the reliability limits (uncertainty) introduced by a range of systematic errors. Such uncertainties may need to be categorised in different ways such as those that may vary within a particular sequence of measurements, those that alter in a similar manner all data measured in a sequence, errors that may alter shape of measured curves or those that alter only scaling, etc. This work may involve analysis of past measurements and possibly further new measurements and will feed in to publishing guidelines and how data should be described.

Discussion on these topics will continue on Monday...

Discussion 30 July 2012

Agenda:

• How to describe uncertainties?
• Recommendations for data formats and publishing
• Who to approach for work on specific projects?
• Timetables

How to describe uncertainties?

Systematic errors may be larger than random errors that arise from counting. This conclusion leads to the need to provide more information about variation of parameters and uncertainty. In general individual components of uncertainty and how they are obtained will be needed to derive estimates of uncertainty in model parameters.

• Some factors could provide correlated or systematic errors that would still allow precise relative values of parameters to be derived from a specific series of measurements but detract from absolute accuracy. For example, an error in an attenuator or direct beam calibration measurement might effect the intensity scale and would alter all intensities to the same extent. Similar errors in wavelength calibration could alter the momentum transfer, Q, scale.

• There can be more complicated patterns of systematic errors - for example an error in 'flood field' or response calibration for an area detector might systematically alter the shape of measured intensity versus Q or the anisotropy of measured scattering patterns.

• At present few papers report these different uncertainties althought many studies implicitly assume that relative errors are smaller than absolute errors.

• Data structures will need to describe 'calibration' uncertainties.

• Sample parameters will also have associated uncertainties. For example sample thickness and concentration will directly relate to the scattered intensity and consequently alter derived values such as molecular mass of macromolecules.

' Even uncertainty or fluctuations in other parameters such as temperature could alter the physical state and hence derived results.

Data formats and publishing

Best practices in data handling and establishment of reliability will require automated transfer of data from the source measurement to the archive format of reduced data that is required for acceptance of publication. This will include both scattering data and metadata that describes sample composition and conditions. The treated data format should facilitate this process. Different scientific fields and experiments may require different parameters to be described.

• In some fields such as biological small-angle scattering there are requirements emerging that will require specific information and data to be included in an archive for publication.

• The description of the reduction process is facilitated by maintaining a record of data treatment history. In particular component measurements such as empty cells, measured background and normalisation may need to be specified.

• The requirements for recording the reduction could be met by including a systematic record in data files, for example a record of the process for each step with a record of the software used. This also facilitates identification of problems that may arise in the reduction.

• Specific information about the intensity of background could be included in data files or deposit data.

• It is to be expected that publication standards may eventually require, in some specific fields, information about 'background', calibration procedures, etc.. This may be very important, particularly when small signals are measured with large backgrounds.

• Some examples of metadata relating to the sample for which recommended tags could be useful are:

SASsample:

DeformationType: Shear, 1D-elongation, 2D-elongation, compression

- Extent of deformation

- frequency

Stress

Temperature (for temperature jump)

- elapsed time since the jump

- temperature change rate

pH

concentration

magnetic field

- strength/amplitude

- frequency

- direction

- elapsed time since application...

electric field

- strength/amplitude

- direction

- frequency

- elapsed time since application...

irradiation with something else (light, UV and others)

- type (UV, wavelength etc.??)

- strength

- elapsed time since the irradiation

Corresponding uncertainty (e.g. systematic, standard deviations of random errors) may be required for any of these sample parameters.

Report on Work at the Meeting

The Report of the working group is available here.StandardsGroupReport

New Projects: Discussion and Comments Added after the Meeting

We are discussing new projects for standardisation here. When they are formulated further, they will be moved to their own pages.

Standards for BioSAS

Katy wood 10:09, 14 August 2012 (EDT)

Re protein standards: On Quokka we've measured Glucose Isomerase (170 kDa, 2 mg/mL) + Lysozyme (14 kDa, 8 mg/mL) in both H2O and D2O. Both are commerically available, are nicely monodisperse, withstand freezing once in solution and scattering fits the published structures well – I’m writing short note up about this now, to show quokka capabilities for structural biology. Molecular weights extracted from I(0) were within 10%.

Quokka data on GI:

Grazing Incidence Small-Angle Scattering

A durable sample with surface structure suitable for repeatable measurements with both X-rays and neutrons is being investigated. Sarah Rogers (ISIS) is investigating what can be fabricated.

Systematic Errors - Understanding Data

The recognition that calibrations, data reduction procedures and other factors such as resolution, multiple scattering and variations with wavelength has been an important conclusion of the previous 'round-robin' exercises. Quantification and understanding magnitudes of these effects is to be an on-going activity.

ARR: 26 August 2012

Here are some preliminary ideas for some information that is needed about procedures and data to make initial progress:

• A number of data reduction programs provide extra information such as resolution - it is valuable to document how this should be interpreted (sigma for Gaussian, FWHM, etc) and how it is estimated or calculated.
• Analysis (fitting) programs need to describe what they expect for resolution data.
• Analysis of multiple scattering requires as a minimum some transmission information with the reduced data. The description may need to include how this is determined (solid-angles of integration, etc.)