Where are processed my data? what file can I find?

  • /PROCESSED_DATA/John/fooprotein/autoprocessing_fooprotein-xtal1_run2_1/grenades_parallelproc/1_88.1_104.1_107.8_90.0_90.1_90.2

holds the results for processing in P1, unit cell dimensions = 88.1, 104.1, 107.8, 90.0, 90.1, 90.2

Files within each space group/unit cell dimensions directory include:

XSCALE.LP:  final integaration statistics 

ccp4.mtz:  Data in CCP4 mtz format

unmerged.mtz:  Data in multi record MTZ format, suitable for SCALA scaling

xscaled_merged.sca:  Scalepack format, merged

xscaled_unmerged.sca: Scalepack format, unmerged

scala.log: Statistics from SCALA

  • Moreover, you can now monitor your results through ISPYB (LIMS).

More details of what is really happening:

  • Integration is done in XDS and the resolution is cut at each round (multiple rounds are used in the first place in order to refine beam X, Y, distance, unit cell parameters etc).
  • After XDS integration, XSCALEing is done four times:

merging, anom ON

merging, anom OFF

no merging, anom ON

no merging, anom OFF

  • In the merged cases, data are converted to MTZ format in two ways:
  1. The result of the XSCALE is converted to a scalepack file, then via SCALEPACK2MTZ, then TRUNCATE, then UNIQUEIFY generating these files:   merged_anom_sc2mtz.mtz and merged_noanom_sc2mtz.mtz
  2. a) POINTLESS is used to convert the XDS_ASCII file from XDS (i.e. pre XSCALE, pre conversion to SCA) to a multirecord MTZ file, suitable for SCALA

          b)SCALA is run on this data , which generates the log file.

  • Truncate and downstream programs are not run at the moment

 Files generated:

merged_anom_pointless_multirecord.mtz

merged_anom_pointless.mtz

merged_noanom_pointless_multirecord.mtz

merged_noanom_pointless.mtz

  • In the POINTLESS version, there is one round of resolution estimation missing so:

The .sca files are usually a better estimate of the resolution of the data. 

If you find this not to be the case, please let Max know.  I would recommend checking the *XSCALE.LP log files to see if it has cut resolution to your standards.

 

Check the results carefully. Below shows a typical case for .sca files and mtz file comparison. The SCA files appear to be truncated to a more realistic resolution than in the POINTLESS files (as expected).

pointless:  Outer shell res, shell I/sig, Rsym = 3.05, 1.0,  87% (!!)

scalepack: Outer shell res, shell I/sig, Rsym  = 3.22, 3.36, 41.4%

Updates

  • February, 2011 Improved indexing reliability by including more images in XDS spot picking, with a failover to LABELIT
  • October 6, 2010 Fixed a bug which was causing the P1 integration to take a long time (and cause subsequent POINTLESS run to be slow as well)
  • October 5, 2010 Added many new files to be attached to ispyb:

merged_anom_sc2mtz.mtz

Merged data, anomalous pairs treated separately, imported via Scalepack2MTZ

merged_anom_pointless_multirecord.mtz

Merged data, anomalous pairs treated separately, imported via pointless, in multirecord MTZ format (for import into SCALA, for example)

merged_anom_pointless.mtz

Merged data, anomalous pairs treated separately, imported via pointless, in MTZ format after SCALA

merged_noanom_sc2mtz.mtz

Merged data, anomalous pairs treated as equivalent imported via Scalepack2MTZ

merged_noanom_pointless_multirecord.mtz

Merged data, anomalous pairs treated as equivalent imported via pointless, in multirecord MTZ format (for import into SCALA, for example)

merged_noanom_pointless.mtz

  Merged data, anomalous pairs treated as equivalent, imported via pointless, in MTZ format after SCALA

merged_anom.sca

Merged data, anomalous pairs treated separately, SCALEPACK format

merged_noanom.sca

Merged data, anomalous pairs treated as equivalent, SCALEPACK format

unmerged_anom.sca

Unmerged data, anomalous pairs treated separately, SCALEPACK format

unmerged_noanom.sca

Unmerged data, anomalous pairs treated as equivalent, SCALEPACK format

merged_anom_pointless.mtz_scala.log

  logfile for "Merged data, anomalous pairs treated separately, imported via pointless, in MTZ format after SCALA"

merged_noanom_pointless.mtz_scala.log

logfile for "Merged data, anomalous pairs treated as equivalent, imported via pointless, in MTZ format after SCALA"

merged_anom_XSCALE.LP 

XSCALE log file, anomalous pairs treated separately

merged_noanom_XSCALE.LP

XSCALE log file, anomalous pairs treated as equivalent

  • Sept 2010 modified code to avoid deadlocking the CONDOR cluster