bm2img : image preprocessing on BM2 beamline
A program, called bm2img has been writen for this purpose. In case of the direct illumination CCD refer to wv2ph or edf2ph programs.
- Short example
- Converting a winview CCD image (.spe) to a 32 bit EsrfDataFormat (.edf)
- Dark current correction
- Flat field correction
- Outputs
- Processing series of data
- Inline help.
- Radial distribution for SAXS image
- Normalisation using spec files
Procedure
All data are stored internally as 32 bits signed integer, this choice has been done in order to reach a good dynamic without losing the counting accuracy. When the image are read they are corrected from dark current (if a dark image has been set) and from flatfield (if set). Then various operation can be done on the 32 bits image. When writing an output file, the dynamic is only modified on the resulting file not on the inernel image. If you rescale the image, remember that integer operations are order dependent.
Short example
Converting a winview CCD image (.spe) to a 32 bit EsrfDataFormat (.edf).
The following command transforms the winview file dani10.spe into the 32 bit file dani10.edf
bm2img dani10 %X
Dark current correction
The following command transforms the winview file dani10.spe into a 32bit file corrected for dark current... dani10d.edffile.
bm2img D=dark dani10 %X
The following similar command applies to a preprocessed dark edf file that is the sum of 15 frames.
bm2img D=dark.edf,U15,X dani10 %X
A reduced view of a SAXS image can be seen below. In the central part it shows a small angle scattering image using 2*2 binning. To show the influence of the correction on 8 bit .gif images, intensities higher than 3 times the image average have been truncated; this occurs near the beamstop and/or on cosmic rays. When an images has been stored in edf format, it is always assumed that this image is made by only one frame, then it is mandatory to use the U parameter to correct this if the dark file contains more than one frame.
Flat field correction
The following images show the transformation of the data. In the central part they show a small angle scattering image using the same binning as for the above flatfield. To display the influence of the flatfield correction on 8 bit .gif images, intensities higher than 3 times the image mean have been truncated : this occurs near the beamstop and/or on cosmic ray spots, that have not be fully cleaned.
bm2img D=dark F=flat.edf,X dani10 %X
| dark corrected | dark and flatfield corrected |
| bm2img D=dark dani10 %X | bm2img D=dark F=flat.edf,X dani10 %X |
 |
 |
Output
The output name is by default
name(d)(f).$EXTENSION
("d" is added to the name if the dark correction has been made)
("f" is added to the name if the flat correction has been made)
$EXTENSION indicates the format of the file.
output of an image
The key commands used to write the corrected image on disk are
| command | %X | %Y | %Z |
| dynamic | 32 bits | 16 bits | 8 bits |
| default | name.edf | name.edf | name.edf |
| .edf | name.edf | name.edf | name.edf |
| .gel | name.gel | name.gel | name.gel |
| .tif | name.tif | name.tif | name.tif |
| .txt | name.txt | ||
Most of the viewers can handle 8-bits tiff, 16-bits gel (or tiff) is also a standard format. Edf file can be read by fit2d or viewFile.
The txt output can be used to extract some binded rows or columns of the images, it does not depend on the selected dynamic. Obviously trying to output a full image using this mode may produces some disk full error!!!
Processing series of data
The first way to process a series of data is to use a script shell but it is more efficient to use bm2img internal redirection if all the images use the same dark and flat.
UseSTDIN=inputfile in the bm2img.ini
file and fill this file with all the order you need. Do not forget
to clean the memory (%F) between each image to process
or they will be added together.
The following lines show an example of inputfile.
D=v2zr51.edf,U5,X img39 %X %R %F img40 %X %R %F img41 ...
The inline help
The commandbm2img -h will display its principal key words.
....
## bm2img : extracting info/preprocessing image
## berar@esrf.fr (d2am), , Nov 26 2001
usage : bm2img [D=|F=]img_file[,Bb,Ee,X,Aa|Mm,Uu,Tt,Ss] [img...] [%M=val] [%[X] [img...]]
example D=dark3,B2,A5 add frame starting at 2 in dark_files dark3 to dark8, U force used frames to u
D=dark_file to substract, F=flatfield_file (must be defined before image
Begin at frame b, End at frame e, X don't correct
Add a files with increasing number (or Merge them, DMI specific)
Inside the program the image are kept as integer with 32 bits
%M=val (%A=val) multiply current image by val (or add to) before reading next arguments
%C=vah[,val] cut counts higher (lower) than vah*mean before reading next arguments
%S=val rescalling dynamics to val
%X[[name].edf|.gel|.tif|.txt] produce outfile (edf default with 32bits)
%Y[[name].edf|.gel|.tif|.txt] produce 16bits outfile (gel default) rescalling the output if mandatory
%Z[[name].edf|.gel|.tif|.txt] produce 8bits outfile (tif default) rescalling the output if mandatory
%R produce radial distribution file (.rad), one angular sector is the default
%P[=m[,o],v]]] produce peak list file (.pks), m=5 peaks is the default, omega osc if not 0, sum>v*max, v=1.5
%H produce an ascii map (h, k, l) of the file using the UB matrix
%F clean current image (but keeps dark and flat)
a specific statistic routine is called when the program name contains the string 'stat'
previous keys are not valid in this case
see also prameters in bm2img.ini file
valid options in .ini files are :
Items used in the .ini file
X_MIN =
Y_MIN =
X_MAX =
Y_MAX =
X_BIND =
Y_BIND =
BEG_FRAME =
END_FRAME =
MULT =
P_EXCLUDED = x1 y1 [v]
C_EXCLUDED = x1 y1 r [v]
R_EXCLUDED = x1 y1 x2 y2 [v]
T_EXCLUDED = x1 y1 x2 y2 x3 y3 [v]
C_ROI = x1 y1 r [v]
R_ROI = x1 y1 x2 y2 [v]
DIR = full path to search for data file
OUT = full path for result (.res) file
DEF = default file extension (if not default for the prg_name : spe2img, edf2img, gel2img)
CR = 0, set to 1 if you wish cr in rad file
DFORMAT = size of ineger field in output name, default same as entry
STDIN = full path for redirecting stdin SPEC = full path to spec file for collecting info FLAT_NAME = full path of the flatfield file to use
SPEC = full path to spec file for collecting info
WSPEC2XL = arguments to the script to extract normation
XL_COL = pos of the selected column in WSPEC2XL
DARK_NAME = full path of the dark file to use
SCA_DARK = scale factor to apply on dark
ADD_DARK = offset to add to dark curent
O_COSMIC = file_name, STDOUT or default to image_name.cosmic
T_COSMIC = used for time removal
S_COSMIC = used for space removal
R_COSMIC = ratio on the standard deviation used for time and space removal
H_COSMIC = lower cosmics on one frame for time removal
P_COSMIC = lower cosmics on the frame sum for spacial removal
SET_LIMIT = remove values lower than LOW_LIMIT if not zero
LOW_LIMIT = values are replaced mean of surrounding values (SET_LIMIT=1), by LOW_LIMIT (SET_LIMIT=2)
ADD_DARK = offset to add to dark curent
REALTIME = 0 or waiting time in secs to ensure data completion
PRINT = ... +64 cosmics, +128 cor, +256 sum, +512 raw
T_DATA = output mode (float...) for edf file
ORIG = pixel_x(=0) pixel_y(=0) center of detector
MAT_UB = values according spec
LAMBDA = 1 (if not given)
GONIO = 0.00 th chi phi
DETEC = alpha tth Rx(=1) Ry(=0) Rz(=0)
SCALE = scal_xx(=1) scal_xy(=0) scal_yx(=0) scal_yy(=1) matrix form pixel to detector
ORIG = pixel_x(=0) pixel_y(=0) center of detector
PLANE = a x+ b y+ c z= d e ecart
following group only concerns radial distribution
RADIUS = radius binning
RAD_LIM = low limit value in radial distribution (not set is no limit)
N_SECTOR = number of angular sectors (num 0 to N_SECTOR-1)
A_SECTOR = origin (degree) of the first sector, if not precised default is to be centered on X axis, use a 0 value to start from X axis
P_SECTOR = if not empty, binning specifications without space
radial distribution example, N=8,A=-22.5,P=0+4,2+6,1+3+5+7 print 3 curves
Radial intensity distribution for SAXS images
In SAXS experiments, it is not always necessary to write a corrected image in a file. You can obtain the radial distribution directly using the following command. In case of anisotropy, more than one angular sector can be specified.Then the following line can be used in many cases, this line can be included in a shell script for processing a lot of images when processing conditions are defined.
bm2img D=dark F=flat.edf,X dani10 %R
An ascii output file is created with the extension ".rad"
specific parameter in the bm2img.ini file
| key | expected | default | |
| ORIG= | x y | CCD center | float values to define the center of the image |
|---|---|---|---|
| N_SECTOR= | n | 1 | number of sectors : angular resolution for anisotropic images
(one sector is obviously isotropic) |
| A_SECTOR= | n | 0 | usefull for anisotropic data : origin (degree) of the first sector, if not precised default is to be centered on X axis, use a 0 value to start from X axis |
| P_SECTOR= | string | empty | print statement for binning sectors, see examples below |
| RADIUS= | n | 1 | binding radius parameter |
| RAD_LIM= | f | -Inf | allow to avoid Zero or Negative values in radial distribution
(values below rad_lim are set to rad_lim |
Default output file
If P_SECTOR is not found, the radial distrubution file looks like:
# ./00jan147df.rad (400-900)/1*(400-800)/1 (00jan147df.gel)
# center 545.90 626.80 -> 145.90 226.80, r_max 420.51
# 16 sectors with origine at 0.00 deg
# r s/n e(s/n) n ...
0 0.00 -1.00 0.0 19.75 -1.00 0.1 0.00 -1.00 0.0 0.00 -1.00 0.0 0.00 -1.00 0.0 86.57 -1.00 0.3 0.00 -1.00 0.0 0.00 -1.00 0.0 0.00 -1.00 0.0 0.00 -1.00 0.0 120.00 -1.00 0.5 0.00 -1.00 0.0 0.00 -1.00 0.0 0.00 -1.00 0.0 49.61 -1.00 0.2 0.00 -1.00 0.0
1 47.73 -1.00 0.2 233.25 -1.00 0.9 62.13 -1.00 0.3 0.00 -1.00 0.0 94.52 -1.00 0.4 176.43 -1.00 0.7 142.05 -1.00 0.6 0.00 -1.00 0.0 164.44 -1.00 0.6 0.00 -1.00 0.0 141.03 -1.00 0.6 141.49 -1.00 0.6 0.00 -1.00 0.0 109.12 -1.00 0.4 206.39 -1.00 0.8 64.66 -1.00 0.3
2 243.13 -1.00 1.0 182.34 -1.00 0.7 182.87 -1.00 0.7 85.77 -1.00 0.3 262.33 -1.00 1.0 246.62 -1.00 1.0 157.70 -1.00 0.6 169.46 -1.00 0.6 254.40 -1.00 1.0 80.24 -1.00 0.3 241.18 -1.00 1.2 255.89 -1.00 1.0 140.50 -1.00 0.5 156.90 -1.00 0.6 203.42 -1.00 0.8 261.12 -1.00 1.0
3 253.68 -1.00 1.0 268.60 -1.00 1.4 231.71 -1.00 0.9 277.05 -1.00 1.0 271.35 -1.00 1.2 255.20 -1.00 1.5 246.90 -1.00 1.1 254.32 -1.00 1.0 260.36 -1.00 1.0 259.13 -1.00 1.1 258.15 -1.00 1.4 273.01 -1.00 1.4 282.03 -1.00 1.0 260.58 -1.00 1.1 267.47 -1.00 1.5 277.58 -1.00 1.1
4 279.85 -1.00 1.0 280.01 4.29 2.3 293.07 -1.00 1.6 281.30 -1.00 1.1 265.20 4.30 2.1 249.44 -1.00 1.3 257.70 -1.00 1.5 244.29 -1.00 1.4 255.63 -1.00 1.5 254.01 -1.00 1.6 270.05 -1.00 1.5 276.85 9.92 2.1 281.49 -1.00 1.3 294.67 -1.00 1.5 286.19 -1.00 1.2 271.42 5.35 2.1
5 289.64 5.37 2.1 304.81 16.00 2.1 300.79 -1.00 1.3 302.34 13.46 2.1 287.04 27.20 2.0 267.82 17.94 2.4 251.79 -1.00 1.7 268.03 18.86 2.1 242.62 20.99 2.0 259.82 -1.00 1.7 279.27 11.21 2.4 288.04 20.69 2.0 301.69 22.29 2.1 318.00 -1.00 1.4 288.49 6.81 2.3 274.35 9.72 2.0
6 297.29 13.17 2.0 315.34 9.24 2.9 333.89 10.67 2.3 336.10 24.43 2.0 329.26 13.30 3.0 294.24 20.22 2.0 249.00 1.66 2.1 260.46 15.27 2.2 252.05 14.14 2.3 263.75 14.57 2.1 284.13 11.89 3.0 314.98 5.94 2.2 355.74 36.16 2.0 338.35 20.64 2.4 320.42 13.93 2.8 295.34 36.50 2.0
7 333.05 26.76 2.8 331.37 31.35 2.7 372.27 27.82 2.1 378.13 14.81 2.9 352.68 8.67 3.1 325.46 23.81 2.8 257.67 8.26 2.3 258.64 10.90 3.1 257.92 7.65 3.1 294.31 15.25 2.3 316.24 15.00 2.9 363.37 34.69 3.1 401.71 28.77 2.2 390.97 31.50 3.0 356.85 36.73 2.7 362.83 24.98 2.9
in which
| r | stand for the radius of the ring |
|---|---|
| s | is the sector number |
| s/n | is the counting per pixel |
| e(s/n) | is the standard deviation on the counting |
| n | is the pixel number taken in account in the ring |
Output file with binned sectors
The P_SECTOR string is interpreted to join the sectors in the output file. The string format consists in the number of the sectors and '+' or ',' signs, space are not allowed. The following example is consistent with sectors of 45deg, the first group is made by two sectors centered on the horizontal axis, the second on sectors from vertical axis and a third consists of the other sectors. Only the specified sectors are printed.# ./00jan147df.rad (400-900)/1*(400-800)/1 (00jan147df.gel)
# center 545.90 626.80 -> 145.90 226.80, r_max 420.51
# 16 sectors with origine at 0.00 deg
# print sectors binned according : '0+8,4+12'
# 4 used sectors (of 16) are binned in 2 group : 0 ( 0 8) , 1 ( 4 12)
# r s/n e(s/n) n ...
0 0.00 -1.00 0.0 0.00 -1.00 0.0
1 212.17 -1.00 0.8 94.52 -1.00 0.4
2 248.74 5.80 2.1 268.51 -1.00 1.5
3 257.01 9.49 2.0 276.33 6.17 2.2
4 265.64 13.20 2.5 271.52 7.71 3.3
5 266.17 16.05 4.1 294.42 15.03 4.1
6 273.35 14.83 4.3 340.04 15.23 5.0
Normalisation using spec files
The %M command allows since Nov, 2001 to normalise the file looking into "SPEC" file. This call the script wspec2xl.pl to do it, but this argument may be changed. Using the following lines in bm2img.iniSPEC=/mat_dsk4/visit/xx...xx/fourc.26Oct01 WSPEC2XL=wspec2xl.pl -m XL_COL=5the command bm2img alpdre33.edf %M=1e7/SPEC
does...
# command : %M=1e7/SPEC spec2xl 1e7/SPEC running wspec2xl.pl -malpdre33 /mat_dsk2/lcontact/berar/Spec/fourc.26Oct01 >bm2img.SPEC using 1e+07/1.404e+07=0.712 as coefficient # alpdre33.edf : 1 frame(s), mean 16742.1 sigma 60758.2 [0@0*0 - 466609@13*58] 5220!=0It is obvious that if you wish to keep some statistical meaning, the efficient coeffficient must be of some order of magnitude (100 by example) as data are integers inside bm2img.
Updated by jf berar on Nov. 26th, 2001
A hydrogen reservoir made of … mostly hydrogen
Inauguration of Refurbished ROBL beamline
Debian for Scientific Facilities Days