Running the Pipeline

Once installed, XDGNIRS is run by typing the xdpiped command, plus a list of options. At a minimum, it needs a list containing the names of the files on which it will operate. For example, to run the script in the same directory as the raw data files with no user interaction:

ls *fits > inputfiles.lst
xdpiped inputfiles.lst

The input file list must contain all the science, acquisition, and calibration data needed for the reduction (see Data Requirements). The files do not need to be sorted according to the data type; the code will handle that.

The available options can be seen by typing

xdpiped -h

In v3.0 these are (with defaults):

rawdir: (./)  [location of raw data]
step: 0-13 (0)  [step at which to start the reduction]
stop: 0-13 (13)  [step after which to stop the reduction]
cleanir_sciflag: nfrq (f) [whether/how to run cleanir.py on the science target spectra]
cleanir_stdflag: nfrq (f) [whether/how to run cleanir.py on the standard star spectra]
cleanir_arcflag: nfrq (n) [whether/how to run cleanir.py on the arc spectra]
cleanir_IRflag: nfrq (n) [whether/how to run cleanir.py on the IR flat spectra]
cleanir_QHflag: nfrq (n) [whether/how to run cleanir.py on the QH flat spectra]
cleanir_pinflag: nfrq (n) [whether/how to run cleanir.py on the pinhole flat spectra]
nsflat_inter: yes/no (no) [run nsflat interactively?]
despike_tgt: yes/no (yes) [attempt to remove spikes from science target data?]
despike_std: yes/no (no) [attempt to remove spikes from standard star data?]
tgt_thresh: (20) [threshold for removing spikes from science target data]
std_thresh: (50) [threshold for removing spikes from standard star data]
nssdist_inter: yes/no (no) [run nssdist interactively?]
nswav_inter: yes/no (no) [run nswavelength interactively?]
nsfit_inter: yes/no (no) [run nsfitcoords interactively?]
nsext_inter: yes/no (no) [run nsextract interactively?]
aperture: 1-40 (6) [extraction aperture, pixels (+/-)]
extras: yes/no (no) [extract full-slit spectrum and in steps along the slit?]
columns: 1-40 (6) [width of each extraction step along the slit (pixels)]
hlines: vega/linefit_auto/linefit_manual/vega_tweak/linefit_tweak/none (linefit_auto) [technique for removing H lines from standard star]
telluric1_inter: yes/no (no) [run 'telluric' interactively when using vega_tweak option?]
telluric2_inter: yes/no (no) [run 'telluric' interactively when dividing by standard star?]
continuum_inter: yes/no (no) [interactive continuum fitting?]
offsets: no/manual (no) [enter specplot, edit order scaling?]
shift_to_rest: yes/no (no) [look up redshift and shift to rest frame?]
error_spectrum: yes/no (no) [generate S/N and error spectra? Only works on data nodded off the slit]
wave_soln_linear: yes/no (no) [Sets the header keyword "CTYPE1" to 'LINEAR']

These are explained in more detail in the relevant sections of this document. XDGNIRS can often produce a useful quick-look spectrum with no user intervention. Fairly frequently, however, the user will need to adjust the radiation event detection thresholds before a sensible result is obtained. Faint targets will probably require manual extraction. Science-quality reductions will normally also involve the interactive removal of intrinsic lines in the telluric standard and perhaps interactive optimization of the telluric line removal. Try saying “yes” to all interactive options to get to know your data. Error handling is somewhat ad-hoc and limited; we have attempted to make the script stop tidily upon encountering problems that we have encountered during testing, but that may not always be the case.

A full data reduction on a data set from the Science Verification Examples section, using the virtual machine takes on average 4 minutes on 2019 MacBook Pro with a 2.6 GHz 6-Core Intel Core i7 processor. The most time-consuming pieces are steps 2 and 5 (preparing and transforming the data). The code can be interrupted at any time with a ctrl-C.