This routine is a wrapper for the p3d main data-reduction graphical user interface (GUI).

The following documentation describes how to use p3d, including all the components of the GUI.

Note: p3d can, in principle, be used with any fiber-fed IFS. However, the actual use is only possible if p3d is first configured for an instrument. The parameter list below contains a complete list of currrently configured instruments.

Note: p3d only works with FITS-formatted images, i.e. fits files. Such files have a header that contains additional information about the observing conditions, the telescope, the instrument setup, and other useful information. Both input and output fits files can be compressed using gzip, to save storage space (also on Windows platforms); p3d handles such compressed files transparently.

How to start and use p3d

The different ways to use p3d and its tools

p3d consists of a set of separate tools for reduction of data of fiber-fed IFSs. The main GUI of p3d provides a straightforward mean to use these tools without the need to configure each component individually, by setting all required options. The actual GUI code is contained in the routine p3d_tool_gui.pro; this routine is a wrapper that calls p3d_tool_gui.pro with the appropriate instrument-specific parameter file and title string, along with the additional options.

If there is a need to do so, many of the tools can be executed separately, outside the GUI, from either the IDL command prompt or from a (shell) terminal (optionally using the IDL Virtual Machine, see below). If you want to do this, you will have to write IDL scripts, shell scripts, or Python scripts, which provide each tool with both the necessary and the optional input parameters. The following tools can be setup in this way:

• The master-bias image creation tool, "p3d_cmbias.pro".

• The trace-mask image creation tool, "p3d_ctrace.pro".

• The dispersion-mask image creation tool, "p3d_cdmask.pro".

• The fiber-flat image creation tool, "p3d_cflatf.pro".

• The science-object spectrum extraction tool, "p3d_cobjex.pro".

• The sensitivity-function creation tool "p3d_fluxsens.pro". This tool is available in the spectrum viewer as well as from the command line or from the shell.

• The flux-calibration tool "p3d_fluxcal.pro".

• The VIMOS extracted images quadrant combination tool, "p3d_cvimos_combine.pro".

• The spectrum viewer, "p3d_sv.pro".

• The single image cosmic-ray hit cleaning tool, "p3d_cr.pro"; in the GUI, this tool is only available with the science-object extraction tool "p3d_cobjex.pro". From the command-line it is also available with "p3d_ctrace.pro", "p3d_cdmask.pro", and "p3d_cflatf.pro".

• The differential atmospheric refraction correction tool, "p3d_darc.pro". This tool is only available from the command line or from the shell.

• The RSS to cube conversion tool "p3d_rss2cube.pro"; this tool is only available from the command line or from the shell.

• The exposures combination tool "p3d_cexposure.pro"; this tool is only available from the command line or from the shell.

• The automatic reductions tool "p3d_autoreduce.pro"; this tool is only available from the command line or from the shell.

• The line-fitting tool "p3d_ifsfit" and the associated plotting tool "p3d_ifsfit_mosaic.pro"; these tools are only available from the command line or from the shell.

Using p3d with the IDL Virtual Machine or with an IDL license

As is described in the README file, p3d can be used either with an IDL license from the command prompt, or from the shell or Python with (or without) the IDL Virtual Machine (IDL VM):

• No IDL license is required when using the IDL VM, although it is still necessary to either use the p3d-specific IDL distribution or install IDL on the computer where p3d is to be used (and this before starting p3d).

• The functionality of all routines of p3d is the same in either approach. However, if you need to modify the code because of, for example, a bug or an additional feature you want to include, it is necessary to recompile p3d – and this step requires an IDL license. If you only need to change an instrument- or method-specific parameter, you should not edit the code, but instead edit your user-parameter file (see below).

• A bug might – depending on its form and location in the code – terminate p3d when it is run using the IDL VM from a shell or Python. Therefore, if you use p3d with the IDL VM from the shell, make sure that the DEBUG option in the Options menu is inactive (this is always the case when the provided scripts are used).

• There are no differences in the execution speed of either approach. Note, however, that the graphics of some of the GUIs have been found to run up to two times faster using IDL version >= 7.0, instead of the older version 6.2.

Detailed descriptions on how to setup p3d to be used in either approach are provided in the README file.

The graphical user interface of p3d

The different graphical tools of p3d look very similar for all instruments, although several GUIs are scaled to make full use of your screen. If you do not tell p3d which instrument to set it up for when you launch it, you will be presented with an instrument-selection GUI before the main control panel is shown. Both these GUIs use a fixed size.

The instrument-selection GUI

This GUI allows you to start p3d configured for the instrument that you select here. Only the fully configured instruments can be selected, by clicking on the respective row under the 'Quick instrument setup' tab. Only the fully configured instruments are available; the remaining ones can also be selected if p3d is started with the debug option. The options in the remaining two tabs ('Program and data paths' and 'Configuration files') allow you to select a data directory and choose the instrument-parameter file manually.

If you do not want to select the instrument each time you start p3d using the GUI, you can use the appropriate keyword on the command line. Check the list of keywords below or see the last column in the instrument-selection GUI

Note: Although you launch p3d using your own instrument-specific configuration files for an unsupported instruments, some parts of p3d will likely not work as intended, such as the wavelength calibration. The reason to this is that many instruments use their own unique approach of specifying important information – some (often minute) changes are required in p3d to make these instruments work.

Note: Creating figures and images with color can be a hassle, which outcome depends on if you are using indexed (256 – "device, decomposed=0") or decomposed (16777216 – "device, decomposed=1") colors. p3d handles each case transparently for you – it is unnecessary to change the decomposed state using the IDL DEVICE command. If you do not have a clue what this is all about, do not worry!

The main control panel, and related GUIs

From the top, the different regions in the main control panel GUI are: the menu bar; a region with six rows, one for each data-reduction task; an object spectra preview section with a big window of the currently selected object on the left-hand side, and one smaller window for each of nine buffers; a status row with a log-file indicator on the left-hand side, and a status field on the right-hand side.

Here is a more detailed description of the different fields and options:

• The menu bar

  The menu bar looks the same on all platforms. Here follows a description of all the available (and unavailable) entries. The string in the last column shows the key-accelerator combination for those tasks where it is available. As an example of a key accelerator 'Ctrl+B' means that while you press down the 'Ctrl' key you also press down the 'B' key (thereafter both keys are released).

  The 'File' sub menu field

  • Create a new master-bias image (Ctrl+B)

  • Create a new trace-mask image (Ctrl+T)

  • Create a new dispersion-mask image (Ctrl+M)

  • Create a new fiber-flat image (Ctrl+F)

  • Extract new object data (Ctrl+O)

  • Flux calibrate extracted data (Ctrl+G)

  • Open an existing master-bias image (Ctrl+Shift+B)

  • Open an existing trace-mask image (Ctrl+Shift+T)

  • Open an existing dispersion-mask image (Ctrl+Shift+M)

  • Open an existing fiber-flat image (Ctrl+Shift+F)

  • Open an existing extracted object (Ctrl+Shift+O)

  • Open an existing flux-calibrated object image (Ctrl+Shift+G)

    The function of these options are described in the "data-reduction tasks" section below.

  • Restart p3d

    Restarts p3d in order to, for example, use another instrument configuration.

    Note: All loaded information disappears when p3d is restarted.

  • Exit (Ctrl+Q)

    Exits p3d.

  The 'View' sub menu field

  • Evaluate telescope focus series

    n/a

  • View a data log file

    Allows the selection of a log file from your observations that is opened in a separate GUI (using the IDL routine XDISPLAYFILE).

  • View the data reduction log file

    Opens a GUI showing the contents of the logfile of your reductions (using the IDL routine XDISPLAYFILE). By default, the file is shown as it is when this option is first selected. In order to update the GUI, use the next option.

    Note: You have to open a logfile before you can use this option [Options->Open a log file].

    Note: Huge files are not opened correctly using XDISPLAYFILE, so if you have such files then consider looking at them outside p3d, using, for example, a shell tool.

  • Set the log view timer [off]

    If you are viewing a logfile using the above option, this option allows to set the GUI-update timer to 5, 20, or 60 s.

  • Select the previous detector (Ctrl+R)

    Select a lower-index detector with instrument configurations that use several detectors.

  • Select the next detector (Ctrl+E)

    Select a higher-index detector with instrument configurations that use several detectors.

  • Select the color table

    Allows you to select the color table used to color all images (axes, labels, and some additional indicators are always drawn with a separate set of default colors).

    Note: The color table does not affect data in any way, but is only used to represent data in a revealing (and appealing) way – so choose whichever color table that makes you satisfied.

    Note: The cubehelix, and two Califa color tables, as well as the (default) Sauron color table are only available from the menu, while the color tables, which are available with the IDL routine LOADCT, can also be selected and modified interactively with XLOADCT.

  The 'Options' sub menu field

  • Using the raw data directory

  • Using a separate output directory

    p3d, by default, writes all output data to the raw-data directory. If you instead want all output data to be written to a separate directory, you can select that directory in the file selector GUI that pops up by selecting this option.

    Note: A faster option than this, is to start p3d from the directory where you want all the output data using the keyword /CWD.

  • Number of used threads: X

    The C routines of p3d normally use all available threads. Choose any of the numbers in the popup menu to use as many threads instead.

  • View the instrument parameter file (Ctrl+I)

    This option pops up a window showing the contents of the (currently used) instrument-specific parameter file.

  • Select a user-parameter file (Ctrl+U)

    Opens a file selector that allows you to pick a user parameter file that can be used to configure all data-reduction tasks. If you do not select an existing file, p3d will copy the master user-parameter file (from ${p3d_path}/data/master_userparfile.dat) to the specified directory, and then use that file.

  • View the user-parameter file (Ctrl+Y)

    This option pops up a window showing the contents of the selected user-parameter file.

    Note: This option is only available after a user-parameter file has been selected.

  • Edit the user-parameter file (Ctrl+A)

    This option pops up a window that allows you to both view and edit the selected user-parameter file.

    Note: This option is only available after a user-parameter file has been selected.

  • D.mask rebin data [dbin=1]

    This option is used when the dispersion-mask creation GUI (dGUI) is opened. The values that can be selected are:

    1.	Data are not rebinned, this is the default.
    2.	Data are rebinned once.
    3.	Data are rebinned until they fit the screen.

    The meaning behind these options is the following. Your screen has a number of pixels on the horizontal axis that may be smaller than the number of pixels on the dispersion axis in your data. With the default setting (dbin=1) your data are kept as they are, and a horizontal scrollbar is added in the dGUI, if the monitor has fewer pixels than the data.

    If the second option is used (dbin=2), your data are rebinned/rescaled to use half as many pixels on the x-axis. This is normally enough to fit all data on the screen (although most likely not with data of the GMOS IFS). With the third option, your data are rescaled by a factor two until all data fit on your screen.

    A good reason to not bin data is that pixel positions become inaccurate when data are rescaled. The recommended option is therefore to not bin data – keep dbin=1!

  • Spectrum viewer – window size [xxxx, yyyy]

    The spectrum-viewer window is normally opened using the maximum number of pixels available on the screen; the number of pixels corresponds to the integer doublet [xxxx, yyyy]. A set of additional pre-defined window sizes is available in the popup menu. Choose any of the available sizes to use that one instead.

  • Spectrum viewer – view sky fibers [no]

    Some of the instruments, for example PPAK, FLAMES, and GMOS, make use of separate sky fibers. The location of these sky fibers are normally not seen in the spatial map of the spectrum viewer. Use this option to change the behavior and also show the sky fibers in the spatial map.

    Note: This could be a useful option to see if your sky fibers are in fact on the sky, or if they point at some bright object, in which case they should not be used to subtract the sky component in your analysis.

    Note: In most cases, it is unlikely that you want to view the sky fibers, because all elements in the spatial map will become smaller when the sky fibers are also shown in the same window.

  • Append to a log file (Ctrl+J)

    Select this option to append log output to an existing log file. When you select this option, a file selector GUI will ask you for an existing (plain-text) file.

    Note: This option is only available if no log file has been opened.

  • Open a log file (Ctrl+L)

    Select this option to open a new log file; a file-selector GUI will query you for a filename.

    Note: The log file will be deleted if it already exists.

    Note: This option is only available if no log file has been opened.

  • Close the log file (Ctrl+K)

    Select this option to close the log file.

    Note: IDL writes to the file when the I/O buffer is full, so unless you close the file you may not see the final operations. (This file is closed automatically when you exit p3d.)

    Note: This option is only available if a log file has been opened.

  • Log level [1]

    The default log level is 1, which writes the most important information to the log file. The second option (2) writes more information to the log file. The third option (3) writes all information to the log file (this should only be a valuable option when debugging a routine).

  • Verbose level [0]

    This option allows you to choose the amount of information written to the standard output (STDOUT). By default, this setting is 'no information'. The other options are the same as for the logging level.

    Note: This option is typically used when debugging some part of p3d, it is more wise to save the information to a log file.

  • Debug [0 {=no}]

    The normal behavior of p3d is to leave any routine when an error is encountered, and return to the main execution level, which allows you to redo your task or do something different. If you instead want p3d to stop when it finds an error, you should use this option.

    Note: Please, also use this option when reporting a bug you have found. It makes it much easier to track down a bug if you copy and paste the output on the console where p3d stopped in your bug report.

    Note: If you use the IDL VM, you should keep this option switched off, otherwise p3d will stop promptly if, and when, a bug is encountered.

  The 'Help' sub menu field

  • About

    Opens a window with some information about p3d.

  • Copying

    Opens a window showing the contents of the GNU license.

  Note: IDL, and therefore p3d, is based on the Motif widget library (http://www.opengroup.org/motif/), which requires key accelerators (on UNIX platforms) to be placed in menu entries. Most options are therefore available in the menus listed above. Although, key accelerators are defined for most options, your window manager (such as, for example, KDE or GNOME) may have reserved any particular configuration, and if this is the case the accelerator will not work.

• The data-reduction tasks

  There are six rows in the data-reduction section of the GUI.

  The label in the first column of each row indicates the task; the six available tasks are: Master bias, Trace mask, Disp. mask, Fiber flat, Object, and Flux cal. The second column shows the filename of the currently selected data set. The third column indicates the currently selected buffer, the numbers are 0 – 8, for buffers 1 – 9. By clicking on the icons in the remaining columns, the following operations are performed:

  • Create a new dataset (empty sheet of paper icon)

    Click this icon to open a file selector where you can choose one, or three or more, raw-data images.

    Note: After the new dataset has been created, p3d will present a GUI where you should select one of the nine buffers where the output data are stored.

    Note: The images must be of the correct type for the desired function. p3d does check if the image size is correct, but does not know how to differ between, for example, an arc image and a bias image.

  • Open an existing data set

    Click this icon to open a file selector where you can choose one already reduced image.

    Note: As soon as the data set has been loaded, p3d will present a GUI where you should select one of the nine buffers where data are stored.

  • Switch between different, and already loaded, data sets

    Click this icon to select the active data set buffer.

    Note: For object data, it is also possible to select the active data set by clicking in the respective panel, of the GUI object-preview section.

    Note: This option is only available if two or more data sets were created or loaded. If only one data set was loaded, that is the active data set.

  • View the loaded data in the p3d image viewer (landscape icon)

    When you click this icon, the p3d image viewer will present you with a simple GUI showing the raw-data image as well as the reduced image in two separate tabs.

    Note: For this option to work properly, the necessary header keywords must be present in the reduced image. p3d adds those keywords automatically for each image type.

    Note: When viewing the trace-mask image, the traces are drawn on top of the spectra in the raw-data image (so the trace 'image' is not actually shown).

  • View the loaded data in the p3d spectrum viewer (diagram icon)

    When you click this icon, the p3d spectrum viewer will present you with a GUI showing the spectrum image as well as a spatial map, and a spectrum that is a combination of any number of selected spatial elements. This option is only available for reduced images that contain spectra, i.e., dispersion-mask images, fiber-flat images, and extracted object spectra images.

    The beam-switched data of files that also contain such beam-switched data can be viewed by clicking on the second diagram icon when it is sensitive; this option is only available with a few instruments (such as the MRS2).

    When viewing the dispersion mask, what is actually shown is the extracted and wavelength-calibrated spectra (where the wavelength calibration is done on the fly using the dispersion-mask image).

    Note: If a dispersion-mask image is selected, the flat-field image will be wavelength calibrated before it is shown (this is otherwise normally only done when the flat-field image is applied to the object spectrum image).

  • View the loaded data with the image viewer ds9 (sun icon)

    When you click on this icon, ds9 is launched with the loaded data, as well as any error data, cosmic-ray mask, and saturated-pixels mask image in separate frames. This option is only available if ds9 is found in the system path when p3d is launched. ds9 can be downloaded from its project web site: http://hea-www.harvard.edu/RD/ds9/site/Home.html

  p3d stores the loaded data in nine separate buffers. It is always the selected buffer – indicated with the filename and the buffer number in columns 2 and 3 in the image above – that is used in the data-reduction steps. The buffer-selection GUI looks like the image on the right-hand side. Simply click in any of the nine fields, and then click 'OK' in the lower left corner. If you change your mind, and do not want to save the data set, click 'Cancel' in the lower-right corner instead.

  The first steps of the data-reduction procedure are meant to be performed in the following order:

  1.	Create a master-bias image, unless the prescan and/or overscan regions are used instead.
  2.	Create a trace-mask image, and optionally calculate line profiles for use in the optimal extraction.
  3.	Create a dispersion-mask image; this step is optional. It is not possible to create a dispersion-mask image before a trace-mask image has been created or loaded into a buffer. It is, however, possible to load an already created dispersion-mask image at any time.
  4.	Create a fiber-flat image; this step is optional. It is not possible to create a fiber-flat image before a trace-mask image has been created or loaded into a buffer. It is, however, possible to load an already created fiber-flat image at any time. When you select this option, a setup interface is shown that allows you to configure the flat-field extraction options (see below for a description of this interface).
  5.	Extract object data. It is not possible to extract spectra in an object-spectrum image before a trace-mask image has been created or loaded into a buffer. When you select this option, a setup interface is shown that allows you to configure the spectrum extraction (see below for a description of this interface). Note: The wavelength-calibration step is optional, to use this functionality you should create or load a dispersion-mask image before starting this task. Note: Normalization of data using a fiber-flat image is optional. To use this functionality, you should create or load a fiber-flat image before starting this task.
  6.	Flux calibrate data; this step is optional. To flux calibrate data, it is necessary to first reduce data of observations of a standard star. The extracted standard-star data are opened with the spectrum viewer to first subtract the sky, and then co-add all spectra that contain flux of the standard star. A single standard-star spectrum can be used to create a sensitivity function using the tool p3d_fluxsens or the sensitivity-function tab in the spectrum viewer. The sensitivity function is then used with the flux-calibration tool p3d_fluxcal to calibrate the extracted spectra.

  • The dispersion-mask configuration GUI

    The dispersion-mask configuration GUI is shown when selecting 'Create a dispersion-mask image'. Here is a description of the different components of this interface, the items are described from the top:

    Raw arc image

    Select the arc raw-data image that will be used to create a dispersion mask. If you select several files, they are at first combined into one image.

    Do not use an input dispersion-mask image

    The normal procedure is to create a dispersion-mask image from scratch, which can be quite tedious work if there are many images to create. Unselect this option and choose an already created dispersion-mask image, to shorten the procedure.

    Note: It is important that the selected dispersion-mask image comes from data that were taken with the exact same instrument configuration as the currently selected raw-data image. Otherwise, it might be very difficult to create a new dispersion mask.

    Use the default arc-line list file

    By default, p3d attempts to select a default line-list file, which is stored in the directory ${p3d_path}/data/tables/linelists/. Unselect this option, to choose an already prepared line-list file yourself.

  • The fiber-flat configuration GUI

    The fiber-flat field GUI is shown when selecting 'Create a fiber-flat image'. Here is a description of the different components of this interface, the items are described from the top:

    Raw flat-field image

    Select the flat field raw-data image that will be used to normalize data. If you select several files, they are at first combined into one image.

    Use the FF raw data for calc. a transmission array

    Unselect this option to choose a file that contains a separately calculated fiber-to-fiber transmission array. Such files are created when a flat-field image is extracted, and are either stored in the TRANSMISSION extension, or in a separate file, which typically has the suffix '_transmission'.

    Use the FF raw data for finding spectrum traces

    By default, the fiber-flat data are themselves used to trace spectra, before the object data are normalized in the object-extraction task. By unselecting this option, you can choose an, already created, trace-mask image instead.

    Note: With twilight flat-field images, you would normally leave this option selected. You might want to unselect this option and choose an already created trace-mask image if you intend to use, for example, a continuum-lamp image to flat field your data; where the same continuum-lamp image was used to calculate the trace-mask image.

    Do not use a dispersion mask with the FF data

    By default, the extracted fiber-flat spectra are wavelength-calibrated using the dispersion-mask image that is selected in the p3d main GUI. If this option is unselected, you can choose a separate, and already created, dispersion-mask image.

    Note: This option is useful if you want to use a separate dispersion-mask image with, for example, your twilight-flat-field images.

  • The object spectrum extraction configuration GUI

    The object-extraction setup GUI is shown when selecting 'Extract object data'. Here is a description of the different components of this interface, the items are described from the top:

    Extract file

    Select the object data that will be extracted. If several files are selected, they are at first combined into one image.

    Subtract the master-bias image

    Select this option to subtract the master-bias image from the raw data. This option must not be selected if prescan and overscan regions are to be used instead when estimating the bias level.

    Note: This option is obligatory if p3d is to calculate an error estimate for the extracted spectra, and no prescan and overscan regions are used to estimate the bias level. In the latter case, the bias error is set to zero.

    Remove cosmic ray hits (when one file is selected)

    Select this option to clean images from cosmic-ray hits; this option has the same effect as setting the keyword CRCLEAN. If you want to use a different configuration in the cosmic ray cleaning than the default setup, it is necessary to use the options in the user-parameter file.

    Wavelength calibrate using the dispersion-mask image

    If you want to wavelength-calibrate your data, select this option.

    Note: The wavelength calibration is applied after the spectra are extracted, and before (the optional) normalization with the fiber-flat image.

    Prompt for wavelength-related properties

    p3d calculates and uses default values for the wavelength array of the data. You can apply your custom values either by entering them in the user-parameter file, or by entering the values in a (self-explanatory) GUI that is shown when selecting this option.

    Align the disp.mask with telluric lines

    Because of flexure in the instrument the data may be shifted in the dispersion direction in the time elapsed between an arc image and an object image. By selecting this option, p3d calculates a median offset of telluric lines that are fitted in the data, and this offset is then applied to the dispersion-mask image. There are three options here:

    • "none|user par.file", no alignment is performed, or the telluric line list specified in the user parameter is used.

    • "Lo-res", uses the low-resolution line list "telluric_lines_lores.dat" (part of the p3d distribution).

    • "Hi-res", uses the high-resolution line list "telluric_lines_hires.dat" (part of the p3d distribution).

    Normalize the resulting spectra using the flat field

    Select this option to normalize the extracted spectra with the selected fiber-flat image.

    Export resulting spectra (also) in the E3D-format

    Select this option to save the outcome in the E3d-format.

    Note: p3d always saves the outcome in the row-stacked-spectrum (RSS) format, which requires a fiber position-table file to reconstruct a spatial map. The E3d-file includes this table in the file header.

    Note: p3d does not read E3D-files.

  • The flux-calibration configuration GUI

    The flux-calibration GUI is shown when selecting 'Flux calibrate extracted data'. Here is a description of the different components of this interface; the items are described from the top:

    Sensitivity-function file

    Select a single sensitivity-function file. This file must be saved in the FITS format, as described in the tool p3d_sensfunc.

    Extinction-data file

    Select a single extinction-data file. See the routine p3d_misc_fluxcal_atmospheric_extinction.pro for details on the accepted format of this file.

    Attempt a qualified quess of the wavelength unit

    Use this option to manually set the wavelength unit used with extinction data provided in a text file; this option corresponds to the keyword WAVEUNIT in p3d_misc_fluxcal_atmospheric_extinction.pro.

    Export resulting spectra (also) in the E3D-format

    Select this option to save the outcome in the E3d-format.

    Note: p3d always saves the outcome in the row-stacked-spectrum (RSS) format, which requires a fiber position-table file to reconstruct a spatial map. The E3d-file includes this table in the file header.

    Note: p3d does not read E3D-files.

• The object spectrum preview section

  Once an object is extracted – or an already extracted object image is loaded – the panels in this section of the GUI show a spatial map of the data. In a spatial map, each element represents a spatial element on the integral-field unit, and the color represents the intensity at the chosen wavelength. The spatial-map preview is only a preview of what can be seen in more detail in the spectrum viewer, but it allows a quick check of the outcome without starting the spectrum viewer.

  The large panel on the left-hand side shows a spatial map of the active/currently selected object. Specifically, the spatial map is shown for a pre-defined wavelength (bin). In the row of widgets below the panel, the leftmost text field shows the currently selected wavelength bin (in pixels; the default is the center pixel); the label immediately to the right shows the corresponding wavelength of that bin (in Ångström). The following two text fields indicate the minimum and the maximum values used when calculating the color of each spatial element. The rightmost widget allows you to select the range of values used to determine the minimum and the maximum values (using a histogram). The default of 95% means that in a histogram of the intensity value of all spatial elements, of all wavelength bins, the minimum value is set above 2.5% of the lowest value bins, and the maximum value is set below 2.5% of the highest value bins. All values in the text fields can be changed by typing in a value manually, and thereafter pressing return. (The fields are also updated when the respective widget loses the focus.) The wavelength bin can also be changed easily by using the slider widget on the left-hand side of the panel.

  Each panel on the right-hand side shows a spatial map of one of the nine buffers. Black panels indicate buffers where no data have been loaded. An additional way to switch between data sets - in addition to using the selector widget on the object-extraction row in the data-reduction section of the GUI – is to click in the panel that should become the active data set.

• The status field

  The status line of the p3d main control panel shows you what p3d is doing, or can do. The text field on the right-hand side shows a descriptive text of the function of each button and field widget – whenever the mouse pointer is kept above the respective widget. The log-file field on the right-hand side is set to 'Log: yes' ('Log: no') when a (no) log file has been opened. If the mouse pointer is kept above this region, the text field on the right-hand side will show the name of the currently opened log file.

The image viewer

The image viewer of p3d is available to allow a quick view and comparison between raw data and the location of spectrum traces (for all tasks, but the master bias). When you click on the image viewer icon, you are presented with a window that is adjusted to fill your screen; unless the image of your particular detector is smaller than your screen.

Two images are shown in two separate tabs. The raw data are shown in the leftmost tab, and the same raw data with overplotted spectrum traces in the rightmost tab. For large images, scroll bars are available to the right and below the image to allow you to look at different parts of the image.

The widgets below the image(s) allow you to control the presentation of the image. The four fields are:

• Alpha

  Drag the scrollbar from 0.0 – 1.0 to change the transparency of the overlaid traces. The alpha threshold value can also be entered in the adjacent text field; the default value is 0.5. The transparency calculations are slow, so be careful with dragging the scrollbar with large images.

• Value range

  The colors are calculated using a minimum and a maximum value, applying a pre-defined color table. Enter the minimum and the maximum values in the text widgets, or select a value in the drop list. A percentage of 97% (this is the default value) means that in a histogram of the intensity value of all pixels, the minimum value is set at 1.5% above the lowest value bins, and the maximum value is set at 1.5% below the highest value bins.

• X, Y

  The values in the label widget show the current position of the mouse pointer when it is dragged across the image. The values are counted from the bottom left corner of the image.

• statistics

  Image statistics are shown for a 3x3 pixel box centered on the current position of the mouse pointer. From the left-hand side, the five values are: the maximum, the minimum, the mean, the median, and the standard deviation.

Note: With multi-block instruments (such as the 4kx4k CCD of PMAS), the different blocks may be read out using a different gain value. If this is the case, and the gain values differ significantly, it could be tricky to set the minimum and the maximum values so that the relevant data are seen easily in all parts (i.e. blocks) of the image.

Note: In several cases, the pre-scan and/or over-scan regions are not removed in the raw image shown in the image viewer. If this is the case, the traces are also shown in these regions, where they, of course, are unimportant.

Note: The image viewer determines which raw data file and trace mask to use from custom header keywords. These keywords are added by p3d when the images are created, so if you use an image that was not created by p3d then the image viewer will (unlikely) find the required information.

Note: The transparency calculations performed when overlaying the traces are slow, and they also need huge amounts of memory. For data of the GMOS IFS you will need >= 4GB of RAM, or your swap-disk will go warm for a while (2GB RAM and 2GB swap is the absolute minimum requirement).

The spectrum viewer

See the file p3d_sv.pro for a full description of all parts and functionality of the spectrum viewer.

Configuring p3d

p3d is designed to be a general data-reduction tool that works with all fiber-fed integral-field spectrographs. All parts of p3d are configured for the separate instruments using instrument-specific configuration files. There are two kinds of configuration files: the instrument-specific configuration files that should not be modified, and a user-parameter file that should be modified to meet any requirements. All these files are, and must be, available in plain-text format. All provided files are typed with 90 columns to show maximum verbosity.

Hint!

Please, read the relevant configuration files, the p3d A&A paper, and the p3d SPIE paper carefully to understand how p3d is configured for your instrument, and to see which parameters you can modify.

Hint!

If you want to print out a parameter file to have a closer look, you could – on *NIX platforms – create an nicely appearing (postscript) printout with the a2ps command. Here are two examples on how to use this command from the p3d directory:

a2ps --columns=2 -o tmp.ps -l 90 data/instruments/pmas/larr4k.prm

or

a2ps -R --columns=1 -o tmp.ps -l 120 \
  data/instruments/pmas/larr4k.prm

You would then print the file tmp.ps (using e.g. 'lp -c tmp.ps').

Note: While all instrument-specific default options are setup automatically, the user-specific reduction parameters are always read from the user-parameter file. Hence, to modify your data reduction, you must provide such a file; either by opening a file from the options menu in the GUI, or by specifying the name of an existing file on the command line (using the input parameter USERPARFILE). The user-parameter file can be edited using the GUI.

The instrument parameter file

These files are all stored in the instrument-specific directories (under ${p3d_path}/data/instruments/). The main configuration file is, for each instrument, the one with the ending '.prm'. All parameters are collected in groups of the same function. The seven parameter groups are:

The meaning of each parameter is described in the file, and some parameters are described in more detail in the p3d papers. Most importantly, all parameters indicated with an asterisk ('*') can, if necessary, be modified from the user-parameter file (see below).

The user-parameter file

When you need to change the behavior of the data-reduction procedure of p3d, the correct way is to set tool-specific keywords, or to edit the user-parameter file. If you use the GUI, p3d will copy a master user-parameter file when a non-existent file is selected. The master user-parameter file is copied to the data-reduction directory from ${p3d_path}/data/master_userparfile.dat. The file can be edited from outside p3d, or it can be edited using the 'Edit user-parameter file' option in the menu.

As is the case with the instrument-specific parameter file, this file is divided into separate categories of parameters. The fourteen different categories are:

Note: The user-parameter file is read anew every time a data-reduction function is launched. This means that if you make a change in the file you only have to run the task where you made a parameter change anew – there is no need to restart p3d.

Note: The (NASA astro-lib) routine READCOL will issue a warning if a user-parameter file is selected where all entries are commented out (i.e. all rows begin with a semi-colon). The warning will not affect the data-reduction process.

General information on how to learn more about this routine

Instrument parameter files (*.prm), which are required to run this routine, are found in the directory "!p3d_path/data/instruments/*/" (outside IDL check "${p3d_path}/data/instruments/*/").

A default (master) user-parameter file is available in "!p3d_path/data/master_userparfile.dat". Copy this file to your data-reduction output directory, rename the file to, for example, "user_p3d.dat" and edit it to suit your needs.

Examples on how to use this routine are provided in the EXAMPLES section below.

See the file README in the root of the p3d package for installation instructions, if p3d is not already installed on the computer.

Additional information, documentation, new versions, etc., can be found at the p3d project web site "https://p3d.sourceforge.io".

Keyword parameters:

	path		- This input string defines the default directory to look for data inside (used by all file dialogs). The default value is: !p3d_path
	cwd		If this keyword is set then all output files are saved in the current directory, instead of in the raw data directory (this is the default).
	sfx		A scalar string that specifies the raw data filename suffix to use. The default value is: .fits
	detector		This scalar integer selects the detector to use, if the instrument is configured in a multi-detector configuration. For example, VIMOS has four detectors, but the Mitchell Spectrograph [VIRUS-P] is also configured for its both fiber bundles using the same instrument-parameter file. In these, and similar, cases it is possible to specify the starting detector/IFU by providing the DETECTOR keyword. The first detector/IFU is 0, the second 1, etc.
	flames,		With these keywords it is possible to selects the nstrument setup (LARR2K is the default). An nstrument-selection GUI is shown if none of these eywords is set.
	userparfile		This scalar string defines a file that is used for user-specific parameters. If this keyword is set here then the corresponding file must already exist (in the specified path). If the file does not yet exist then copy the master user-parameter file (${p3d_path}/data/master_userparfile.dat) to your output directory and provide the new filename, including the path, in this parameter.
	logfile		A scalar string that specifies the name of a logfile. If a file already exists, with the name LOGFILE, it is overwritten. Use this keyword to save the log output of the various routines of p3d. Use the VERBOSE keyword instead to only print information on the command line or in the terminal.
	loglevel		A scalar integer, which determines the amount of information recorded in the log file LOGFILE. LOGLEVEL can be set to 1-3, where a higher number means that more information is written to the log file.
	setuponly	X	Set this keyword to exit this tool after setting up the IDL !path to include all parts of p3d; this could be useful when one only wants to use one or some of the subroutines of p3d, but not the tools themselves.
	colortable		A scalar string or integer that specifies which color table should be loaded. If COLORTABLE is a string, the string must contain two parts separated by a comma, without any whitespace between. The first part before the comma specifies the color-table number in the file name specified after the comma; the file must be formatted as described in the documentation for MODIFYCT. As an example, COLORTABLE='25,brewer.tbl'. Components in the ColorBrewer color-table file "brewer.tbl" found in the resource directory can be specified explicitly by preceeding the string with "CB"; for example, COLORTABLE='CB25'. If COLORTABLE is an integer, the integer must be given in the range -4, -3, -2, -1, 0, ..., up to the maximum number of available tables in IDL, plus the available tables in the ColorBrewer file (resource/) "brewer.tbl". Select -4, -3, -2, or -1 to load the cubehelix, 2 * Califa, and the Sauron color tables, the other values use the respective color map as defined by LOADCT. These are the permitted values: -4 Loads the cubehelix color table. -3 Loads the Califa project intensity color table, as defined September 2012. -2 Loads the Califa project velocity field color table, as defined September 2012. -1 Loads the Sauron color table (default). 0-74 (IDL version >= 8.3) Loads the corresponding color table with LOADCT. 0-40 (IDL version < 8.3) Loads the corresponding color table with LOADCT. 'CB0'-'CB85' Loads the corresponding ColorBrewer color table; the color tables are defined in the file "resource/brewer.tbl" 75-159 (IDL version >= 8.3) Loads the ColorBrewer color table corresponding to this number after 75 is subtracted; the color tables are defined in the file "resource/brewer.tbl" 41-125 (IDL version < 8.3) Loads the ColorBrewer color table corresponding to this number after 41 is subtracted; the color tables are defined in the file "resource/brewer.tbl" 'x,ctfile' Loads the x:th entry ('x' must be an integer) in the color-table file 'ctfile. The default value is: -1
	invert		Set this keyword to invert the loaded color table.
	bottom		A scalar integer that specifies the lower index of the color table used to scale the color map (the upper index is the maximum possible; see p3d_misc_colortable).
	cindex		An array of integers specifying which color indices to use for reserved colors. It doesn't make sense to use more than 4 elements in the CINDEX array. The indices should also be set as low as possible in order to allow the remaining indices to be used by the (scaled) COLORTABLE (see p3d_misc_colortable).
	title		This scalar string is used as the title of the GUI. Only use this parameter if you do not want to use any of the pre-configured instrument setups.
	parfile		This scalar string can be set to the name of a file that contains instrument-specific parameters for use in p3d. Only set this parameter if none of the pre-configured instrument setups work.
	display_name		Set this keyword equal to a string that specifies the name of the X Windows display on which the base should be displayed; this keyword has no effect on Microsoft Windows platforms.
	font		Set this scalar string to the name of the font to be used with all widget components of this tool. It is advised to use a fixed font, the outcome is otherwise difficult to predict. Set the keyword to 'big' to use the following pre-defined font ed-medium-r-normal--18-120-100-100-c-90-iso8859-1'; this could be useful on a system using a 4k screen (with 3160x2560 pixels). Another option to specify a font is to set the environment variable p3d_font or the p3d IDL system variable !p3d_font. The default font used with the widgets is, for your information, printed to the screen when FONT is unset and the highest verbosity level is used (verbose = 3). On Linux machines, use a shell command such as 'xlsfonts' to list all available fonts.
	cmdline	X	This keyword is used when the tool is launched by the p3d server or using the IDL VM.
	verbose		Set this parameter to a positive scalar integer to make p3d write some information on STDOUT about what is going on. The following four values are acknowledged: 0 Writes no information at all. This is the default. 1 Writes the more important information; regarding subroutine configurations, mostly. 2 Writes most information; includes a more verbose output than 1. 3 Writes all information, including information on the execution state of GUI subroutines. This may be a useful mode when debugging the code.
	quiet		Set this keyword to not print the disclaimer on the screen when p3d starts up.
	debug	X	No error handler is setup if this keyword is set. The default is otherwise to setup an error handler (using the routine CATCH), which makes each subroutine exit quietly in case a bug is encountered. Use this keyword when debugging p3d.
	help	X	Set this keyword to show this routine documentation, and then exit.

Examples:

This routine can be called in three different ways (for use on WINDOWS platforms, replace the path character '/' with '\'):

i) From the IDL command line

It is possible to start p3d without using any input parameter; all parameters can be set interactively. However, it may be more convenient to give the options already on the command line.

Recommended options are: the instrument keyword, the keyword specifying if the current directory is the output directory (/cwd), and the name of a(n output) log file:

userparfile = './user_p3d.dat' ;; This file must exist if
                               ;; it is specified here.
p3d, /cwd, /mitchell, detector=0, userparfile=userparfile, $
    logfile='dred.log', loglevel=2, /verbose

Here are some additional examples of how to start the p3d GUI from the command line:

p3d, /debug    ;; Start p3d with the DEBUG option, to allow all
               ;;   configured instruments to be selected.
p3d, /ppak4k   ;; Start p3d for the PMAS/PPAK/4kx4k CCD instrument.
p3d, /mitchell ;; Start p3d for the Mitchell Spectrograph [VIRUS-P]
               ;;   instrument, which is setup for the newer fiber
               ;;   bundle 2.
p3d, /larr2k, /debug, /cwd, logfile='test.log', loglevel=1
               ;; Start p3d for the PMAS/LARR/2kx4k CCD, turn on
               ;;   debugging, write all output to the current
               ;;   directory, write log messages to a file called
               ;;   'test.log', and set the logging level to 1.

ii) As a stand-alone tool from a terminal, using the IDL VM

As in the previous case, it is possible to start p3d without using any input parameter; all parameters can be set interactively. However, it may be more convenient to give the options already on the command line.

Recommended options are: the instrument keyword, the keyword specifying if the current directory is the output directory (/cwd), and the name of a(n output) logfile.

The recommended approach is to use the provided shell script p3d_vm.sh or the Python tool p3d_dispatch.py:

# Using the shell script, without using the p3d queue system:
userparfile=./user_p3d.dat # This file must exist if it
                           # is specified here.
p3d_vm.sh --noqueue /cwd /mitchell detector=0 \
    userparfile=$userparfile logfile=dred.log /verbose

# Using the Python tool, using the p3d queue system:
userparfile=./user_p3d.dat # This file must exist if it
                           # is specified here.
p3d_dispatch.py p3d /cwd /bvimos \
    userparfile=$userparfile logfile=dred.log /verbose

Note: In this approach, it is necessary to specify all options using the full names of the keywords. Partially correct keyword names are not permitted, as they are inside IDL. Hence, as an example, you must type "p3d_vm.sh userparfile=$userparfile" and -NOT- "p3d_vm.sh us=$userparfile".

Note: Simplify your reduction work by starting p3d from the reduction output directory.

Note: Parameters indicated with an 'X' in the list above are unavailable in this approach.

iii) As a stand-alone tool from Python, using the IDL VM

This approach is analogous to iii), but is executed from Python. Here is an example:

# Launch p3d in a single IDL session, without using the
# p3d queue system:
from p3d_dispatch import *
userparfile = 'user_p3d.dat'
commandline = '/mitchell detector=0 userparfile=' + userparfile +
  ' logfile=dred.log loglevel=2 /verbose nthreads=8'
p3d_dispatch('p3d', commandline, noqueue=True)

Once you've become accustomed to using p3d on a regular basis, it will be easier to start the tool with some extra keywords, so that you do not have to configure everything using the menus each time. Here are a few examples of useful startup calls, which can be used when running p3d with an IDL license. Most of these examples also work when p3d is used from the terminal (although some of the input parameters are unavailable in this case, see the list of parameters above for more details):

• The header of all routines of p3d can be accessed using the help keyword. The main routines to setup the main control panel of p3d are 'p3d.pro' and 'p3d_tool_gui.pro':

  p3d, /help
  p3d_tool_gui, /help

• Start p3d for the PPAK IFU of PMAS, for the new 4kx4k CCD. Additionally, instruct p3d to save all the outcome in the current directory, write all logging output to the file 'dred.log', and set the logging level to '1':

  p3d, /ppak4k, /cwd, logfile='dred.log', loglevel=1

• As the previous example, but use the Mitchell Spectrograph [VIRUS-P] instrument instead, and also use the already existing user-parameter file 'user_p3d.dat' in the current directory:

  p3d, /mitchell, /cwd, logfile='dred.log', loglevel=1, $
       userparfile='user_p3d.dat'

• When debugging p3d it is useful to use the debug option, as well as increasing the amount of textual output to the command line. Since the instrument is not selected it has to be chosen using the instrument-selection GUI (see above):

  p3d, logfile='dred.log', loglevel=2, verbose=3, $
       userparfile='user_p3d.dat', /debug

• If, in your opinion, using the right color table is important for a visually appealing data-reduction session, then you can set it using the COLORTABLE keyword. The following command uses the green/white linear color table (8) instead of the default Sauron color table (-1):

  p3d, colortable=8

  A separate color-table file can beloaded by specifying COLORTABLE as a string with an index of the color table to use in the file name provided after a comma, e.g.:

  p3d, colortable='25,brewer.tbl'

  The color-table file must be formatted according to the description in the documentation of the tool MODIFYCT. For more details, see p3d_misc_colortable.

  Here is a list of all available color tables in IDL. The first four entries are custom p3d color tables; IDL version < 8.3 only include the entries 0-40:

  -4 :             cubehelix         36 :               Volcano
  -3 :      Califa intensity         37 :                 Waves
  -2 : Califa velocity field         38 :             Rainbow18
  -1 :                Sauron         39 :       Rainbow + white
   0 :            B-W LINEAR         40 :       Rainbow + black
   1 :            BLUE/WHITE         41 :             CB-Accent >=8.3
   2 :       GRN-RED-BLU-WHT         42 :              CB-Dark2 >=8.3
   3 :       RED TEMPERATURE         43 :             CB-Paired >=8.3
   4 : BLUE/GREEN/RED/YELLOW         44 :            CB-Pastel1 >=8.3
   5 :          STD GAMMA-II         45 :            CB-Pastel2 >=8.3
   6 :                 PRISM         46 :               CB-Set1 >=8.3
   7 :            RED-PURPLE         47 :               CB-Set2 >=8.3
   8 :    GREEN/WHITE LINEAR         48 :               CB-Set3 >=8.3
   9 :   GRN/WHT EXPONENTIAL         49 :              CB-Blues >=8.3
  10 :            GREEN-PINK         50 :               CB-BuGn >=8.3
  11 :              BLUE-RED         51 :               CB-BuPu >=8.3
  12 :              16 LEVEL         52 :               CB-GnBu >=8.3
  13 :               RAINBOW         53 :             CB-Greens >=8.3
  14 :                 STEPS         54 :              CB-Greys >=8.3
  15 :         STERN SPECIAL         55 :            CB-Oranges >=8.3
  16 :                  Haze         56 :               CB-OrRd >=8.3
  17 :   Blue - Pastel - Red         57 :               CB-PuBu >=8.3
  18 :               Pastels         58 :             CB-PuBuGn >=8.3
  19 :   Hue Sat Lightness 1         59 :               CB-PuRd >=8.3
  20 :   Hue Sat Lightness 2         60 :            CB-Purples >=8.3
  21 :       Hue Sat Value 1         61 :               CB-RdPu >=8.3
  22 :       Hue Sat Value 2         62 :               CB-Reds >=8.3
  23 :  Purple-Red + Stripes         63 :               CB-YlGn >=8.3
  24 :                 Beach         64 :             CB-YlGnBu >=8.3
  25 :             Mac Style         65 :             CB-YlOrBr >=8.3
  26 :                 Eos A         66 :               CB-BrBG >=8.3
  27 :                 Eos B         67 :               CB-PiYG >=8.3
  28 :             Hardcandy         68 :               CB-PRGn >=8.3
  29 :                Nature         69 :               CB-PuOr >=8.3
  30 :                 Ocean         70 :               CB-RdBu >=8.3
  31 :            Peppermint         71 :               CB-RdGy >=8.3
  32 :                Plasma         72 :             CB-RdYlBu >=8.3
  33 :              Blue-Red         73 :             CB-RdYlGn >=8.3
  34 :               Rainbow         74 :           CB-Spectral >=8.3
  35 :            Blue Waves

  These are the ColorBrewer color tables available in the file 'brewer.tbl':

  CB00:         YlGn (Sequential)    CB43: Pastel1 (Qualitative) -  9
  CB01:       YlGnBu (Sequential)    CB44:    Set1 (Qualitative) -  3
  CB02:         GnBu (Sequential)    CB45:    Set1 (Qualitative) -  4
  CB03:         BuGn (Sequential)    CB46:    Set1 (Qualitative) -  5
  CB04:       PuBuGn (Sequential)    CB47:    Set1 (Qualitative) -  6
  CB05:         PuBu (Sequential)    CB48:    Set1 (Qualitative) -  7
  CB06:         BuPu (Sequential)    CB49:    Set1 (Qualitative) -  8
  CB07:         RdPu (Sequential)    CB50:    Set1 (Qualitative) -  9
  CB08:         PuRd (Sequential)    CB51: Pastel2 (Qualitative) -  3
  CB09:         OrRd (Sequential)    CB52: Pastel2 (Qualitative) -  4
  CB10:       YlOrRd (Sequential)    CB53: Pastel2 (Qualitative) -  5
  CB11:       YlOrBr (Sequential)    CB54: Pastel2 (Qualitative) -  6
  CB12:      Purples (Sequential)    CB55: Pastel2 (Qualitative) -  7
  CB13:        Blues (Sequential)    CB56: Pastel2 (Qualitative) -  8
  CB14:       Greens (Sequential)    CB57:    Set2 (Qualitative) -  3
  CB15:      Oranges (Sequential)    CB58:    Set2 (Qualitative) -  4
  CB16:         Reds (Sequential)    CB59:    Set2 (Qualitative) -  5
  CB17:        Greys (Sequential)    CB60:    Set2 (Qualitative) -  6
  CB18:           PuOr(Diverging)    CB61:    Set2 (Qualitative) -  7
  CB19:           BrBG(Diverging)    CB62:    Set2 (Qualitative) -  8
  CB20:           PRGn(Diverging)    CB63:   Dark2 (Qualitative) -  3
  CB21:           PiYG(Diverging)    CB64:   Dark2 (Qualitative) -  4
  CB22:           RdBu(Diverging)    CB65:   Dark2 (Qualitative) -  5
  CB23:           RdGy(Diverging)    CB66:   Dark2 (Qualitative) -  6
  CB24:         RdYlBu(Diverging)    CB67:   Dark2 (Qualitative) -  7
  CB25:       Spectral(Diverging)    CB68:   Dark2 (Qualitative) -  8
  CB26:         RdYlGn(Diverging)    CB69:  Paired (Qualitative) -  3
  CB27:    Set3 (Qualitative) -  3   CB70:  Paired (Qualitative) -  4
  CB28:    Set3 (Qualitative) -  4   CB71:  Paired (Qualitative) -  5
  CB29:    Set3 (Qualitative) -  5   CB72:  Paired (Qualitative) -  6
  CB30:    Set3 (Qualitative) -  6   CB73:  Paired (Qualitative) -  7
  CB31:    Set3 (Qualitative) -  7   CB74:  Paired (Qualitative) -  8
  CB32:    Set3 (Qualitative) -  8   CB75:  Paired (Qualitative) -  9
  CB33:    Set3 (Qualitative) -  9   CB76:  Paired (Qualitative) - 10
  CB34:    Set3 (Qualitative) - 10   CB77:  Paired (Qualitative) - 11
  CB35:    Set3 (Qualitative) - 11   CB78:  Paired (Qualitative) - 12
  CB36:    Set3 (Qualitative) - 12   CB79:  Accent (Qualitative) -  3
  CB37: Pastel1 (Qualitative) -  3   CB80:  Accent (Qualitative) -  4
  CB38: Pastel1 (Qualitative) -  4   CB81:  Accent (Qualitative) -  5
  CB39: Pastel1 (Qualitative) -  5   CB82:  Accent (Qualitative) -  6
  CB40: Pastel1 (Qualitative) -  6   CB83:  Accent (Qualitative) -  7
  CB41: Pastel1 (Qualitative) -  7   CB84:  Accent (Qualitative) -  8
  CB42: Pastel1 (Qualitative) -  8

Extracts spectra in raw object image files automatically.

This tool makes use of already prepared data-reduction products that are available in one directory (PATH_REDUCED) and reduces raw data available in another directory (PATH_DATA). The results are placed in a third directory (OPATH).

Right before the object is reduced, the previously reduced data are removed.

Usage

The typical use of this tool is to reduce data that are continuously placed in the input directory (PATH_DATA). The procedure and results can be viewed using the tool p3d_fastview, which is also able to launch this tool (with the basic parameters set).

p3d_autoreduce communicates with p3d_fastview using plain-text files that are placed in the user directory of p3d_autoreduce. An example of such a path is:

/home/christer/.idl/p3d/p3d_autoreduce-1-1_0/

where the last digits indicate a version-specific path. Specifically, information about the currently reduced data are placed in the file: "p3d_autoreduce_state.pref". The tool p3d_fastview looks for and reads the contents of that file.

This tool enters a loop where it waits for new object data files that are put in the selected directory. The loop exits when a file with the name "p3d_autoreduce_STOP" is found in the p3d_autoreduce user directory.

Reduction procedure

The main working procedure is that the tool enters a loop and performs the following actions:

• Every POLLTIME_REDUCED seconds the directory with reduced data is polled for the latest available product of each of the following four types: masterbias, trace mask, dispersion mask, and flat field.

  It is possible to avoid this step for each of the four product types by specifying a fixed file using the keywords MASTERBIAS, TRACEMASK, DISPERSIONMASK, and FLATFIELD.

• Every POLLTIME_DATA seconds the raw object data directory is polled for the latest [one] available object image.

• The reduced data products and the latest object data file are passed on to the tool p3d_cobjex. Using the approach below, if activated, the object data are at first reduced using fast tophat extraction and thereafter using slower optimal extraction, unless a newer object data file is available.

  Extracted spectrum data files are placed in a third directory (OPATH).

• The raw object image file is removed from the input directory when the spectra are extracted.

Requirements on the instrument-parameter file to use this tool

Reduced data-reduction products are identified using a instrument- specific header keyword that is defined using the REQDTYPE parameter in the instrument-keywords file; the default name is something like "P3DFTYPE".

Each data-reduction product must have this parameter set to one of the following four values (the keyword is properly set when the data are reduced using [a recent version of p3d that supports this functionality and REQDTYPE is set in the instrument-keywords file]:

• Master bias image: "MASTER BIAS"

• Trace mask image: "MASTER TRACE MASK"

• Dispersion mask image: "DISPERSION MASK"

• Flat field image: "MASTER FLAT FIELD"

Note: It is not possible to run this tool when this parameter is unavailable in the data-reduction products.

Fast extraction followed by slower, more precise extraction

The raw data are at first reduced as fast as possible using tophat extraction and thereafter using optimal extraction *if* three conditions are met:

• The FAST keyword is set.

• Optimal extraction reductions are setup in the instrument-parameter file.

• A user-parameter file is provided that activates optimal extraction.

General information on how to learn more about this routine

Instrument parameter files (*.prm), which are required to run this tool, are found in the directory "!p3d_path/data/instruments/*/" (outside of IDL, check "${p3d_path}/data/instruments/*/").

A default (master) user-parameter file is available in "!p3d_path/data/master_userparfile.dat". Copy this file to your data-reduction output directory, rename the file to, for example, "user_p3d.dat" and edit it to suit your needs.

Examples on how to use this routine are provided in the EXAMPLES section below.

See the file README in the root of the p3d package for installation instructions, if you have not already installed p3d on your computer.

Additional information, documentation, new versions, etc., can be found at the p3d project web site "https://p3d.sourceforge.io".

Requirements and format of the input and the output data

p3d only works with FITS-formatted images, i.e. fits files. Such files have a header that contains additional information about observing conditions, the telescope, the instrument setup, and other useful information. Both input and output fits files can be compressed using gzip, to save storage space (also on Windows platforms); p3d handles such compressed files transparently.

This routine takes two or more row-stacked spectrum (RSS) images as input; this is the default format of all extracted data of p3d.

Extracted data can, in addition to an RSS file, also be saved using the Euro3D format (use the keyword SAVEE3D or the user parameter 'savee3d'), and data of instruments using square-shaped spatial elements are easily converted to cube format using the tool p3d_rss2cube. In addition to the standard fits-file header keywords, the RSS image header contains additional p3d-specific keywords:

An additional file is written when output bad-pixels mask images are used and SATURATED is unset; this file contains a final mask, which indicates pixels that did not contain any unmasked pixels at all:

Input parameters:

	path_data		A scalar string that specifies the object data path; these data will be reduced.
	parfile		A scalar string that specifies the name of a file with instrument-specific setup parameters. These files follow a strict format. Already properly formatted files are available for all configured instruments in the directories p3d_path/data/instruments/*/. It is sufficient to specify the file basename, without the full path of the p3d subdirectory; for example: "nvimos_hr.prm".
	path_reduced		A scalar string that specifies a path of already reduced data; these are used when reducing the data in PATH_DATA. The expected products in this directory are: master bias, trace mask, dispersion mask, flat field. The tool will use the most recent product. Use the corresponding keywords below (MASTERBIAS, TRACEMASK, DISPMASK, and FLATFIELD) to make use of specific [fixed] files instead.

Keyword parameters:

	polltime_data		- A scalar decimal value that specifies the polling time of the raw object data directory (PATH_DATA). The time is not fixed, but depends on available resources. If the time since the last poll is greater than this value, the tool checks for the next available file to reduce. The default is 1.0 second. The default value is: 1.0
	polltime_reduced		- A scalar decimal value that specifies the polling time of the reduced data directory (PATH_REDUCED). The time is not fixed, but depends on available resources. If the time since the last poll is greater than this value, the tool checks the reduced data directory for the most recent reduced files to use. The default is 30.0 seconds. The default value is: 30.0
	fast		Set this keyword to reduce the data as fast as possible, using tophat extraction. Unless a new object file is available thereafter, the same file is then reduced using optimal extraction (if setup). The default value is: 1
	group		An optional integer array with as many elements as the FILENAME argument. The elements of FILENAME with the same group number are combined if this keyword is set.
	masterbias		A scalar string with the name of an existing master bias image file.
	tracemask		A scalar string with the name of an existing master trace-mask image file.
	dispmask		A scalar string with the name of an existing master-arc (dispersion-mask) image file.
	flatfield		A scalar string with the name of an extracted master flat-field image file.
	The		from p3d_cobjex – allow specific already reduced
	data
	bpmask		A scalar string with the name of a bad-pixels mask file; this could be, for example, a cosmic-ray mask file. Note! This mask is only used if optimal extraction is switched on in the user-parameter file.
	crmask		A scalar string with the name of a cosmic-ray mask image file; this could be, for example, a bad-pixels mask image file. Note! This mask is only used if optimal extraction is switched on in the user-parameter file. Note! CRMASK in not used if CRCLEAN is set.
	biaspx		Set this keyword to use the x-axis prescan region when subtracting a bias. Specifically, a median value is calculated for each pixel on the y axis in the x-axis prescan region. The same array is used with all y-axis columns of bias subtracted data.
	biaspy		Set this keyword to use the y-axis prescan region when subtracting a bias. Specifically, a median value is calculated for each pixel on the x axis in the y-axis prescan region. The same array is used with all x-axis rows of bias subtracted data.
	biasox		Set this keyword to use the x-axis overscan region when subtracting a bias. Specifically, a median value is calculated for each pixel on the y axis in the x-axis overscan region. The same array is used with all y-axis columns of bias subtracted data.
	biasoy		Set this keyword to use the y-axis overscan region when subtracting a bias. Specifically, a median value is calculated for each pixel on the x axis in the y-axis overscan region. The same array is used with all x-axis rows of bias subtracted data.
	biasconstant		This keyword can be used in two ways: Set this keyword to use a constant value as a bias value; otherwise an array is used. All prescan and overscan regions specified using the keywords BIASPX, BIASPY, BIASOX, and BIASOY can be used when this keyword is set; otherwise, only one of them. If this keyword is set to a double type array or to a comma-separated string with values greater than one (when called from the shell), these values are used to create a bias. The number of elements must be the same as the number of blocks or extensions.
	savebiassub		Set this keyword to write the bias-subtracted raw-data image before spectrum extraction. The name of the output file is the same as the (combined) input file, with the added intermediate suffix '_subbias'. Also see the user parameter 'savebiassub'.
	drizzle		Unset this keyword to use a linear interpolation instead of the default one-dimensional drizzling algorithm when resampling data from pixels to wavelengths (p3d_wavecal_dispersion_correction). The default value is: 1
	resample_startpx		A scalar value that specifies the starting pixel in the calculation of an output wavelength array when resampling pixel values to wavelength values. The keyword can be set to 'first', 'middle' [default], or 'last', to use of the first, the middle, or the last output pixel, respectively. Alternatively, the keyword can be set to an integer within the same range. Negative values, and values higher than the maximum number of pixels, short of one, are truncated. The default value is: 'middle'
	skyalign		A scalar string that contains the name of a line-list file with sky-emission lines, or an array of floating-point values with the wavelengths of sky-emission lines (the unit is Angstrom [Å]); this keyword is used when re-aligning the wavelength array in the dispersion-mask image due to effects of flexure (p3d_wavecal_dispersion_correction).
	oneskyoffset		Set this keyword to use one median offset value on all wavelength arrays in the dispersion mask. Otherwise, each wavelength array in the dispersion mask is offset individually. Additionally, when this parameter is unset, the offset at each wavelength bin is weighted with the wavelength extent of the pixel relative to the corresponding extent at the selected sky-emission line (since uncalibrated wavelength arrays generally deviate from a linear function).
	maxskyoffset		A scalar floating-point value that specifies the imum allowed offset of sky-emission lines, from wavelength provided in the dispersion-mask ge; the unit is Angstrom [Å]. See _wavecal_dispersion_correction for more details. The default value is: 2.0
	userparfile		A scalar string with the name of a file with user-defined parameters. The following parameters are read here from this file: 'keywordsfile' The name of the file with instrument-specific header keywords. 'combineimages_ob' To specify if input images are to be combined or not. 'compress' Corresponds to COMPRESS. See p3d_cobjex for a list of additionally accepted parameters. Note! If any of the corresponding keywords is specified, that value takes precedence.
	compress		Set this keyword to compress all output files using gzip; the file suffix '.gz' is appended to all output filenames. The default value is: 0
	opath		A scalar string that specifies the path where the output data are saved. The default value is: '.'
	opfx		A scalar string that specifies a prefix for the (automatically generated) output filename. The default value is: ''
	sfx		A scalar string that specifies the raw data filename suffix to use. The default value is: .fits
	detector		A scalar integer that specifies the currently selected detector; DETECTOR is a zero-based value. The default value is: 0
	logfile		This keyword specifies the name of an optional log file when p3d_autoreduce is launched as a separate program.
	loglevel		If this routine is launched as a separate program, and LOGFILE is used, this keyword specifies the log level; [1], 2, 3 (the higher the value, the more output). The default value is: 1
	recenterval		Set this keyword or RECENTERLIMVAL to activate the use of profile recentering on science images (recenterprofiles). This keyword configures several of the recentering options. If RECENTERVAL is a: string the telluric line mode is used ('recenterusetelluric'). The default file is used when the keyword is set to 'telluric' while any other string is interpreted as a filename ('recentertellines'). value, -2.0 <= value <= 2.0 the value is used as a preset offset (recenteroffset). other values are used as a median-filter width with data; the used width is the integer of the absolute value.
	recenterlimval		Set this keyword or RECENTERVAL to activate the use of profile recentering on science images ('recenterprofiles'). This keyword allows to filter out all spectra that have a lower maximum count than specified with this value [ADU], before calculating an offset using the median-filter method (when it is used). Note! This keyword is only considered if the user parameter 'recenterdpixpos' contains a single element.
	nf2f		Spectra are normalized fiber-to-fiber with a one-dimensional array if this keyword is set. The array is selected using the filename defined in the IMTRM header keyword of the flat-field image file (FLATFIELD). If the header keyword IMTRM is unavailable, and NF2F is set, data are normalized using a calculated transmission array of the flat-field image. In this case, data must not be normalized with the average spectrum (see the user parameter 'ff_normdisp'). For example, in the standard setup, if TRANSMISSION was set to the transmission-array file of a twilight flat while running p3d_cflatf, data are corrected for fiber-to-fiber transmission using that twilight-flat data here (see p3d_tool_flatfield), while the spectrum normalization is done using the flat-field image selected using FLATFIELD.
	pcutlow		A scalar floating-point value that specifies the extent of the total number of pixels on the dispersion axis removed from each spectrum at the lower (blue) end before calculating a fiber-to-fiber transmission array. This value is specified in per cent. Also see the user parameter 'ff_lowerpx'; This parameter os only used if NF2F is set. The default value is: 5.0
	pcuthigh		A scalar floating-point value that specifies the extent of the total number of pixels on the dispersion axis removed from each spectrum at the upper (red) end before calculating a fiber-to-fiber transmission array. This value is specified in per cent. Also see the user parameter 'ff_upperpx'; this parameter is only used if NF2F is set. The default value is: 5.0
	scattlightsubtract		Set this keyword to first calculate and thereafter subtract a scattered-light image. Such an image is calculated by smoothing and interpolating the raw-data image, using regions on the CCD that lie far enough from nearby spectra; considering only the cross-dispersion axis. The scattered-light image is subtracted from the masterbias-subtracted raw image before (calculating the error image and) spectra are extracted. Note! This procedure only works with data where there is some "empty" space left near the edges, and preferrably also between spectrum traces on the CCD, on the cross-dispersion axis, which can be used to measure the extent of scattered light.
	savefinalflat		Set this keyword to write the final normalized flat-field image to the disk; this is otherwise not done. Also see the user parameter 'ff_save_final_image'.
	satmask		Set this keyword to write saturated-pixels mask images to the disk. Such images indicate if any saturated pixels in the input images were used when calculating a spectrum image.
	crnthreads		A scalar integer that specifies how many threads are used when cleaning single input images of cosmic-ray hits. If this keyword is not set, NTHREADS is used instead. If nthreads was not set either, the default is to use the maximum number of available threads minus one, but the maximum number is 16: (!cpu.hw_ncpu – 1) < 16. The default value is: max – 1
	nthreads		A scalar integer that specifies how many threads to use in the parallelized line profile calculations. The calculation speed scales nearly linearly with the number of threads. The default is to use the maximum number available, but no more than 16: max = !cpu.hw_ncpu < 16. The default value is: max
	crclean		Set this keyword to clean all uncombined input images of cosmic-ray hits before the extraction procedure is started; either a master-bias image or prescan and overscan regions must be used to provide the bias level to use this functionality.
	nocrmask		Set this keyword to avoid using the cosmic-ray hit mask image, which is created when CRCLEAN is set, when extracting the spectra optimally. Normally, it is recommended to keep this keyword unset; it is available to allow the user the ultimate choice. The following keywords are used to configure the cosmic-ray cleaning procedure; they are only used if CRCLEAN is set; see p3d_cr and p3d_tool_crays_lapycosmic for more details:
	sigclip		A scalar decimal value that specifies the clipping sigma.
	objlim		A scalar decimal value that specifies the object limit value; this value is only used if IMAGEMETHOD is set.
	ratlim		A scalar decimal value that specifies the ratio limit value; this value is only used if IMAGEMETHOD is unset.
	crfwhm		A decimal scalar value, or a two-element array, that specifies the full width at half maximum of the gaussian kernel used to convolve the raw data. This keyword corresponds to FWHM in p3d_cr. This value is only used if IMAGEMETHOD is unset.
	gausskernelsize		A scalar integer that specifies the size of the Gaussian kernel used when convolving the input image with a Gaussian kernel using the PyCosmic algorithm. This value is by default set to 2 * q + 1, where q is the maximum value of the x and the y pixel where a one-dimensional Gaussian reaches 15% of its maximum value (1.0); this condition makes use of the full-width-at-half-maximum value of FWHM. The value must be odd, and the maximum value is in any case 19. Be aware that convolution times increase strongly with the size of the Gaussian kernel.
	sigfrac		A scalar decimal value that specifies a scale factor for pixels surrounding a cosmic-ray masked pixel; this parameter is only used if GROWRADIUS is non-zero. The default value is: 0.5
	growradius		Set this keyword to grow all cosmic-ray hit pixels by a frame that is this many pixels wide; this is done to check if the surrounding pixels are affected by cosmic-ray hits as well. The default value is 1, the values 0 and 2 are also accepted. Surrounding pixels are checked if they are cosmic-ray affected using the first criterion S' > SIGFRAC * SIGCLIP. The default value is: 1
	maxiter		A scalar integer value that specifies the maximum number of iterations used when identifying cosmic-ray hits in each image.
	imagemethod		Set this keyword to use the original algorithm of P. van Dokkum (L.A. Cosmic), which was developed for cosmic-ray detection in images. The default is otherwise to use the IFS-modified algorithm with IMAGEMETHOD unset.
	imageclean		Set this keyword to use the original [5 * 5 px²] two-dimensional median filter when replacing the flux in masked pixels with that of the surrounding pixels. An [11 px] one-dimensional median filter is used otherwise. The default value is: 1
	dispmedian		Set this keyword to a positive integer > 1 to subtract sampling flux along the dispersion axis. The value defines the width of the median window.
	writeall		Set this keyword to write all intermediate images in addition to the cleaned image and the cosmic-ray mask. is executed.
	nocrc		Set this keyword to force the use of the slower IDL-based cosmic-ray cleaning routine instead of the faster C-based routine.
	noobcheck		Set this keyword to avoid checking the OB header keyword; this only applies to the instruments that use it. To see if it applies to your instrument, check if the OBID parameter is defined in the instrument keywords file.
	fnotify		Set this keyword to the name of a file that is written when the tool has finished; this can be a useful feature when using the p3d queue handler. In that case, set this keyword and check if the file exists before continuing. And do not forget to remove the file.
	cmdline	X	This keyword is used then the tool is launched by the p3d server.
	logunit	X	Messages are saved to the file pointed to by this logical file unit, if it is defined.
	verbose		Set this parameter to a positive scalar integer to make p3d write some information on STDOUT about what is going on. The following four values are acknowledged: 0 Writes no information at all. This is the default. 1 Writes the more important information; regarding subroutine configurations, mostly. 2 Writes most information; includes a more verbose output than 1. 3 Writes all information, including information on the execution state of GUI subroutines. This may be a useful mode when debugging the code.
	quiet		Set this keyword to not print the disclaimer on the screen when this tool is started from the command line or from the shell.
	error	X	This scalar integer returns an error code if set. Any value different from zero means that an error has occurred.
	debug	X	An error handler is not setup if this keyword is set. Otherwise, the error handler is setup using the routine CATCH, which makes each subroutine exit quietly when a bug is encountered. Use this keyword when debugging p3d.
	help	X	Set this keyword to show this routine documentation, and then exit.

Examples:

This routine can be called in three different ways (for use on WINDOWS platforms, replace the path character '/' with the backslash '\'):

i) As a stand-alone tool from the IDL command line

In addition to the input raw data path and the [provided] instrument- specific parameter file, it is necessary to specify the reduced data path as well as any additional arguments:

path_data = 'path_raw_input'
parfile = 'plasmark.prm'
path_reduced = 'path_reduced_input'
opath = 'path_reduced_output'
userparfile = opath + path_sep() + 'user_p3d.dat' ;; If used
p3d_autoreduce, path_data, parfile, path_reduced, $
    userparfile=userparfile, opath=opath, $
    logfile=opath + 'dred.log', /verbose

ii) As a stand-alone tool from a terminal, using the IDL VM

In addition to the input raw data path and the [provided] instrument- specific parameter file, it is necessary to specify the reduced data path as well as any additional arguments. The recommended approach is to use the provided shell script p3d_autoreduce_vm.sh or the Python tool p3d_dispatch.py:

# Use a shell script, that makes use of the p3d queue system:
path_data=path_raw_input
parfile=plasmark.prm
path_reduced=path_reduced_input
opath=path_reduced_output
userparfile=${opath}/user_p3d.dat # Set if used
p3d_autoreduce_vm.sh ${path_data} $parfile ${path_reduced} \
    userparfile=$userparfile opath=$opath \
    logfile=$opath/dred.log /verbose

# Use the Python tool, that does not use the p3d queue system:
path_data=path_raw_input
parfile=plasmark.prm
path_reduced=path_reduced_input
opath=path_reduced_output
userparfile=${opath}/user_p3d.dat # Set if used
p3d_dispatch.py --noqueue autoreduce ${path_data} $parfile \
    ${path_reduced} userparfile=$userparfile opath=$opath \
    logfile=$opath/dred.log /verbose

Note: In this approach, it is necessary to specify all options using the full names of the keywords. Partially correct keyword names are not permitted, as they are inside IDL. Hence, as an example, you must type "p3d_autoreduce_vm.sh userparfile=$userparfile" and -NOT- "p3d_autoreduce_vm.sh user=$userparfile".

Note: The tool launches in the current directory, which is why it is unnecessary to specify full paths if files reside in the current directory.

Note: Parameters indicated with an 'X' in the list above are unavailable in this approach.

iii) As a stand-alone tool from Python, using the IDL VM

This approach is analogous to ii), but is executed from the Python command line. Here is an example:

# Launch p3d_autoreduce in a single IDL session, without using the
# p3d queue system:
from p3d_dispatch import *
path_data = "path_raw_input"
parfile = "plasmark.prm"
path_reduced = "path_reduced_input"
opath = "path_reduced_output"
userparfile = os.path.join(opath, "user_p3d.dat") # Set if used
commandline = path_data + ' ' + parfile + ' ' + path_reduced +
  ' userparfile=' + userparfile + ' opath=' + opath +
  ' logfile=' + os.path.join(opath, "dred.log") + ' /verbose'
p3d_dispatch('autoreduce', commandline, noqueue=True)

This routine creates a dispersion-mask image from a set of arc images that is then used to wavelength calibrate spectra that are extracted using p3d.

The selected arc images are at first combined, if required, to create a combined (master) arc image. The instrument-specific parameters and the information in the data header are used to determine a default arc-line list file, and calculate a first-guess wavelength range. The spectra of the (combined) arc image are extracted. Thereafter, the interactive graphical user interface (GUI; see below, and the routine p3d_wavecal_dispmask_gui) is opened. The interactive procedure consists of matching entries in the arc-line list with features in the extracted arc spectra. More accurate pixel positions are calculated for each entry in the line list and spectrum, before a polynomial of pre-defined order is fitted to each spectrum separately. The fitting parameters are saved for each spectrum separately; this is the dispersion-mask image.

Note: Once a dispersion-mask image is created, it is possible to use that image as a starting point when creating an image for another set of data that use the same instrument setup. To use this functionality, specify the already created dispersion-mask image file with the DISPMASKIN keyword.

Please read through the documentation below, and also see the p3d A&A paper as well as the master user-parameter file and the p3d documentation WIKI, for help on how to setup the parameters to achieve the highest accuracy possible in the wavelength calibration.

This routine calls the routine p3d_cr, if requested to do so, which cleans single input images of cosmic-ray hits (see the keyword CRCLEAN). The outcome of the cleaning procedure depends both on properties of the input image as well as the algorithm parameters. Different parameters might produce a different outcome. There is no guarantee whatsoever that the cleaning removes only cosmic-ray hits. Check the output images to build your own opinion regarding the quality of the outcome.

The output-data files – consisting of the extracted spectrum image, its error image, the bad-pixels mask image, the saturated-pixels mask image, and the data-quality mask image – are by default written to the same file, in consecutive extensions. They are instead written to separate files if the keyword ALLINONE is unset, alternatively, unset the user parameter 'allinone'. There is no real advantage to either approach; although, if separate files are used and an output file is renamed, it will not be found by other tools.

Here follows a description in full detail of the creation of the line list, the calculation of the wavelength range, and all parts of the GUI.

How p3d determines which arc-line list file to use

Before the dispersion mask GUI is opened, p3d has to determine which arc-line list file to use. The arc-line file is a plain-text file where the each line contains the known wavelength of an arc (emission) line, as well as an optional [relative] line intensity. The following methods are used, ordered after order of priority:

Format of the line-list file

The line-list file is a plain-text file, which could be saved using the ASCII or ISO-8859-1 encodings. Lines that begin with a semicolon (;) are comments with the exception of a line in the header that begins with the string "; wavelength unit: "; such a line specifies the wavelength unit. The following units are available when the mentioned string is followed with the following strings:

• "µm", "micrometer", "micron" :: Micrometer (1e-4 cm).

• "nm", "nanometer" :: Nanometer (1e-7 cm).

• "Angstrom", "Å" :: Ångström (1e-8 cm). This is the deafult.

Are remaining lines specify the wavelength and an optional relative intensity for each arc line; the wavelength and the intensity are separated with whitespace.

Remove entries in the line-list file

The default arc-line list file may contain more lines than can be identified in the image, or it may be the case that some of the lines are saturated. If this applies to the used data, the corresponding entries should be removed from the arc-line list before creating the dispersion mask.

Remove arc lines that are too faint

Arc lines that are too faint in the used image can be removed at the time that the line-list file is read by setting the keyword LINELISTILIMIT or the user-parameter 'linelistilimit_dm' to a lower limit value. Naturally, this feature demands that the used arc-line list file contains a second column with [relative] intensities (see above).

Remove arc lines interactively

Superfluous entries in the line list can be removed interactively when this tool is being used, and here is how this is done (this is also described below, in the GUI section on the 'Delete lines' function).

Click the 'Delete lines' button in the 'Line Mask Functions' tab of the main control panel. The (pixel) position of each entry in the line list is shown as a cross. When the mouse cursor is moved over the spectrum plot, the wavelength (in Angstrom) is shown of the entry in the arc-line list that lies the closest to the cursor. Click the left mouse button to delete the entry from the line list. Click the right button to finish the deletion procedure.

When done, the resulting arc-line list can be saved to a file by selecting the menu entry File->Save the current line list. The same arc-line list file can then be loaded into any subsequent wavelength-calibration session; either using the ARCLINEFILE keyword, the user-parameter file, or the menu option File->Read line list(s).

An additional consideration

In some cases, the difference between the fitted wavelength and the tabulated wavelength of any one line may always be larger than for the other lines. It is possible that the accuracy of the wavelength of that particular entry is low.

If this should occur, remove the entry from the arc-line list, or correct the value, and then try anew. If you find it necessary to edit any line list in the p3d line list directory, please notify us, so that we can apply the proper correction to the data file in question.

How p3d initially determines the wavelength range

Before the dispersion mask GUI is launched, p3d determines an initial wavelength range covered in the arc image. This initial wavelength range is used when determining the first-guess pixel position of each entry in the arc-line list. The determination of the wavelength range depends on the instrument; here follows a description of how the wavelength range is set with some of the instruments of p3d:

The graphical user interface of the dispersion-mask creation tool

After the spectra in the (combined) arc image have been extracted, the dispersion-mask creation GUI tool is launched. This tool consists of six separate regions: a menu bar, the spectrum image, a wavelength-range indicator field, a plot of the reference spectrum, a status-line field, and a control panel. Here is a more complete description of each of these regions. Key accelerators are, where available, right justified.

The menu bar

Here follows a description of all the available (and sometimes unavailable) entries in the menu bar.

The 'File' sub menu field

• View The Fits Header

  Open a window that displays the fits-file header of the used arc image.

• View The Fits Header – Second File

  Open a window that displays the fits-file header of the used second arc image. This entry is only visible when two arc image files are used.

• Read Line-List File(s)

  Open a file dialog that allows any number of arc-line list files to be selected; use the shift and ctrl key modifiers to select multiple files.

• Save The Current Line List

  Open a file dialog where you can choose the name of a single file that will contain the currently used arc-line list. The output is a plain-text file where each line contains the wavelength of an arc line. The wavelength values [Å] increase monotonically.

• View The Current Line List

  Open a window that contains a numbered list of the currently used arc lines. The format of the list depends on if the data of the used instrument are regular IFU data or if each spectrum shows a separate order of a complete spectrum. These are the two setups:

  IFU data:

  The list contains three columns: a line number, the arc line wavelength [Angstrom], and the pixel position of each line.

  Spectrum-order (non-IFU) data:

  The list contains four columns: a line number, the spectrum order, the arc line wavelength [Angstrom], and the pixel position of each line. Entries of each spectrum order are treated separately, and they are also shown separated with horizontal dashed lines.

  Lines, which would be placed left of as well as right of the shown window are shown with the pixel position -1.000. Some, or sometimes all, of either group of lines can be shifted into the pixel range using the 'shift mask' function.

  The shown line list is dynamically updated as emission lines are deleted using the 'Delete lines' function, and also when arc line wavelengths are matched and fitted with the data.

  Select the menu option File->Exit in the line-list GUI to quit the same GUI.

• Undo All Changes (Ctrl+U)

  All parts of the GUI return to their state as they were when the GUI was first launched.

• Exit (Ctrl+Q)

  Quit the dispersion-mask creation GUI, without saving any dispersion mask that might have been created.

The 'Viewing Options' sub menu field

• Invert the Fg/Bg Colors

  Swap the foreground and background colors.

• Color map; use 100% of the pixel values (F1)

• Color map; use 99.5% of the pixel values (F2)

• Color map; use 99% of the pixel values (F3)

• Color map; use 98% of the pixel values (F4)

• Color map; use 97% of the pixel values (F5)

• Color map; use 95% of the pixel values (F6)

• Color map; use 90% of the pixel values (F7)

• Color map; use 80% of the pixel values (F8)

• Color map; use 70% of the pixel values (F9)

  The function of these options are described in the spectrum-image section below.

• Invert the Colors

  Toggles the minimum and the maximum pixel values with the effect that the color table is "inverted"; for example, instead of black->white, the colors would become white->black.

• Select The Color Table

  Allows the selection of another color table than the preset one, either using the (standard) interactive routine XLOADCT, or by choosing any of the pre-configured menu entries.

The 'Function' sub menu field

• Match Curvature (Ctrl+1)

• Shift mask (Ctrl+2)

• Delete lines (Ctrl+3)

• Match Centroids (Ctrl+4)

• Fit Centroids (Ctrl+5)

• Create Dispersion Mask (Ctrl+6)

• Save the Mask and Exit (Ctrl+W)

  Each of these functions / actions are described in detail below.

The 'Options' sub menu field

• Subtract the Background

• Set the Reference Position

• Change Dispersion

• Reset Disp

  Each of these functions / actions are described in detail below.

The spectrum image

The top-most field shows an image with all extracted spectra stacked on top of each other. Each (horizontal) line represents a spectrum, where each spectrum is one pixel wide. The first spectrum in the file is shown at the bottom, and the last spectrum at the top. Shorter (longer) wavelengths are always shown on the left-hand (right-hand) side.

Depending on the user parameter 'dbin', the image is not rescaled (dbin = 1, this is the default) or rescaled (dbin = 2, 3) on the horizontal axis:

The (false) colors used to show the spectrum image are calculated by a linear scaling of a defined intensity range. The intensity range values can be set explicitly (see the view tab options in the control panel below) or by using a histogram function with a set of nine preset x-values: 100%, 99.5%, 99%, 98%, 97%, 95%, 90%, 80%, and 70%. The meaning of the values is as follows. In a histogram of all pixel values in the spectrum image, the lowest (100 – x) / 2 per cent, as well as the highest (100 – x) / 2 per cent, of all pixel values are discarded before setting the minimum and the maximum intensity values. The overall minimum and maximum values are used with the 100% setting. The color table is chosen using the COLORTABLE keyword; the default is the so-called Sauron color table.

The initial pixel position for each entry in the line-list file is shown as a vertical white line, which is plotted on top of the spectrum image. These white lines are initially always straight, in comparison to arc lines in the spectrum image, which are mostly curved, to some degree; the curvature reflects differences in the optical path across the surface of the integral-field unit.

A reference spectrum is shown in the reference spectrum plot. The reference spectrum and a reference pixel are defined as a spectrum located near to the center, and the center pixel. The reference pixel is shown with a red vertical bar, and the reference spectrum is shown with a red bullet. The initial reference spectrum and the reference pixel can both be changed using the options under the "Reference spectrum and Reference pixel" tab. The reference spectrum should be chosen as a spectrum where, ideally, only arc lines are visible. Blue- or redshifted emission lines, originating in the object, can otherwise decrease the accuracy of the dispersion-mask calculations if they lie close to the arc lines.

Note: The spectrum image is a very small region with instruments that only use a small number of spectra or spectrum orders.

The wavelength range indicator field

When the dispersion-mask tool is launched, the first-guess wavelength range of the initial "reference spectrum" is indicated with the lowest wavelength on the left-hand side and with the highest wavelength on the right-hand side of the narrow wavelength-range indicator field. The center wavelength is shown in the middle.

When the spectrum-image field is shown with a horizontal scroll bar, an arrow is shown in front of (behind) the lowest (highest) wavelength value. The pixel position of the center value shifts along with the horizontal scroll bar; if the pixel position falls outside of the shown range, the center wavelength is shown with a green color, and a green arrow is also shown to indicate that the center pixel is located to the left (right) of the lower (upper) boundary.

A plot of the reference spectrum

A plot of the reference spectrum is shown in the third field from the top; this is, mostly, the second largest field [but when there are only a small number of spectra, it is the largest field]. The range of pixel values of this plot always agrees with the range of the spectrum image and the wavelength-range indicator field. The same range of intensity values used to calculate the colors of the spectrum image is used to scale this plot.

Each entry in the line-list file is shown as a white x symbol; the positions of the x:s agree with the positions of the straight lines in the spectrum image. The wavelength of each emission line is shown above the x when the left mouse button is clicked on an x.

The currently selected line-list entry and its wavelength are shown with a red color when the 'Delete lines' function of the 'Line Mask Function' tab is active.

The status line

When the mouse cursor is positioned above an interaction field (i.e. a widget), such as, for example, a drawing area or a button, a descriptive text of the function of the field is shown on the status line. Some functions also provide dynamic instructions in this field.

The control panel

All functions and options to control the dispersion-mask creation procedure are collected under tabs of similar actions of the control panel. The six tabs are: Line Mask Functions, Dispersion, Check The Current Fit, Check The Final Fit, Viewing Options, and Reference Spectrum And Pixel. Here follows a description of each tab and its components:

Line mask functions

This tab collects all the basic functions necessary to create a dispersion mask. The functions should be executed one by one, preferrably starting from the left-hand side. Not all functions are available when the tool starts up (the corresponding buttons are not sensitive), those ones become available as the previous functions provide the required information.

The functions are ideally executed once, moving from the left to the right on the row of buttons, with IFU instrument data where each spectrum is assigned a unique spatial position. With instruments that instead use multiple orders of the same spectrum, it is necessary to fit the arc lines to each spectrum order separately, before a dispersion mask can be created.

Here follows a description of each of the seven functions:

• Match curvature (Ctrl+1)

  There is, when the tool is first started, no information available about differences in the optical path of separate spectra. The initial pixel position of each emission line is thus independent of the spectrum. Extracted arc spectrum images, however, show that the optical path varies, with the result that the arc lines appear curved – as a function of spectrum number – instead of being straight lines. The 'Match Curvature' function uses the arc image to calculate a "curvature" for all entries in the arc-line list as function of wavelength. The curvature is calculated by following the maximum position of an initially selected arc line through subsequent spectra along the dispersion axis.

  The function is activated with a left mouse button click on the 'Match Curvature' button. There are thereafter two options to proceed:

  Use one arc line

  Click with the left mouse button on top of a fairly bright, but not saturated emission line. A curvature is then calculated for all spectra and entries in the arc-line list, assuming a constant dispersion with wavelength.

  Use two arc lines

  Click with the right mouse button on top of at first one, and thereafter a second, fairly bright, emission line. A curvature is then calculated for all spectra and entries in the line list, assuming that the dispersion changes linearly with the wavelength. In order to increase the accuracy of the curvature calculation, it is both important to select sufficiently bright arc lines that are also not saturated, and to choose arc lines that are very well separated to properly estimate the change in dispersion. Try to choose one arc line near the blue edge and one arc line near the red edge of the image.

  Attention: This option is unavailable when using an input dispersion-mask image using the DISPMASKIN keyword.

  Attention: This option is unavailable with instruments that use separate spectrum orders of one spectrum.

• Shift mask (Ctrl+2)

  The first-guess wavelength range is likely shifted on the dispersion axis relative to the positions of the arc lines in the input image – the shift is greater for some instruments than for others. The line mask, consisting of all entries in the arc-line list, can be shifted linearly on the dispersion axis using the 'Shift mask' function.

  The function is activated by clicking with the left mouse button on the 'Shift mask' button. The mouse cursor must thereafter be positioned in the spectrum-image field above the field that shows the wavelength range.

  Click the left mouse button and drag the line mask left or right until (while keeping the mouse pointer in the same field) the crosses match the emission lines satisfactorily. The function finishes once the left mouse button is released.

  It may be necessary to change the first-guess value of the dispersion (given in Å/pixel), this can be done with the options in the Dispersion tab. Use the 'Change Disp.' function to change the constant dispersion value. Few instruments, however, show a constant, or even linear, dispersion across a larger wavelength range. In such cases, the best approach is to first shift the line mask to allow an identification of some known lines, and thereafter use the 'Manual Disp.' function on the 'Dispersion' tab.

  Note: The 'Shift mask' function assumes a constant, or linear, dependence of the dispersion with wavelength, which is why it MUST NOT be used after the 'Manual Disp.' function is used.

  Attention: This option is unavailable when using an input dispersion-mask image using the DISPMASKIN keyword.

• Delete lines (Ctrl+3)

  In particular, the default line lists contain entries of (faint) arc lines that are unlikely to be found in the extracted arc image. Such entries can be removed from the line list using the 'Delete lines' function. In case a tailored line-list file is used, which only contain entries that are present in the arc image, there is, of course, no need to use this function.

  The function is activated with a click on the left mouse button on the 'Delete lines' button. Thereafter, the mouse cursor should be positioned in the spectrum-plot image. The arc-line entry that lies the closest to the mouse cursor is indicated in red color; its wavelength is also shown in the same red color [Angstrom]. Remove the entry marked in red with a click on the left mouse button (note that the entry also is removed immediately in the line-list view, if it is shown). The function is active, and the same procedure is repeated, until the right mouse button is clicked.

• Match centroids (Ctrl+4)

  At this point, the entries of the arc-line file should have been shifted, and the dispersion modified, if necessary. The x symbol of all available arc lines should lie close to a peak of the respective emission line in the arc image.

  The (non-interactive) function 'Match centroids' traverses all entries of the arc-line list and checks if there is an intensity peak in the arc image near the corresponding pixel position of each entry. A positive identification requires two conditions to be fulfilled:

  1.	The pixel distance to the arc-image peak must be less than, or equal to, 'linewidth_dm' pixels (see the instrument-specific parameter file).
  2.	The maximum of the peak must lie above three times the noise level, which is estimated from data in the region near the peak.

  New arc-line positions, which are weighted with the peaks in the arc-image data, replace the input positions. The procedure is iterated five times to further increase the accuracy of the new positions. All entries where no peak is found are removed from the arc-line list.

  The function is used with a click with the left mouse button on the 'Match centroids' button. After the function has finished its execution, an //rms// value, which relates the //n// new positions (//pos_new//) to the //n// input positions (pos_input), is shown in the spectrum-plot field. This //rms// value is calculated using the equation:

  rms = sqrt(sum((pos_new – pos_input)²) / n).

  Note: This function can be used repeatedly until the outcome looks satisfactory; for example, if you decide that you want to remove another entry in the arc-line list after using either this function or the 'Fit centroids' function, delete the line and thereafter click the 'Match centroids' button anew.

  Note: For more details, see the routine 'p3d_wavecal_match_maskwd'.

  Attention: This option is unavailable when using an input dispersion-mask image using the DISPMASKIN keyword.

• Fit centroids (Ctrl+5)

  In order to create an accurate dispersion mask, it is necessary that all pixel positions of the used arc-line entries are accurate. The (non_interactive) 'Fit centroids' function calculates such accurate positions for each entry in the arc-line list. Two methods are available, using the parameter centermethod_dm (see the instrument-specific parameter file):

  1.	The first approach is to first fit a "Gaussian" profile to the peak; the initial arc-line position is thereafter replaced with the center position of the fitted profile.
  2.	The second approach is to replace the initial arc-line position with a "weighted" position; in this case, the arc-image spectrum is used as a weight. This approach is mostly less accurate than the Gaussian approach, but it has the advantage that it is much faster.

  The function is used with a click with the left mouse button on the 'Fit centroids' button. The procedure can be repeated as many times as required, although the increase in accuracy is minute in subsequent calls. After the function has finished, an //rms// value, which relates the //n// new positions (//pos_new//) to the //n// input positions (//pos_input//), is shown in the spectrum-plot field. This //rms// value is calculated using the equation:

  rms = sqrt(sum((pos_new – pos_input)²) / n).

  Note: This function can only be used AFTER the 'Match centroid' function has been used.

  Note: For more details, see the routine 'p3d_wavecal_correct_maskpos'.

• Select Next Spectrum Order (Ctrl+N)

  With instruments that use data with stacked spectrum orders of different wavelength ranges of the same one spectrum, this button (shown with a next-track symbol) allows a quick change to the next spectrum order where arc line entries have not yet been fitted to the raw data.

  Attention: This is a shortcut to the 'Reference Spectrum' function in the tab 'Reference Spectrum and Pixel'.

  Attention: This function is unavailable with instruments that use IFU data where spectra represent different spatial coordinates.

• Create dispersion mask (Ctrl+6)

  The dispersion-mask image is created as follows. A polynomial of order //n_p// is fitted to each spectrum individually, using the known wavelengths of the arc-line list entries as function of their respective pixel position. The default polynomial order (//n_p//), which is shown immediately to the right of the control panel (where it can also be changed), is provided by the parameter 'polynomialorder_dm' (see the instrument-specific parameter file).

  This (non-interactive) function is used with a click with the left mouse button on the 'Create dispersion mask' button. After the dispersion mask has been created, the extracted image in the spectrum-image field is replaced with a wavelength corrected image. If the fit was successful, the initially curved arc lines (of instruments that use IFU data) should now be straight. One way to check the quality of the dispersion mask is to study the plots now available under the 'Check The Final Fit' tab.

  With instruments that use an IFU setup, the fitting procedure is performed twice; with non-IFU data it is only done once. In the first step, it is done for all spectra in the input image. A zero-weight is assigned to all arc-line entries, for each spectrum individually, where the input-to-fitted wavelength residual is larger than, or equal to, the pre-defined value 'residualcut_dm' (see the instrument-specific parameter file). Only arc-line entries that are positively identified in at least half of the spectra in the first fitting loop are used when fitting the polynomial in the second fitting loop.

  Note: This function can only be used AFTER the 'Fit centroids' function is used. None of the previous five (IFU data) or six (non-IFU data) functions can be used after this function has been called.

  Note: For more details and additional options, see the routine 'p3d_wavecal_fit_maskwd'.

• Save The Mask And Exit (Ctrl+7)

  In order to save the disperson-mask image and thereafter close the graphical user interface, click the 'Save The Mask And Exit' button.

  Note: This function can only be used AFTER the 'Create Dispersion Mask' function has been used.

Dispersion

This tab collects all functions that are used to adjust the dispersion, i.e. the measure Å [Angstrom] per pixel. The functions can be used in any order, as required, but if the 'Manual Disp.' function is used, the use of any of the other functions afterwards will discard the just created non-linear solution.

• Change Disp.

  The constant dispersion value can be changed interactively using the 'Change Disp.' function. To use this function, click the corresponding button and thereafter position the mouse cursor in the spectrum-image field, but not atop the (red) reference pixel.

  The wavelength of the associated pixel is changed by first clicking with the left mouse button. Thereafter, drag the mouse cursor away from the initial position. The mouse cursor must always be kept on the same side of the reference pixel as when the dragging was initiated. The function is finished when the left mouse button is released. The new value of the dispersion is determined as follows: let the initial separation between the mouse cursor and the reference pixel be //x_init//, and let the 'dragged' separation be //x_dragged//, then the new dispersion will be set to //x_dragged / x_init//.

  Note: When changing the dispersion using this function, a higher precision is achieved if the left mouse button is first clicked when the mouse cursor is not too close to the reference pixel. Thereafter, only move the mouse cursor with small movements to avoid too drastic changes of the dispersion value. If necessary, use the 'Reset Disp.' function to restore the initial dispersion value.

• Reset Disp.

  Click the 'Reset Disp.' button to reset all the dispersion-related values to the routine entry values. These three numbers include: the constant dispersion value, the relative change in dispersion between pixels (delta; this value allows for a linear function of the dispersion), and the wavelength of the center pixel.

• Manual Disp.

  Few instruments show a constant, or even linear, dispersion across the full wavelength range. With the manual dispersion function, it is possible to calculate a custom non-linear dispersion solution using all available arc lines.

  To use this interactive function, first click with the left mouse button on the 'Manual Disp.' button. Thereafter, position the mouse cursor in the spectrum-plot field. The next task is to create a list of new (modified) arc-line pixel positions. It is necessary to repeatedly use the following procedure when creating this new list:

  1.	Click a first time, with the left mouse button, near the chosen entry in the arc-line list; the symbol of the entry that will be chosen is indicated with a red color.
  2.	Click a second time, again with the left mouse button, at the (exact) position where the arc-line entry should be moved. The old and the new positions are then shown with a red arrow that indicates whether the change is towards a shorter or a longer wavelength.
  3.	Should the last modified position be useless, it can be removed from the new list by clicking the middle mouse button. All entries in the new position list can be removed by using this function repeatedly.

  Once finished adding pixel positions to the new list, for all the arc-line list entries that are to be used, the procedure is finalized by clicking the right mouse button. Subsequently, a rational spline is fitted to the new pixel positions; as wavelength[pixel]. This dispersion solution should – if the new pixel-position list was created carefully – be more accurate than the initial constant-dispersion solution. To make possible the creation of an accurate dispersion solution at this step, follow these guidelines, which will also subsequently allow the creation of an accurate dispersion mask:

  1.	Make sure all arc lines are identified correctly. If you do not know which arc lines are present in your spectrum, study a printed arc spectrum of your wavelength range, which uses the same arc lamps.
  2.	Cover the full wavelength range with arc lines. The spline function is unable to create an accurate dispersion solution for the lowest or highest pixel regions if no arc lines are used in these regions. If the range is large, as is the case if there are, for example, no entries at all across half of the CCD, the spline routine will likely fail completely.
  3.	It is not necessary to specify exact pixel positions in the new list. More exact positions are calculated by the 'Fit centroids' function.
  4.	Avoid abrupt changes in the dispersion across few pixels. If you cannot achieve an accurate dispersion mask, first check your arc-line spectrum again to see if you might have associated the wrong arc line with an entry in the arc-line list.

  Note: The dispersion solution created by the 'Manual Disp.' function is not accurate enough to calculate a dispersion mask. The only purpose of this preparatory function is to produce a dispersion solution that allows p3d to match as many arc lines as possible in the input spectra with the entries in the arc-line list.

• Disp.

  The editable field immediately to the right of this label shows the value of the constant dispersion; in [Å/pixel]. Enter a new value in the field number to replace the currently used value.

  Note: This value is not used after the 'Manual Disp.' function is used.

• Delta

  The editable field immediately to the right of this label shows the value of the differential, and linear, change of the dispersion; in [Å/pixel]. Enter a new number in the field to replace the currently used value. The dispersion is constant when this value is equal to zero, otherwise it is linear.

  Note: This value is not used after the 'Manual Disp.' function is used.

• Reference Pixel Wavelength

  The editable field immediately to the right of this label shows the value of the wavelength at the reference pixel; in [Å]. Enter a new value in the field to replace the currently used value.

  Note: This value is not used after the 'Manual Disp.' function is used.

Check The Current Fit

This tab provides a mean to check both the quality of a (linear) polynomial fit to the current arc-line pixel positions (click on the 'Fit' button) and the residual of each arc-line entry of the preset-wavelength minus the fitted-wavelength. The check is only done for the currently shown reference spectrum.

The spectrum image (in the upper panel) – or with instrument setups that use few spectra the spectrum plot (in the lower panel) – is replaced with a plot of the wavelength as function of pixel – using the full wavelength range. The fitted curve is drawn with a solid line between the individual arc-line entries, which are shown with the x symbol.

In the polynomial-fit mode, four different informational strings are shown in the upper left corner of the plot. The four fields are separated by two colons and show: the polynomial order, the output status code of the IDL routine POLY_FIT (0 == successful completion), the chi-square value of the fit, and the root-mean-square (//rms//) value of the fit. Only the //rms// value is shown in the residual-mode plot.

Please note that the status check of this tab does not show quantitatively what the quality of the dispersion-mask image will be. However, if it is executed after the 'Fit Centroids' function, it provides a quick mean to check if the polynomial order is appropriate, and if the used arc-line entries provide a sufficient accuracy. Perhaps, some entries could be removed to improve the accuracy?

Note: This results checking mode is only available before a dispersion-mask image has been created using the 'Create Dispersion Mask' function.

Check The Final Fit

This tab provides a mean to check the quality of the (linear) polynomial fit to the created dispersion mask. In comparison to the current fit tab, all spectra can be checked here.

The spectrum image (in the upper panel) – or with instrument setups that use few spectra the spectrum plot (in the upper half of the lower panel) – is replaced with a plot of the wavelength as function of pixel – using the full wavelength range. The fitted curve is drawn with a solid line between the individual arc-line entries, which are shown with the x symbol. Some information about the fit is shown in the upper left corner of the panel.

The information contains the spectrum number (IFU data) or spectrum order (non-IFU data). The spectrum numbers are 1-based, i.e., 1, 2, 3, ...; inside of a pair of brackets; note that any calibration spectra are removed from the image prior to launching the dispersion-mask creation tool, which is why there might be an offset compared to the actual spectrum number in the image), the polynomial-fit order, and the chi-square number of the fit. If the shown spectrum is listed in the instrument-specific dead-fibers file, the text in the second column of that file is shown to indicate if the spectrum: originates in a low-transmission fiber, in a dead fiber, or is not used (unused; which would be the case with spectra falling outside the detector). Spectra of these latter types are often more difficult, and even impossible, to calibrate compared to the normal spectra – there is no cause for alarm if they cannot be well calibrated.

The plot of the reference spectrum (in the lower panel) – or with instrument setups that use few spectra the spectrum plot (in the lower half of the lower panel) – is replaced with a plot of the residuals for each arc-line entry (of the preset-wavelength minus the fitted-wavelength) as function of pixel – also here using the full wavelength range. A horizontal dotted line indicates the zero level. Some information about the fit is shown in the upper left corner of the panel as well. The information contains the spectrum number (IFU data) or spectrum order (non-IFU data) inside a pair of brackets; see the above description of the upper panel, and the maximum residual value, which is determined from all arc-line entries of the currently shown spectrum.

The currently shown spectrum can be selected using the scroll bar, which is shown in the control panel below the panels, or by typing a (valid) spectrum number (IFU data) or spectrum order (non-IFU data) in the field to the left of the scroll bar. The arrow button at the left (right) end of the scroll bar can be used to decrement (increment) the spectrum number / order by one.

The upwards pointing arrow button on the right-hand side of the scroll bar selects the spectrum with the maximum residual; dead or unused spectra, as specified in the instrument-specific dead-fibers file, are ignored when using this function.

Note: This results checking mode is only available after a dispersion-mask image has been created with the 'Create Dispersion Mask' function.

Viewing Options

This tab provides a set of options to control the presentation of the spectrum image, the spectrum plot, and the fitting procedure. From the left-hand side to the right-hand side, the different options are:

If a continuum is present in the arc spectra, it can be subtracted with a click on the 'Subtract The Background' button; this button can only be used once.

The current minimum (maximum) value used to calculate the colors (using any pre-defined color table), and set the ranges of the spectrum plot, is shown in the left (right) editable field. The (false) colors used to show the spectrum image are calculated by a linear scaling of a defined intensity range. The intensity range values can be set explicitly by typing any (valid) values into the fields, or by using the droplist on the right-hand side of the editable fields. With the help of the droplist, a histogram function is used, with a set of nine preset x-values: 100%, 99.5%, 99%, 98%, 97%, 95%, 90%, 80%, and 70%. The meaning of the values is as follows. In a histogram of all pixel values in the spectrum image, the lowest (100 – x) / 2 per cent, as well as the highest (100 – x) / 2 per cent, of all pixel values are discarded before setting the minimum and the maximum intensity values. The overall minimum and maximum values are used with the 100% setting. The droplist values can also be set using the key accelerators F1 – F9 (see the Viewing Options menu item).

The rightmost button determines whether the arc-line center position fits are shown or not. There are four options, which are cycled by clicking the button repeatedly:

Reference Spectrum And Pixel

This tab provides a set of options to set the reference spectrum and the reference pixel (see the section above on the spectrum image for more details). From the left-hand side to the right-hand side, the different options are:

The reference spectrum and the reference pixel can be selected interactively with a click using the left mouse button on the button (icon) with a yellow arrow. Thereafter, place the mouse cursor in the spectrum image and click (repeatedly) with the left mouse button on the required spectrum (row) and pixel (column). Finish the selection with a click on the right mouse button.

The selected reference spectrum number (IFU data) or spectrum order (non-IFU data) and reference pixel are shown in the editable fields on the right-hand side of the arrow button. Any (valid) integer can be typed into the fields to choose another value. The previous (next) spectrum number or spectrum order can be selected by clicking on the left-pointing (right-pointing) arrow button on the left-hand (right-hand) side of the left text field.

Pol. Order

The editable field on the right-hand side of the label shows the preset (linear) polynomial order. The preset value is read from the instrument-specific parameter file, using the parameter name 'polynomialorder_dm'; any positive integer is valid.

How to create a dispersion-mask image

There are no strict directions on how to create a dispersion mask using the tool of p3d. Here are, however, some guidelines on how you could proceed with your data.

1. Use an already created dispersion-mask image

Assume there is already a created dispersion-mask image for the same instrument configuration as the current data. If this is the case, this image can be used to create a good first-guess dispersion solution – using very few interactive steps.

At first, specify the starting-point disperson-mask image using the DISPMASKIN keyword. If the match between the already calculated line positions and the arc image look OK, proceed with the creation of the new dispersion mask; continue with step vi in the 'Fit the arc-line entries to the data'-subsection under section 4 below.

If the match does not appear to be OK, it might be necessary to create a new dispersion-mask image from scratch (using the following steps).

2. Determine exactly which arc lines are present in the data

p3d is in most cases able to make a qualified guess regarding which arc lamp(s) were used during the observations. In some cases, however, this is not possible. For example, the PPAK IFU of the PMAS IFS is (nowadays), as well as the VIRUS-P IFS, used with arc lamps placed in the dome; the data headers do not contain any information about the actually used lamps. Please see your notes, or make your own (qualified) guess on which arc-line lists to use.

When you know which arc lamps are present in your data, it is necessary to find out exactly which emission lines are present. The arc-line list files mostly contain the strong lines, as well as the very faint ones. Depending on the exposure time(s), the strong ones may be saturated – and should not be used then – and the faintest lines may not even be visible. Use available arc spectrum plots of separate arc lamps to identify the lines.

The VIMOS and FLAMES IFUs have pre-configured arc-line lists for all used instrument configurations. Use these line lists and save some work (plain-text versions of those arc-line files, which are used by p3d, are available in the directory p3d_path/data/tables/linelists).

3. Modify the minimum and the maximum intensities

Depending on the data, the default maximum and minimum intensities (histogram, 95%) might not reveal the faintest arc lines. Try to modify the minimum and maximum values by hand to find a range that works with the data. For example, click on the 'Viewing Options' tab and try the minimum value 0.0 and the maximum value 5000.0.

It might be easier to identify stronger lines with the help of an existing arc-spectrum plot if the 100% range (droplist) option is used.

4. Line mask functions – create the dispersion mask

The following functions, which are all found under the 'Line mask functions' tab, with the exception of the 'Manual Disp.' function found under the 'Dispersion' tab, are now used to create the dispersion-mask image.

i) Match the curvature

The initial dispersion mask is the same for all spectra; i.e., all arc-line list entries are straight lines in the spectrum image. To account for optical paths that vary with different spectra, by tracing the curved arc lines, use this function on either one or two of the brighter arc lines.

ii) Shift the mask

The wavelength-to-pixel accuracy that can be achieved with the first-guess dispersion solution might be low – possibly because of instrument flexure. Use this function to shift the entire mask of arc-line entries towards lower or higher-value pixels, to achieve a better match.

In case the amplitude shifts show a non-constant variation from the blue side to the red side, you could start by shifting the arc-line mask to allow you to identify the arc lines, before you proceed to the next step.

iii) Create a dispersion solution manually

For most instruments, the dispersion varies between the blue end and the red end. If this is the case with the data, it is necessary to replace the initial constant (or linear) dispersion solution with a dispersion solution created manually. To do this, follow the instructions for the 'Manual Disp.' function, which is found under the 'Dispersion' tab in the control panel.

iv) Delete the saturated and faint arc-line entries

Most default arc-line list files, but those for the VIMOS and the FLAMES IFSs, are likely to include many more entries than can be seen in the data. Over-exposed lines should not be used since they are not symmetric. Create a more accurate dispersion-mask image by deleting all superfluous entries; use the 'Delete lines' function.

It is a good idea to save the arc-line list once it is satisfactory. The next time any data are reduced, using the same instrument configuration, use the same arc-line file; either from within the dispersion-mask creation tool, or using the user-parameter file.

v) Match the arc-line entries with the data

In the next step, select the 'Match Centroids' function to have p3d match all the entries of the arc-line list with the data in the extracted image. Arc-line entries not found in the data, are then deleted. The arc-line entries are also better centered on top of the data.

vi) Fit the arc-line entries to the data

With the 'Fit Centroids' function, each entry in the arc-line mask is fitted accurately to the arc lines in the data.

Click on the long horizontal button under the 'Viewing Options' tab to view all or some of the fits – this allows eventual problems to be spotted.

vii) Check the Current Fit

Select the tab 'Check The Current Fit' and click the 'Residuals' button. Check the residuals for different values of the polynomial order to make a preliminary estimate of which order you can use. You might also spot problematic entries in the arc-line entry list.

viii) Create the dispersion mask

Select the 'Create Dispersion Mask' function to fit a polynomial to spectrum – where the wavelength as a function of pixel.

ix) Check the final fit

Use the 'Check The Final Fit' tab to interactively inspect the quality of the just created dispersion mask; thereby learn which arc-line entries are problematic, if any, and what the accuracy of the dispersion-mask image is.

x) Save the mask and exit

The final step is to save the dispersion-mask image. After selecting this function, the tools exits.

General information on how to learn more about this routine

The procedure of creating a dispersion mask is described in detail in the p3d A&A paper: Sandin et al. 2010, A&A, 515, 35

Instrument parameter files (*.prm), which are required to run this routine, are found in the directory "!p3d_path/data/instruments/*/" (outside IDL check "${p3d_path}/data/instruments/*/").

A default (master) user-parameter file is available in "!p3d_path/data/master_userparfile.dat". Copy this file to your data-reduction output directory, rename the file to, for example, "user_p3d.dat" and edit it to suit your needs.

Any plain text file can be used as an arc line-list file, where the wavelengths of (well) known emission lines are given in the first column (specified in [Å]ngström; columns are separated by white space). Default line-list files are available in the directory "!p3d_path/data/tables/linelists/". If you want to use your own arc line-list file, copy one of the default files to your data-reduction output directory and edit it, or create your own file. Edit the file(s) to suit your needs. To use the file in your reduction work you can select it from the dispersion-mask creation GUI, or by specifying it (or them) in your user-parameter file using the parameter "arclinefile".

NOAO provides spectrum plots of some arc lamps at the following address "http://www.noao.edu/kpno/specatlas/", use these plots, or similar ones for your arc lamp, to identify all the arc lines.

Examples on how to use this tool are provided in the EXAMPLES section below. Also see the tutorials in the directory "!p3d_path/doc/".

See the file README in the root of the p3d package for installation instructions, if p3d is not already installed on the computer.

Additional information, documentation, new versions, etc., can be found at the p3d project web site "https://p3d.sourceforge.io".

Requirements and format of the input and the output data

p3d only works with FITS-formatted images, i.e. fits files. Such files have a header that contains additional information about the observing conditions, the telescope, the instrument setup, and other useful information. Both input and output fits files can be compressed using gzip, to save storage space (also on Windows platforms); p3d handles such compressed files transparently.

This routine takes one, or several, raw arc images as input. The images are combined (by default), but p3d can also calculate a dispersion mask for each file if the parameter combineimages_dm is set to "no" (in the user-parameter file). Prescan and overscan regions must not be removed in the input data, they are removed by p3d during the reduction. Input files of instruments using several readout ports must also not be combined in advance, p3d combines the images automatically.

Additionally a file with a list of known arc lines is needed; p3d will attempt to use a default line list if it can. If p3d cannot figure out which file to use by itself, it queries for a file. It is also possible to specify a file (or several files) in the user-parameter file.

The output – the dispersion-mask image – consists of a list of //n + 1// parameters for each spectrum. These are the //n + 1// parameters used to fit an //n//:th order polynomial to each spectrum. There are always the same, pre-defined, number of extracted spectra in the output file. The number of pixels on the dispersion axis is the input size (where the prescan and the overscan regions have been removed). The spectra are ordered as in the input image – this is the so-called row-stacked spectrum (RSS) format. To create a spatial map of the intensities at any wavelength, it is necessary to use a fiber position table. In addition to the standard fits-file header keywords, the extracted spectrum images contain additional p3d-specific keywords:

The dispersion-mask image header also contains a few additional p3d-specific keywords:

If several input files were specified, which were thereafter combined by p3d, before extracting the spectra, the combined image contains the following p3d-specific header keywords:

Saved, and optionally already combined, bias-subtracted raw-data images contain only one additional header keyword:

Input parameters:

	filename		An array of strings that specifies the names of the raw data files, which are used when creating the dispersion mask.
	parfile		A scalar string that specifies the name of a file with instrument-specific setup parameters. These files follow a strict format. Already properly formatted files are available for all configured instruments in the directories p3d_path/data/instruments/*/. It is sufficient to specify the file basename, without the full path of the p3d subdirectory; for example: "nvimos_hr.prm".

Keyword parameters:

	masterbias		A scalar string with the name of a master-bias image file. This file is required if optimal extraction is used and none of the following four keywords is used: BIASPX, BIASPY, BIASOX, BIASOY. Note! This keyword must not be set if the prescan and the overscan regions should be used to estimate the bias level.
	tracemask		A scalar string with the name of a trace-mask image file.
	dispmaskin		A scalar string with the name of a starting dispersion-mask image file. When this keyword is present, the corresponding file is used to create the first-guess wavelength solution; instead of matching all arc lines individually, the new dispersion-mask image can be created immediately (and faster) using this functionality.
	bpmask		A scalar string with the name of a bad-pixels mask file; this could be, for example, a cosmic-ray mask file. Note! This mask is only used if optimal extraction is switched on in the user-parameter file.
	crmask		A scalar string, or a two-element array, with the name of a cosmic-ray mask image file; this could be, for example a bad-pixels mask image file. Note! This mask is only used if optimal extraction is switched on in the user-parameter file. Note! CRMASK in not used if CRCLEAN is set.
	biaspx		Set this keyword to use the x-axis prescan region when subtracting a bias. Specifically, a median value is calculated for each pixel on the y axis in the x-axis prescan region. The same array is used with all y-axis columns of bias subtracted data.
	biaspy		Set this keyword to use the y-axis prescan region when subtracting a bias. Specifically, a median value is calculated for each pixel on the x axis in the y-axis prescan region. The same array is used with all x-axis rows of bias subtracted data.
	biasox		Set this keyword to use the x-axis overscan region when subtracting a bias. Specifically, a median value is calculated for each pixel on the y axis in the x-axis overscan region. The same array is used with all y-axis columns of bias subtracted data.
	biasoy		Set this keyword to use the y-axis overscan region when subtracting a bias. Specifically, a median value is calculated for each pixel on the x axis in the y-axis overscan region. The same array is used with all x-axis rows of bias subtracted data.
	biasconstant		This keyword can be used in two ways: Set this keyword to use a constant value as a bias value; otherwise an array is used. All prescan and overscan regions specified using the keywords BIASPX, BIASPY, BIASOX, and BIASOY can be used when this keyword is set; otherwise, only one of them. If this keyword is set to a double type array or to a comma-separated string with values greater than one (when called from the shell), these values are used to create a bias. The number of elements must be either the same as the number of blocks or extensions, or if two input images are specified, two times that.
	savebiassub		Set this keyword to write the bias-subtracted raw-data image before the spectrum extraction. The name of the output file is the same as the (combined) input file, with the added intermediate suffix '_subbias'. Also see the user parameter 'savebiassub'.
	userparfile		A scalar string with the name of a file with user-defined parameters. The following parameters are read here from this file: 'keywordsfile' The name of the file with instrument-specific header keywords. 'gratingsfile_dm' To specify the instrument-specific gratings file. 'deadfibersfile' To specify a dead-fibers file. 'lampsfile_dm' To specify a lamps file. 'combineimages_dm' To specify if input images are to be combined or not. 'compress' Corresponds to COMPRESS. 'opfx' Corresponds to OPFX. 'sfx' Corresponds to SFX. 'icsfx' To specify the data output file intermediate suffix of combined files. 'dmsfx' To specify the data output file intermediate suffix of the created dispersion-mask image. 'mosfx' To specify the data output file intermediate suffix of the created bad pixels-mask image. 'mmsfx' To specify the data output file intermediate suffix of the created saturated pixels-mask image. 'obsfx' To specify the data output file intermediate suffix of the extracted spectrum image. 'exmonitor' Corresponds to EXMONITOR. 'a2' To use A3 instead of A4. 'a3' To use A3 instead of A4. 'a5' To use A3 instead of A4. 'letter' To use US Letter instead of A4 with PostScript and PDF output. 'legal' To use US Letter instead of A4 with PostScript and PDF output. 'tabloid' To use US Tabloid instead of A4 with PostScript and PDF output. 'psusletter' To use US Letter instead of A4 with PostScript and PDF output. 'methimcombine_dm' To specify the method used to combine input images. 'allinone' Corresponds to ALLINONE. 'nthreads_cr' Corresponds to CRNTHREADS. 'nthreads' Corresponds to NTHREADS. 'writesatmask' Corresponds to SATMASK. 'satlimit' To define the limit of saturated values. 'noobcheck' Corresponds to NOOBCHECK. 'crclean' Corresponds to CRCLEAN. 'nocrmask' To avoid using the cosmic-rays mask when images are cleaned of cosmic rays. 'biaspx' Corresponds to BIASPX. 'biaspy' Corresponds to BIASPY. 'biasox' Corresponds to BIASOX. 'biasoy' Corresponds to BIASOY. 'biasconstant' Corresponds to BIASCONSTANT. 'savebiassub' Corresponds to SAVEBIASSUB. 'methodextract_dm' To setup the spectrum-extraction method. 'dbin' To set the DBIN keyword of p3d_wavecal_dispmask_gui 'linewidth_dm' To setup the linewidth parameter. 'refdist_dm' To setup the refdist parameter. 'nrows_dm' To setup the nrows parameter. 'residualcut_dm' To setup the residualcut parameter. 'polynomialorder_dm' To setup the polynomial order parameter. 'fwhm_tr' To setup the fwhm parameter. 'centermethod_dm' To setup the centermethod parameter. 'linelistilimit_dm' To setup the linelistilimit parameter. 'vignettedfile_dm' To setup the vignetted spectra file. Additionally, the following user parameters are used by p3d_cr: 'laimagemethod' Corresponds to IMAGEMETHOD. 'laimageclean' Corresponds to IMAGECLEAN. 'ladispmedian' Corresponds to DISPMEDIAN. 'crwriteall' Corresponds to WRITEALL. 'nocrc' Corresponds to NOCRC. 'sigclip' Corresponds to SIGCLIP. 'objlim' Corresponds to OBJLIM. 'ratlim' Corresponds to RATLIM. 'crfwhm' Corresponds to FWHM. 'crgausskernelsize' Corresponds to GAUSSKERNELSIZE. 'fwhm_tr' Used to set the width of the Gaussian profile when crfwhm is unavailable and IMAGEMETHOD is unset. 'sigfrac' Corresponds to SIGFRAC. 'growradius' Corresponds to GROWRADIUS. 'maxiter' Corresponds to MAXITER. 'cleansfx' Corresponds to CLEANSFX. 'masksfx' Corresponds to MASKSFX. 'laplsfx' Corresponds to LAPLSFX. 'poflsfx' Corresponds to POFLSFX. 'siprsfx' Corresponds to SIPRSFX. 'laprsfx' Corresponds to LAPRSFX. 'finesfx' Corresponds to FINESFX. 'wallsfx' Corresponds to WALLSFX. Note! If any of the corresponding keywords is specified, that value takes precedence.
	arclinefile		A scalar or one-dimensional array of strings with the names of plain-text arc-line list file(s); this keyword is passed on to the routine p3d_wavecal_set_linelist.pro.
	linelistilimit		A scalar decimal value that can be used to select arc lines brighter than the value defined by this parameter; LINELISTILIMIT > 0. This parameter is only considered if the arc-line list file(s) contain(s) at least two columns, where the arc-line intensity is set in the second column. Entries that do not have any intensity specified in the second column are always used. Also see the user parameter 'linelistilimit_dm'.
	ofilename	X	This keyword returns the full name of the created dispersion-mask image file.
	opath		A scalar string that specifies the path where the output data are saved. The default value is: '.'
	opfx		A scalar string that specifies a prefix for the (automatically generated) output filename. The default value is: ''
	sfx		A scalar string that specifies the raw data filename suffix to use. The default value is: .fits
	detector		A scalar integer that specifies the currently selected detector; DETECTOR is a zero-based value. The default value is: 0
	dbin		This parameter determines if the input data are re- binned on the dispersion axis (DBIN = 2 || 3), or if data are kept as they are (DBIN = 1). A good reason to rebin data are if all pixels should fit on the screen. The different values allowed are: DBIN = 1: Data are not rebinned. DBIN = 2: Data are rebinned by a factor 2. DBIN = 3: Data are rebinned by a factor 2 until all pixels fit on the screen. The default value is: 1
	track		If this keyword is set, the dispersion mask creation GUI is set up with a status line. The default value is: 1
	exmonitor		If this keyword is set, a line-profiles viewer is shown after all fitting is done when using optimal extraction.
	compress		Set this keyword to compress all output files using gzip; the file suffix '.gz' is then appended to all output filenames. The default value is: 0
	logfile		This keyword specifies the name of an optional log file when p3d_cdmask is launched as a separate program.
	loglevel		If this routine is launched as a separate program, and LOGFILE is used, this keyword specifies the log level; [1], 2, 3 (the higher the value, the more output). The default value is: 1
	colortable		A scalar string or integer that specifies which color table should be loaded. If COLORTABLE is a string, the string must contain two parts separated by a comma, without any whitespace between. The first part before the comma specifies the color-table number in the file name specified after the comma; the file must be formatted as described in the documentation for MODIFYCT. As an example, COLORTABLE='25,brewer.tbl'. Components in the ColorBrewer color-table file "brewer.tbl" found in the resource directory can be specified explicitly by preceeding the string with "CB"; for example, COLORTABLE='CB25'. If COLORTABLE is an integer, the integer must be given in the range -4, -3, -2, -1, 0, ..., up to the maximum number of available tables in IDL, plus the available tables in the ColorBrewer file (resource/) "brewer.tbl". Select -4, -3, -2, or -1 to load the cubehelix, 2 * Califa, and the Sauron color tables, the other values use the respective color map as defined by LOADCT. These are the permitted values: -4 Loads the cubehelix color table. -3 Loads the Califa project intensity color table, as defined September 2012. -2 Loads the Califa project velocity field color table, as defined September 2012. -1 Loads the Sauron color table (default). 0-74 (IDL version >= 8.3) Loads the corresponding color table with LOADCT. 0-40 (IDL version < 8.3) Loads the corresponding color table with LOADCT. 'CB0'-'CB85' Loads the corresponding ColorBrewer color table; the color tables are defined in the file "resource/brewer.tbl" 75-159 (IDL version >= 8.3) Loads the ColorBrewer color table corresponding to this number after 75 is subtracted; the color tables are defined in the file "resource/brewer.tbl" 41-125 (IDL version < 8.3) Loads the ColorBrewer color table corresponding to this number after 41 is subtracted; the color tables are defined in the file "resource/brewer.tbl" 'x,ctfile' Loads the x:th entry ('x' must be an integer) in the color-table file 'ctfile. The default value is: -1
	invert		Set this keyword to invert the loaded color table.
	cinv		Set this keyword to plot white lines on a black background instead of the default, which is to plot black lines on a white background.
	bottom		A scalar integer that specifies the bottom colormap index to use with the data. Note, 0 <= BOTTOM <= !d.table_size – 1.
	cindex		An array of integers specifying the color indices used as reserved colors. CINDEX ideally has 7 elements, P3D_SV uses 7 reserved colors. If the number of elements is smaller, the upper range of reserved colors will use the largest index available (also see CINDV).
	cindv		An array of 7 integers specifying which of the indices in CINDEX are used as the reserved colors: If CINDEX has 7 elements then: cindv = CINDEX[0:6] If CINDEX has 3 elements then: cindv = [CINDEX[0], CINDEX[1], CINDEX[2], CINDEX[2], CINDEX[2], CINDEX[2], CINDEX[2]]
	ch_start		This scalar decimal value defines the color when COLORTABLE = -4. The default value is: 0.5
	ch_rots		This scalar decimal value defines the rotation in color when COLORTABLE = -4. The default value is: -1.5
	ch_hue		This scalar decimal value defines the hue intensity scaling when COLORTABLE = -4. The default value is: 1.2
	ch_gamma		This scalar decimal value defines the gamma correction factor when COLORTABLE = -4. The default value is: 1.0
	satmask		Set this keyword to write saturated-pixels mask images to the disk. Such images indicate if any saturated pixels in the input images were used when calculating a spectrum image.
	crnthreads		A scalar integer that specifies how many threads used when cleaning single input images of mic-ray hits. If this keyword is not set, READS is used instead. If nthreads was not set her, the default is to use the maximum number of ilable threads minus one, but the maximum number 16: (!cpu.hw_ncpu – 1) < 16. The default value is: max-1
	nthreads		A scalar integer that specifies how many threads to use in the parallelized line profile calculations. The calculation speed scales nearly linearly with the number of threads. The default is to use the maximum number available, but no more than 16: max = !cpu.hw_ncpu < 16. The default value is: max
	allinone		Unset this keyword to write the output data to separate files, instead of writing the extracted spectra to the first (DATA) extension, its error to the ERROR extension, the bad-pixels mask image to the MASK extension, the saturated-pixels mask image to the SATMASK extension, and the data-quality mask image to the QUALITY extension. The default value is: 1
	crclean		Set this keyword to clean all uncombined input images of cosmic-ray hits before the extraction procedure is started; either a master-bias image or prescan and overscan regions must be used to provide the bias level to use this functionality.
	nocrmask		Set this keyword to avoid using the cosmic-ray hit mask image, which is created when CRCLEAN is set, when extracting the spectra optimally. Normally it is recommended to keep this keyword unset; it is available to allow the user the ultimate choice. The following keywords are used to configure the cosmic-ray cleaning procedure; they are only used if CRCLEAN is set; see p3d_cr and p3d_tool_crays_lapycosmic for more details:
	sigclip		A scalar decimal value that specifies the clipping sigma.
	objlim		A scalar decimal value that specifies the object limit value; this value is only used if IMAGEMETHOD is set.
	ratlim		A scalar decimal value that specifies the ratio limit value; this value is only used if IMAGEMETHOD is unset.
	crfwhm		A decimal scalar value, or a two-element array, that specifies the full width at half maximum of the gaussian kernel used to convolve the raw data; this keyword corresponds to FWHM in p3d_cr, and it is only used if IMAGEMETHOD is unset.
	gausskernelsize		A scalar integer that specifies the size of the Gaussian kernel used when convolving the input image with a Gaussian kernel using the PyCosmic algorithm. This value is by default set to 2 * //q// + 1, where //q// is the maximum value of the //x// and the //y// pixel where a one-dimensional Gaussian reaches 15% of its maximum value (1.0); this condition makes use of the full-width-at-half-maximum value of FWHM. The value must be odd, and the maximum value is in any case 19. Be aware that convolution times increase strongly with the size of the Gaussian kernel.
	sigfrac		A scalar decimal value that specifies a scale factor for pixels surrounding a cosmic-ray masked pixel; this parameter is only used if GROWRADIUS is non-zero. The default value is: 0.5
	growradius		Set this keyword to grow all cosmic-ray hit pixels by a frame this many pixels wide; this is done to check if surrounding pixels are affected by cosmic-ray hits as well. The default value is 1, the values 0 and 2 are also accepted. Surrounding pixels are checked if they are cosmic-ray affected using the first criterion S' > SIGFRAC * SIGCLIP. The default value is: 1
	maxiter		A scalar integer value that specifies the maximum number of iterations used when identifying cosmic-ray hits in each image.
	imagemethod		Set this keyword to use the original algorithm of P. van Dokkum (L.A. Cosmic), which was developed for cosmic-ray detection in images. The default is otherwise to use the IFS-modified algorithm with IMAGEMETHOD unset.
	imageclean		Set this keyword to use the original two-dimensional median filter when replacing the flux in masked pixels with that of the surrounding pixels. Otherwise an [11 pixel] one-dimensional median filter is used. The default value is: 1
	dispmedian		Set this keyword to a positive integer > 1 to subtract sampling flux along the dispersion axis. The value defines the width of the median window.
	writeall		Set this keyword to write all intermediate images in addition to the cleaned image and the cosmic-ray mask.
	showcrgui		Set this keyword to show a properties GUI of the cosmic-ray cleaning parameters before the algorithm is executed.
	nocrc		Set this keyword to force the use of the slower IDL-based cosmic-ray cleaning routine instead of the faster C-based routine.
	fnotify		Set this keyword to the name of a file that is written when the tool has finished; this can be a useful feature when using the p3d queue handler. In that case, set this keyword and check if the file exists before continuing. And do not forget to remove the file.
	font		Set this scalar string to the name of the font to be used with all widget components of this tool. It is advised to use a fixed font, the outcome is otherwise difficult to predict. Set the keyword to 'big' to use the following pre-defined font ed-medium-r-normal--18-120-100-100-c-90-iso8859-1'; this could be useful on a system using a 4k screen (with 3160x2560 pixels). Another option to specify a font is to set the environment variable p3d_font or the p3d IDL system variable !p3d_font. The default font used with the widgets is, for your information, printed to the screen when FONT is unset and the highest verbosity level is used (verbose = 3). On Linux machines, use a shell command such as 'xlsfonts' to list all available fonts.
	cmdline	X	This keyword is used then the tool is launched by the p3d server.
	gui	X	If this keyword is set, the routine is assumed to be called from the p3d GUI – meaning that there is no initialization of the p3d system variables.
	eventwid	X	This variable is set to the widget id of a button widget that will receive an event from p3d_wavecal_dispmask_gui once it finishes.
	stawid	X	If set, various messages are written to the p3d GUI status line (this must be the widget id of that label widget).
	topwid	X	If set, error messages are displayed using DIALOG_MESSAGE, using this widget id as DIALOG_PARENT, instead of MESSAGE.
	logunit	X	Messages are saved to the file pointed to by this logical file unit, if it is defined.
	verbose		Set this parameter to a positive scalar integer to make p3d write some information on STDOUT about what is going on. The following four values are acknowledged: 0 Writes no information at all. This is the default. 1 Writes the more important information; regarding subroutine configurations, mostly. 2 Writes most information; includes a more verbose output than 1. 3 Writes all information, including information on the execution state of GUI subroutines. This may be a useful mode when debugging the code.
	noobcheck		Set this keyword to avoid checking the OB header keyword; this only applies to the instruments that use it. To see if it applies to your instrument, check if the OBID parameter is defined in the instrument keywords file.
	quiet		Set this keyword to not print the disclaimer on the screen when this tool is started from the command line or from the shell.
	error	X	This scalar integer returns an error code if set. Any value different from zero means that an error has occurred.
	debug	X	An error handler is not setup if this keyword is set. Otherwise, the error handler is setup using the routine CATCH, which makes each subroutine exit quietly when a bug is encountered. Use this keyword when debugging p3d.
	cdebug		A keyword as DEBUG used with the called C routines; valid values are 0-6 where a higher number results in more output to STDERR.
	help	X	Set this keyword to show this routine documentation and then exit.

Output parameters:

	saved	X	An integer scalar, set to 1 if a dispersion mask was created, otherwise it is set to 0.
	out	X	Upon a successful execution, this variable contains the dispersion mask (data) when the routine exits.

Side effects:

The default widget font is changed if the keyword FONT is used.

Examples:

This routine can be called in four different ways (for use on WINDOWS platforms, replace the path character '/' with '\'):

i) Using the p3d GUI

There is no need to setup the input parameters, p3d handles this transparently.

ii) As a stand-alone tool from the IDL command line

In addition to your input arc images, it is necessary to pass the name of the (provided) instrument-specific parameter file, a master-bias image file or (at least) one of the bias prescan and overscan parameters (if you use optimal extraction), a trace-mask image file, as well as any additional arguments to p3d:

files = ['arcfile_1.fits.gz', 'arcfile_2.fits.gz']
parfile = 'larr2k.prm'
opath = 'Output'
masterbias = opath + '/masterbias.fits.gz'
tracemask = opath + '/tracemask.fits.gz'
userparfile = opath + '/user_p3d.dat' ;; If you use this file
detector = 0 ;; Required with multi-detector configurations.
p3d_cdmask, files, parfile, masterbias=masterbias, $
    tracemask=tracemask, userparfile=userparfile, opath=opath, $
    detector=detector, logfile=opath + 'dred.log', /verbose

iii) As a stand-alone tool from a terminal, using the IDL VM

In addition to your input arc images, it is necessary to pass the name of the (provided) instrument-specific parameter file, a master-bias image file or (at least) one of the bias prescan and overscan parameters (if you use optimal extraction), a trace-mask image file, as well as any additional arguments to p3d. The recommended approach is to use the provided shell script p3d_cdmask_vm.sh or the Python tool p3d_dispatch.py:

# Using the shell script, making use of the p3d queue system:
files=arcfile_1.fits.gz,arcfile_2.fits.gz
parfile=larr2k.prm
opath=Output
masterbias=${opath}/masterbias.fits.gz
tracemask=${opath}/tracemask.fits.gz
userparfile=${opath}/user_p3d.dat # If you use this file
detector=0 # Required with multi-detector configurations.
p3d_cdmask_vm.sh $files $parfile masterbias=$masterbias \
    tracemask=$tracemask userparfile=$userparfile opath=$opath \
    detector=$detector logfile=$opath/dred.log /verbose

# Using the Python tool, without using the p3d queue system:
files=arcfile_1.fits.gz,arcfile_2.fits.gz
parfile=larr2k.prm
opath=Output
masterbias=${opath}/masterbias.fits.gz
tracemask=${opath}/tracemask.fits.gz
userparfile=${opath}/user_p3d.dat # If you use this file
detector=0 # Required with multi-detector configurations.
p3d_dispatch.py --noqueue cdmask $files $parfile \
    masterbias=$masterbias tracemask=$tracemask \
    userparfile=$userparfile opath=$opath detector=$detector \
    logfile=$opath/dred.log /verbose

Note: In this approach, it is necessary to specify all options using the full names of the keywords. Partially correct keyword names are not permitted, as they are inside IDL. Hence, as an example, you must type "p3d_cdmask_vm.sh tracemask=$tracemask" and -NOT- "p3d_cdmask_vm.sh tr=$tracemask".

Note: The tool launches in the current directory, which is why it is unnecessary to specify full paths if files reside in the current directory.

Note: When several files are specified in the first argument, they must be separated with either a comma or a semicolon.

Note: Parameters indicated with an 'X' in the list above are unavailable in this approach. This includes the two output variable parameters OFILENAME, and OUT.

iv) As a stand-alone tool from Python, using the IDL VM

This approach is analogous to iii), but is executed from the Python command line. Here is an example:

# Launch p3d_cdmask in a single IDL session, without using the
# p3d queue system:
from p3d_dispatch import *
files = 'arcfile_1.fits.gz,arcfile_2.fits.gz'
parfile = 'larr2k.prm'
opath = 'Output'
userparfile = opath + '/user_p3d.dat' # If you use this file
detector = 0 # Required with multi-detector configurations.
masterbias = opath + '/masterbias.fits.gz'
tracemask = opath + '/tracemask.fits.gz
commandline = files + ' ' + parfile + ' masterbias=' + masterbias +
  ' tracemask=' + tracemask ' userparfile=' + userparfile +
  ' opath=' + opath + ' detector=' + detector +
  ' logfile=' + opath + '/dred.log /verbose'
p3d_dispatch('cdmask', commandline, noqueue=True)

Combines several spectrum-image files with extracted spectra covering the same region, either just taking the average or accounting for saturated regions in the longer exposures.

There are (at least) two reasons to take multiple exposures of the same region on the sky. The first reason is to achieve an increased signal-to-noise ratio. Several exposures are therefore taken, possibly using different exposure times, and these exposures are then combined. A second reason is that one desires complete measurements of an object that shows a large dynamic range; some parts of the object might be saturated at longer exposure times.

In the first case, shorter exposures could be combined already before spectra are extracted, but this is not recommended with longer exposures. With longer exposure times, the observing conditions are likely significantly different for the separate exposures. Important effects that affect the observations include instrument flexure and differential atmospheric refraction, which both vary with time. In the latter case, it is therefore recommended to combine the exposures after they have been extracted and corrected.

This routine handles both a straight combination of files, and accounting for saturated pixels in the raw data using different exposure times (see the keyword SATURATED). The result is, in both cases, a new merged data set.

The output data files – where a set consists of a combined spectrum image, a combined error image, a combined bad-pixels mask image, a combined saturated-pixels mask image, and a combined data-quality mask image – are by default written to the same file, in consecutive extensions. They are instead written to separate files if the keyword ALLINONE or the user-parameter 'allinone' is unset. There is no real advantage to either approach; although, if separate files are used and the error file, the combined bad-pixels mask image file, or the combined saturated-pixels mask image, are renamed they will not be found by other tools.

Note: It is assumed that the exact same region was observed in all data files. Data are not checked to ensure this assumption.

The method of combining data – just adding them up

• The input files are checked for consistency

  Here, the selected input files are first checked for consistency. In this case, it means that the images of all files must be of the same size, and they must all be wavelength calibrated with the same value on CRVAL and CDELT (to produce the exact same wavelength array).

  If the extracted-spectrum image error file (IMERR) is found, a merged error file is created along with the main data file.

• Data are merged

  The separate images are added up, dividing with the number of exposures, weighting with the individual exposure time.

  Output bad-pixels mask images are used here if such files exist (these files are indicated with the keyword IMONC in the file header). Only pixels where the bad-pixels mask is smaller than or equal to a limiting number are used. The limiting number is set with the keyword BADPIXNLIMIT or the user parameter 'badpixnlimit'; its default value is 3, which means that three pixels across the cross-dispersion profile of the raw science image were masked as bad pixels or cosmic-ray affected pixels.

• The output filename

  Based on the first input image file, a suffix is added at the end of the input filename.

The method of combining data – accounting for saturated spectra

• Creation of a saturated-pixels mask image

  During the (object) spectrum extraction a saturated-pixels mask image is written to the disk along with the extracted spectra; if the SATMASK keyword is set. That mask indicates which spectrum elements, for all spectra and wavelength bins, included saturated pixels in the raw data.

• The input files are checked for consistency

  Here the selected input files are first checked for consistency. In this case it means that the images of all files must be of the same size, they must all be wavelength calibrated with the same value on CRVAL and CDELT (to produce the exact same wavelength array), and the exposure time must be different in all files.

  The filename of the saturated-pixels mask image must be available in the IMSMK header keyword, and the corresponding file must also exist in the same directory as the extracted spectrum image file. Additionally, if the extracted-spectrum image error file (IMERR) is found, then a merged error file is created along with the main data file.

• Data are merged

  The extracted spectrum image, along with its saturated-pixels mask image, that used the longest exposure time is selected as reference. The remaining images are sorted with decreasing exposure time.

  For each spectrum in the main image, all pixels p in the range p – PIXRANGE <= p <= p + PIXRANGE are masked to be replaced with the corresponding spectrum pixel value in the next shorter exposure. The pixels p are the previously masked pixels of the current spectrum in the saturated-pixels mask image. The value of the PIXRANGE parameter is changed using the keyword with the same name. Before the spectrum pixels of the shorter exposure are inserted in the main image they are multiplied with the scaling factor; exptime(reference) / exptime(shorter).

  If the image error files corresponding to the main data are found, they are used to create a modified error image file using the same masks.

• The output filename

  Based on the input image file with the longest exposure time, adding a suffix at the end of the input filename.

General information on how to learn more about this routine

Instrument parameter files (*.prm), which are required to run this routine, are found in the directory "!p3d_path/data/instruments/*/" (outside of IDL, check "${p3d_path}/data/instruments/*/").

A default (master) user-parameter file is available in "!p3d_path/data/master_userparfile.dat". Copy this file to your data-reduction output directory, rename the file to, for example, "user_p3d.dat" and edit it to suit your needs.

Examples on how to use this routine are provided in the EXAMPLES section below.

See the file README in the root of the p3d package for installation instructions, if you have not already installed p3d on your computer.

Additional information, documentation, new versions, etc., can be found at the p3d project web site "https://p3d.sourceforge.io".

Requirements and format of the input and the output data

p3d only works with FITS-formatted images, i.e. fits files. Such files have a header that contains additional information about observing conditions, the telescope, the instrument setup, and other useful information. Both input and output fits files can be compressed using gzip, to save storage space (also on Windows platforms); p3d handles such compressed files transparently.

This routine takes two or more row-stacked spectrum (RSS) images as input; this is the default format of all extracted data of p3d.

Extracted data can, in addition to an RSS file, also be saved using the Euro3D format (use the keyword SAVEE3D or the user parameter 'savee3d'), and data of instruments using square-shaped spatial elements are easily converted to cube format using the tool p3d_rss2cube. In addition to the standard fits-file header keywords, the RSS image header contains additional p3d-specific keywords:

An additional file is written when output bad-pixels mask images are used and SATURATED is unset; this file contains a final mask, which indicates pixels that did not contain any unmasked pixels at all:

Input parameters:

	filenames		An array of strings that specifies the names of the extracted and calibrated spectrum-image files that will be combined.
	parfile		A scalar string that specifies the name of a file with instrument-specific setup parameters. These files follow a strict format. Already properly formatted files are available for all configured instruments in the directories p3d_path/data/instruments/*/. It is sufficient to specify the file basename, without the full path of the p3d subdirectory; for example: "nvimos_hr.prm".

Keyword parameters:

	saturated		Set this keyword to use the second combination approach, where saturated pixels in the extracted data are accounted for by replacing data of longer exposures in affected regions with data of shorter exposures.
	badpixnlimit		Output bad-pixels images contain a number for ach spectrum and wavelength bin. This number is he number of bad pixels (or cosmic-ray masked ixels) used during spectrum extraction. Only pectrum-image pixels that have a bad-pixels image ixel value less than or equal to the value in this arameter are considered in the image combination, hen SATURATED is unset. The resulting spectrum ntensity (error) is set to 0.0 (1.0) if the same ixel in all images are masked as bad pixels with value higher than BADPIXNLIMIT. Allowed values re 1 <= BADPIXNLIMIT <= 5. This keyword is only onsidered if output bad-pixels mask images are ound. The default value is: 3
	pixrange		A scalar integer that specifies a range of pixels on the lower side and the upper side of the saturated pixel of each spectrum. All pixels in this range are replaced with the corresponding pixel value of the file with the next shorter exposure. The default value is: 30
	compress		Set this keyword to compress all output files using gzip; the file suffix '.gz' is appended to all output filenames. The default value is: 0
	savee3d		The output data are saved in the Euro3D (e3d) format in addition to the usual RSS format when this keyword is set.
	postable		An optional scalar string with the name of a fiber positition-table file. This keyword is only required if the data were not reduced using p3d. In that case, the header keywords IMTYPE (and POSTAB) are undefined; p3d uses these keywords to determine which file to use. This keyword is only considered if SAVEE3D is set.
	userparfile		A scalar string with the name of a file with user- defined parameters. The following parameters are read here from this file: 'keywordsfile' The name of the file with instrument-specific header keywords. 'compress' corresponds to COMPRESS. 'allinone' corresponds to ALLINONE. 'savee3d' corresponds to SAVEE3D. 'opfx' corresponds to OPFX. 'cexsfx' To specify the data output file suffix. 'mmsfx' To specify the data output file suffix. 'mosfx' To specify the data output file suffix. 'badpixnlimit' correponsds to BADPIXNLIMIT.
	ofilename	X	This keyword returns the full name of the combined spectrum-image file.
	opath		A scalar string that specifies the path where the output data are saved. The default value is: '.'
	opfx		A scalar string that specifies a prefix for the (automatically generated) output filename. The default value is: ''
	detector		A scalar integer that specifies the currently selected detector; DETECTOR is a zero-based value. The default value is: 0
	deadfibersfile		An optional scalar string with the name of a file that lists dead, unused, and low-transmission fibers. The file must conform to the standard of the regular dead-fibers files of p3d. If this keyword is not set, the corresponding file is used as provided in the instrument-specific parameter file PARFILE.
	logfile		This keyword specifies the name of an optional log file.
	loglevel		If this routine is launched as a separate program, and LOGFILE is used, this keyword specifies the log level; [1], 2, 3 (the higher the value, the more output). The default value is: 1
	allinone		Unset this keyword to write the output data to separate files, instead of writing the combined spectra to the first (DATA) extension, the error image to the ERROR extension, the combined bad-pixels mask image to the MASK extension, the combined saturated-pixels mask image to the SATMASK extension, and the combined data-quality mask image to the QUALITY extension. The default value is: 1
	fnotify		Set this keyword to the name of a file that is written when the tool has finished; this can be a useful feature when using the p3d queue handler. In that case, set this keyword and check if the file exists before continuing. And do not forget to remove the file.
	cmdline	X	This keyword is used then the tool is launched by the p3d server.
	gui	X	If this keyword is set, the routine is assumed to be called from the p3d GUI – meaning that there is no initialization of the p3d system variables.
	subroutine	X	If set, parts of the routine opening message will not be written to the log file.
	stawid	X	If set, various messages are written to the p3d GUI status line (this must be the widget id of that label widget).
	topwid	X	If set, error messages are displayed using DIALOG_MESSAGE, using this widget id as DIALOG_PARENT, instead of MESSAGE.
	logunit	X	Messages are saved to the file pointed to by this logical file unit, if it is defined.
	verbose		Set this parameter to a positive scalar integer to make p3d write some information on STDOUT about what is going on. The following four values are acknowledged: 0 Writes no information at all. This is the default. 1 Writes the more important information; regarding subroutine configurations, mostly. 2 Writes most information; includes a more verbose output than 1. 3 Writes all information, including information on the execution state of GUI subroutines. This may be a useful mode when debugging the code.
	quiet		Set this keyword to not print the disclaimer on the screen when this tool is started from the command line or from the shell.
	error	X	This scalar integer returns an error code if set. Any value different from zero means that an error has occurred.
	debug	X	An error handler is not setup if this keyword is set. Otherwise, the error handler is setup using the routine CATCH, which makes each subroutine exit quietly when a bug is encountered. Use this keyword when debugging p3d.
	help	X	Set this keyword to show this routine documentation, and then exit.

Output parameters:

out

X

Upon a successful execution this variable contains the combined spectrum-image data when the routine exits.

Examples:

This routine can be called in three different ways (for use on WINDOWS platforms, replace the path character '/' with '\'):

i) As a stand-alone tool from the IDL command line

In addition to the extracted and calibrated RSS images, it is necessary to pass the name of the (provided) instrument-specific parameter file as well as any additional arguments:

files = ['file1.fits.gz', 'file2.fits.gz']
parfile = 'nvimos_hr.prm'
opath = 'Output'
userparfile = opath + '/user_p3d.dat' ;; If you use this file
p3d_cexposure, files, parfile, pixrange=40, $
    userparfile=userparfile, opath=opath, $
    logfile=opath + 'dred.log', /verbose

ii) As a stand-alone tool from a terminal, using the IDL VM

In addition to your RSS images, it is necessary to pass the name of the (provided) instrument-specific parameter file as well as any additional arguments to p3d_cexposure. The recommended approach is to use the provided shell script p3d_cexposure_vm.sh or the Python tool p3d_dispatch.py:

# Using the shell script, making use of the p3d queue system:
files=file1.fits.gz,file2.fits.gz
parfile=nvimos_hr.prm
opath=Output
userparfile=${opath}/user_p3d.dat # If you use this file
p3d_cexposure_vm.sh $files $parfile pixrange=40 \
    userparfile=$userparfile opath=$opath \
    logfile=$opath/dred.log /verbose

# Using the Python tool, without using the p3d queue system:
files=file_oextr_1.fits.gz,file_oextr_2.fits.gz
parfile=nvimos_hr.prm
opath=Output
userparfile=${opath}/user_p3d.dat # If you use this file
p3d_dispatch.py cexposure $files $parfile pixrange=40 \
    userparfile=$userparfile opath=$opath \
    logfile=$opath/dred.log /verbose

Note: In this approach, it is necessary to specify all options using the full names of the keywords. Partially correct keyword names are not permitted, as they are inside IDL. Hence, as an example, you must type "p3d_cexposure_vm.sh userparfile=$userparfile" and -NOT- "p3d_cexposure_vm.sh user=$userparfile".

Note: The tool launches in the current directory, which is why it is unnecessary to specify full paths if files reside in the current directory.

Note: When several files are specified in the first argument, they must be separated with either a comma or a semicolon.

Note: Parameters indicated with an 'X' in the list above are unavailable in this approach. This includes the three output variable parameters OFILENAME, OUT, and DOUT.

iii) As a stand-alone tool from Python, using the IDL VM

This approach is analogous to ii), but is executed from the Python command line. Here is an example:

# Launch p3d_cexposure in a single IDL session, without using the
# p3d queue system:
from p3d_dispatch import *
files = 'file1.fits.gz,file2.fits.gz'
parfile = 'vimos_hr.prm'
opath = 'Output'
userparfile = opath + '/user_p3d.dat' # If you use this file
commandline = files + ' ' + parfile + ' pixrange=40' +
  ' userparfile=' + userparfile + ' opath=' + opath +
  ' logfile=' + opath + '/dred.log /verbose'
p3d_dispatch('cexposure', commandline, noqueue=True)

Creates an image with extracted spectra for the subsequent flat fielding (normalization) of extracted science spectra.

Flat-field spectra are used to normalize extracted science spectra for instrument differences in the optical path. The approach of p3d is to first extract the spectra in a flat-field (continuum) image through a call to this routine, and thereafter perform the normalization using these spectra when calling p3d_cobjex. The actual normalization is carried out by p3d_tool_flatfield (which is called from p3d_cobjex), after data have been wavelength calibrated.

All parts of the normalization procedure, including all related user-parameters, and also including the normalization performed first when calling p3d_cobjex, are described below. The best procedure depends on properties of available data and observing conditions. The default behavior also dependends on the instrument.

Please, after having read through the documentation below, also see the p3d A&A paper as well as the master user-parameter file and the p3d documentation WIKI, for help on how to setup the parameters to allow an accuracy as high as possible in the flat fielding.

The output-data files – consisting of the extracted spectrum image, the error image, the bad-pixels mask image, the saturated-pixels mask image, the data-quality mask image, and the fiber-to-fiber transmission array – are by default written to the same file in consecutive extensions. They are instead written to separate files if the keyword ALLINONE, or the user-parameter 'allinone', is unset. There is no real advantage to either approach; although, if separate files are used and the error file, the output bad-pixels mask image file, the saturated-pixels mask image file, or the fiber-to-fiber transmission array file are renamed, they will not be found by other tools.

This tool calls p3d_cr if requested to do so, which cleans single input images of cosmic-ray hits; see the keyword /CRCLEAN. The outcome of the cleaning procedure depends both on properties of the input image as well as the algorithm parameters. Different parameters might produce a different outcome. There is no guarantee whatsoever that the cleaning removes only cosmic-ray hits. Check the output images to build an own opinion regarding the quality of the outcome.

Data normalization; replace data with polynomial fits

The first task is to smooth data and replace them with a polynomial fit (see the user parameters 'smowidth_ff' and 'deg_polyfit_ff'). This smoothing is made to reduce the influence of errors in the flat-field data. These tasks can both be switched off by setting the respective parameter to zero.

The flat-field data smoothing is performed both by this tool, and when the final flat-field normalization is made through a call from the tool p3d_cobjex. However, when called from here, the smoothed data are only used to calculate a fiber-to-fiber transmission array.

This task is by default used by all instruments, with an exception for VIMOS. Data of the VIMOS instrument show wavelength-dependent fringes for several grisms and spectra, which could not be corrected for if flat-field data are first smoothed. Check the instrument parameter file for the default values on the two parameters mentioned above.

Data normalization; dividing all spectra with their mean

This task is used when flat-field data are normalized after a call to the tool p3d_cobjex; it is also activated by default, please see the user-parameter 'ff_normdisp' for more details.

To perform a wavelength-dependent normalization of flat-field data, all spectra are divided by the mean spectrum. Only spectra not listed in the dead-fibers file (if present), are used when calculating a mean spectrum.

Data normalization; making a fiber-to-fiber correction

This task is used when flat-field data are normalized after a call to the tool p3d_cobjex; it is deactivated by default since its main use is with twilight flat-field images. Please see the keyword NF2F and the user parameter 'ff_normf2f' for more details.

After all flat-field spectra have been replaced with fitted polynomials (if this functionality is used), and have been divided by a mean spectrum (if this functionality is used), p3d can normalize data anew using a normalized fiber-to-fiber transmission array. Such a transmission array is always calculated and saved (but not applied) when spectra are first extracted (through a call to p3d_cflatf). The name of the transmission-array file is written to the extracted flat-field image data header, using the keyword IMTRM. When the flat-field data are finally normalized in the subsequent call to p3d_cobjex, it is the transmission-array file recorded in the IMTRM keyword of the flat-field data header that is used.

Instead of calculating a new fiber-to-fiber transmission array, this tool allows the specification of a separate transmission-array file; using the input keyword TRANSMISSION, or the user parameter 'ff_trm_file'. Specify such a separate file when calling p3d_cflatf if the intention is to use the transmission file of extracted twilight flat-field image(s), and otherwise lamp-flat image(s). The same data are used for this purpose if the keyword IMTRM is unavailable in the extracted flat-field header.

Before the fiber-to-fiber transmission array is calculated, the lower PCUTLOW (user parameter 'ff_lowerpx') pixels and the upper PCUTHIGH (user parameter 'ff_upperpx') pixels are removed from the spectrum. These two parameters are used to exclude regions on the CCD where there is no signal. The transmission array could contain spurious values if this functionality is not used. The default values are 5% of the pixel range on the dispersion axis. As an example, with the HR-blue grism of the VIMOS instrument it is recommended to use PCUTLOW = 32.0.

Attention: The fiber-to-fiber transmission correction is mostly, and perhaps only, useful if the intention is to use a separate twilight flat-field image for the fiber-to-fiber transmission correction (please see the instructions in the next section on how to do this). It is in other cases recommended to use only a normalization by the flat-field mean spectrum, and keep NF2F unset.

Using both a twilight flat-field image and a lamp-flat image

Use the following approach if the intention is to use a twilight-flat image to make a fiber-to-fiber transmission correction and a lamp-flat image to normalize all spectra with the mean spectrum on the dispersion axis.

At first, use p3d_cflatf on the twilight flat-field images. Thereafter, use p3d_cflatf on the lamp-flat images using the NDISP keyword (or the 'ff_normdisp' user parameter; this is also the default), and remember to specify the transmission-array file from the twilight-flat flat-field creation with the TRANSMISSION keyword (or the user parameter 'ff_trm_file'). Finally, when calling the tool p3d_cobjex on the science images, set the NF2F keyword or the user parameter 'ff_normf2f' parameter.

Following these recommendations, the fiber-to-fiber transmission correction of the extracted science data is at first removed, before the provided fiber-to-fiber transmission correction is applied.

About the normalization procedure of previous revisions

In historical versions of p3d (release <= 1.1.1, revisions < 460), the normalization was made by dividing extracted flat-field spectra with the mean spectrum before the same spectra were replaced with fitted polynomials; spectra of dead and low-transmission fibers were included in the calculation of the mean spectrum. If you want to use this behavior, perhaps because you want to compare the different approaches, set the user parameter 'ff_originalnorm', which automatically disables the other normalization options.

General information on how to learn more about this routine

The normalization procedure is mentioned briefly in the p3d paper: Sandin et al. 2010, A&A, 515, 35. However, it has been significantly modified since the writing of the paper, which is why you are encouraged to learn about the current procedure by reading the documentation here (above).

Instrument parameter files (*.prm), which are required to run this routine, are found in the directory "!p3d_path/data/instruments/*/" (outside of IDL, check "${p3d_path}/data/instruments/*/").

A default (master) user-parameter file is available in "!p3d_path/data/master_userparfile.dat". Copy this file to your data-reduction output directory, rename the file to, for example, "user_p3d.dat" and edit it to suit your needs.

Examples on how to use this tool are provided in the EXAMPLES section below. Also see the tutorials in the directory "!p3d_path/doc/".

See the file README in the root of the p3d package for installation instructions, if p3d is not already installed on the computer.

Additional information, documentation, new versions, etc., can be found at the p3d project web site "https://p3d.sourceforge.io".

Requirements and format of the input and the output data

p3d only works with FITS-formatted images, i.e. fits files. Such files have a header that contains additional information about the observing conditions, the telescope, the instrument setup, and other useful information. Both input and output fits files can be compressed using gzip, to save storage space (also on Windows platforms); p3d handles such compressed files transparently.

This routine takes one, or several, continuum-data raw images as input. The images are combined (by default), but p3d can also calculate a flat-field image for each file if the parameter combineimages_ff is set to "no" (in the user-parameter file). Prescan and overscan regions must not be removed in the input data, they are removed by p3d during the reduction. Input files of instruments using several readout ports must also not be combined in advance, p3d combines the images automatically.

Spectra in data are at first extracted. Thereafter, all elements <= 0.0 are masked, and a normalized transmission-correction array is calculated; with as many elements as there are spectra. Extracted flat-field spectra are then saved in an output file; the data normalization is made through a separate, and subsequent, call to the tool p3d_cobjex. The transmission-correction array is saved in a separate binary-table fits file, with one column and as many rows as there are spectra; p3d_cobjex alwayss uses the transmission-array file recorded in the IMTRM header keyword.

The output file always contains a pre-defined number of extracted spectra (with as many pixels on the cross-dispersion axis). The number of pixels on the dispersion axis is the input size (where the prescan and the overscan regions have been removed). The spectra are ordered as in the input image – this is the so-called row-stacked spectrum (RSS) format. To create a spatial map of the intensities at any wavelength, it is necessary to use a fiber position table. In addition to the standard fits-file header keywords, extracted spectrum images contain additional p3d-specific keywords:

The fiber-to-fiber transmission-array header contains four additional p3d-specific keywords:

The combined image contains the following three p3d-specific header keywords if several input files were specified that were thereafter combined by p3d, before extracting the spectra:

Saved, and optionally already combined, bias-subtracted raw-data images contain only one additional header keyword:

Input parameters:

	filename		An array of strings that specifies the names of the raw flat-field data files used when creating the master flat-field image.
	parfile		A scalar string that specifies the name of a file with instrument-specific setup parameters. These files follow a strict format. Already properly formatted files are available for all configured instruments in the directories p3d_path/data/instruments/*/. It is sufficient to specify the file basename, without the full path of the p3d subdirectory; for example: "nvimos_hr.prm".

Keyword parameters:

	masterbias		A scalar string with the name of a master-bias image file. This file is required if optimal extraction is used and none of the following four keywords is used: BIASPX, BIASPY, BIASOX, BIASOY. Note! This keyword must not be set if the prescan and the overscan regions should be used to estimate the bias level.
	tracemask		A scalar string with the name of a trace-mask image file. A new trace-mask image is calculated using the (combined) input raw-data file if TRACEMASK is not present.
	dispmask		A scalar string with the name of a dispersion-mask image file to use; by p3d_cobjex, when wavelength calibrating the flat-field image data.
	bpmask		A scalar string with the name of a bad-pixels mask file; this could be, for example, a cosmic-ray mask file. Note! This mask is only used if optimal extraction is switched on in the user-parameter file.
	crmask		A scalar string with the name of a cosmic-ray mask image file; this could be, for example, a bad-pixels mask image file. Note! This mask is only used if optimal extraction is switched on in the user-parameter file. Note! CRMASK in not used if CRCLEAN is set.
	biaspx		Set this keyword to use the x-axis prescan region when subtracting a bias. Specifically, a median value is calculated for each pixel on the y axis in the x-axis prescan region. The same array is used with all y-axis columns of bias subtracted data.
	biaspy		Set this keyword to use the y-axis prescan region when subtracting a bias. Specifically, a median value is calculated for each pixel on the x axis in the y-axis prescan region. The same array is used with all x-axis rows of bias subtracted data.
	biasox		Set this keyword to use the x-axis overscan region when subtracting a bias. Specifically, a median value is calculated for each pixel on the y axis in the x-axis overscan region. The same array is used with all y-axis columns of bias subtracted data.
	biasoy		Set this keyword to use the y-axis overscan region when subtracting a bias. Specifically, a median value is calculated for each pixel on the x axis in the y-axis overscan region. The same array is used with all x-axis rows of bias subtracted data.
	biasconstant		This keyword can be used in two ways: Set this keyword to use a constant value as a bias value; otherwise an array is used. All prescan and overscan regions specified using the keywords BIASPX, BIASPY, BIASOX, and BIASOY can be used when this keyword is set; otherwise, only one of them. If this keyword is set to a double type array or to a comma-separated string with values greater than one (when called from the shell), these values are used to create a bias. The number of elements must be the same as the number of blocks or extensions.
	savebiassub		Set this keyword to write the bias-subtracted raw-data image before the spectrum extraction. The name of the output file is the same as the (combined) input file, with the added intermediate suffix '_subbias'. Also see the user parameter 'savebiassub'.
	transmission		A scalar string with the filename of a normalized fiber-to-fiber transmission array. The file is actually not used by this tool, but the filename is written to the header of the output flat-field image as the IMTRM keyword, and that keyword is later used by the tool p3d_cobjex to select the transmission-array file when NF2F is set. Note! A transmission file can also be specified in the user-parameter file using the parameter 'ff_trm_file'.
	pcutlow		A scalar floating-point value that specifies the extent of the total number of pixels on the dispersion axis removed from each spectrum at the lower (blue) end before calculating a fiber-to-fiber transmission array. This value is specified in per cent. Also see the user parameter 'ff_lowerpx'. The default value is: 5.0
	pcuthigh		A scalar floating-point value that specifies the extent of the total number of pixels on the dispersion axis removed from each spectrum at the upper (red) end before calculating a fiber-to-fiber transmission array. This value is specified in per cent. Also see the user parameter 'ff_upperpx'. The default value is: 5.0
	userparfile		A scalar string with the name of a file with user-defined parameters. The following parameters are read here from this file: 'keywordsfile' The name of the file with instrument-specific header keywords. 'combineimages_ff' To specify if input images are to be combined or not. 'compress' Corresponds to COMPRESS. 'opfx' Corresponds to OPFX. 'sfx' Corresponds to SFX. 'icsfx' To specify the data output file intermediate suffix of combined files. 'ffsfx' To specify the data output file intermediate suffix of the extracted spectrum image. 'mosfx' To specify the data output file intermediate suffix of the created bad pixels-mask image. 'mmsfx' To specify the data output file intermediate suffix of the created saturated pixels-mask image. 'trmsfx' To specify the data output file intermediate suffix of the created fiber-to-fiber transmission binary-table file. 'scattlightsubtract' Corresponds to SCATTLIGHTSUBTRACT. 'sls_writefile' To write an image with the scattered-light component. 'methimcombine_ff' To specify the method used to combine input images. 'allinone' Corresponds to ALLINONE. 'nthreads_cr' Corresponds to CRNTHREADS. 'nthreads' Corresponds to NTHREADS. 'noobcheck' Corresponds to NOOBCHECK. 'crclean' Corresponds to CRCLEAN. 'savebiassub' Corresponds to SAVEBIASSUB. 'biaspx' Corresponds to BIASPX. 'biaspy' Corresponds to BIASPY. 'biasox' Corresponds to BIASOX. 'biasoy' Corresponds to BIASOY. 'biasconstant' Corresponds to BIASCONSTANT. 'nocrmask' Corresponds to NOCRMASK. 'writesatmask' Corresponds to SATMASK. 'satlimit' To define the limit of saturated values. 'exmonitor' Corresponds to EXMONITOR. 'a2' To use A3 instead of A4. 'a3' To use A3 instead of A4. 'a5' To use A3 instead of A4. 'letter' To use US Letter instead of A4 with PostScript and PDF output. 'legal' To use US Letter instead of A4 with PostScript and PDF output. 'tabloid' To use US Tabloid instead of A4 with PostScript and PDF output. 'psusletter' To use US Letter instead of A4 with PostScript and PDF output. 'smowidth_ff' To smooth the flat-field image, see the master user-parameter file. 'deg_polyfit_ff' To replace the flat-field image with a polynomial fit, see the master user-parameter file. 'deadfibersfile' To specify a dead-fibers file. 'ff_originalnorm' see the master user-parameter file. 'ff_normdisp' see the master user-parameter file. 'ff_lowerpx' see the master user-parameter file. 'ff_upperpx' see the master user-parameter file. 'ff_trm_file' Corresponds to TRANSMISSION. 'methodextract' see p3d_misc_extract_method. Additionally, the following parameters are used: 'laimagemethod' Corresponds to IMAGEMETHOD. 'laimageclean' Corresponds to IMAGECLEAN. 'ladispmedian' Corresponds to DISPMEDIAN. 'crwriteall' Corresponds to WRITEALL. 'nocrc' Corresponds to NOCRC. 'sigclip' Corresponds to SIGCLIP. 'objlim' Corresponds to OBJLIM. 'ratlim' Corresponds to RATLIM. 'crfwhm' Corresponds to FWHM. 'crgausskernelsize' Corresponds to GAUSSKERNELSIZE. 'fwhm_tr' Used to set the width of the Gaussian profile when crfwhm is unavailable and IMAGEMETHOD is unset. 'sigfrac' Corresponds to SIGFRAC. 'growradius' Corresponds to GROWRADIUS. 'maxiter' Corresponds to MAXITER. 'cleansfx' Corresponds to CLEANSFX. 'masksfx' Corresponds to MASKSFX. 'laplsfx' Corresponds to LAPLSFX. 'poflsfx' Corresponds to POFLSFX. 'siprsfx' Corresponds to SIPRSFX. 'laprsfx' Corresponds to LAPRSFX. 'finesfx' Corresponds to FINESFX. 'wallsfx' Corresponds to WALLSFX. Note! If any of the corresponding keywords is specified, that value takes precedence.
	ofilename	X	This keyword returns the full name of the created master flat-field image file.
	opath		A scalar string that specifies the path where the output data are saved. The default value is: '.'
	opfx		A scalar string that specifies a prefix for the (automatically generated) output filename. The default value is: ''
	sfx		A scalar string that specifies the raw data filename suffix to use. The default value is: .fits
	detector		A scalar integer that specifies the currently selected detector; DETECTOR is a zero-based value. The default value is: 0
	exmonitor		If this keyword is set, a line-profiles viewer is shown after all fitting is done when using optimal extraction. The default value is: 1
	compress		Set this keyword to compress all output files using gzip; the file suffix '.gz' is appended to all output filenames. The default value is: 0
	logfile		This keyword specifies the name of an optional log file when p3d_cflatf is launched as a separate program.
	loglevel		If this routine is launched as a separate program, and LOGFILE is used, this keyword specifies the log level; [1], 2, 3 (the higher the value, the more output). The default value is: 1
	scattlightsubtract		Set this keyword to first calculate and thereafter subtract a scattered-light image. Such an image is calculated by smoothing and interpolating the raw-data image, using regions on the CCD that lie far enough from nearby spectra; considering only the cross-dispersion axis. The scattered-light image is subtracted from the masterbias-subtracted raw image before (calculating the error image and) spectra are extracted. Note! This procedure only works with data where there is some "empty" space left near the edges, and preferrably also between spectrum traces on the CCD, on the cross-dispersion axis, which can be used to measure the extent of scattered light.
	detsepgrp		See p3d_ctrace.
	detsepnum		See p3d_ctrace.
	cinv		Set this keyword to plot white lines on a black background instead of the default, which is to plot black lines on a white background.
	crnthreads		A scalar integer that specifies how many threads used when cleaning single input images of mic-ray hits. If this keyword is not set, READS is used instead. If nthreads was not set her, the default is to use the maximum number of ilable threads minus one, but the maximum number 16: (!cpu.hw_ncpu – 1) < 16. The default value is: max-1
	nthreads		A scalar integer that specifies how many threads to use in the parallelized line profile calculations. The calculation speed scales nearly linearly with the number of threads. The default is to use the maximum number available, but no more than 16: max = !cpu.hw_ncpu < 16. The default value is: max
	satmask		Set this keyword to write saturated-pixels mask images to the disk. Such images indicate if any saturated pixels in the input images were used when calculating a spectrum image.
	allinone		Unset this keyword to write the output data to separate files, instead of writing the extracted spectra to the first (DATA) extension, its error to the ERROR extension, the bad-pixels mask image to the MASK extension, the saturated-pixels mask image to the SATMASK extension, the data-quality mask image to the QUALITY extension, and the fiber-to-fiber binary-table transmission array to the TRANSMSN extension. The default value is: 1
	noobcheck		Set this keyword to avoid checking the OB header keyword; this only applies to the instruments that use it. To see if it applies to your instrument, check if the OBID parameter is defined in the instrument keywords file.
	crclean		Set this keyword to clean all uncombined input images of cosmic-ray hits before the extraction procedure is started; either a master-bias image or prescan and overscan regions must be used to provide the bias level to use this functionality.
	nocrmask		Set this keyword to avoid using the cosmic-ray hit mask image, which is created when CRCLEAN is set, when extracting the spectra optimally. Normally, it is recommended to keep this keyword unset; it is available to allow the user the ultimate choice. The following keywords are used to configure the cosmic-ray cleaning procedure; they are only used if CRCLEAN is set; see p3d_cr and p3d_tool_crays_lapycosmic for more details:
	sigclip		A scalar decimal value that specifies the clipping sigma.
	objlim		A scalar decimal value that specifies the object limit value; this value is only used if IMAGEMETHOD is set.
	ratlim		A scalar decimal value that specifies the ratio limit value; this value is only used if IMAGEMETHOD is unset.
	crfwhm		A decimal scalar value, or a two-element array, that specifies the full width at half maximum of the gaussian kernel used to convolve the raw data. This keyword corresponds to FWHM in p3d_cr. This value is only used if IMAGEMETHOD is unset.
	gausskernelsize		A scalar integer that specifies the size of the Gaussian kernel used when convolving the input image with a Gaussian kernel using the PyCosmic algorithm. This value is by default set to 2 * q + 1, where q is the maximum value of the x and the y pixel where a one-dimensional Gaussian reaches 15% of its maximum value (1.0); this condition makes use of the full-width-at-half-maximum value of FWHM. The value must be odd, and the maximum value is in any case 19. Be aware that convolution times increase strongly with the size of the Gaussian kernel.
	sigfrac		A scalar decimal value that specifies a scale factor for pixels surrounding a cosmic-ray masked pixel; this parameter is only used if GROWRADIUS is non-zero. The default value is: 0.5
	growradius		Set this keyword to grow all cosmic-ray hit pixels by a frame that is this many pixels wide; this is done to check if the surrounding pixels are affected by cosmic-ray hits as well. The default value is 1, the values 0 and 2 are also accepted. Surrounding pixels are checked if they are cosmic-ray affected using the first criterion S' > SIGFRAC * SIGCLIP. The default value is: 1
	maxiter		A scalar integer value that specifies the maximum number of iterations used when identifying cosmic-ray hits in each image.
	imagemethod		Set this keyword to use the original algorithm of P. van Dokkum (L.A. Cosmic), which was developed for cosmic-ray detection in images. The default is otherwise to use the IFS-modified algorithm with IMAGEMETHOD unset.
	imageclean		Set this keyword to use the original two-dimensional median filter when replacing the flux in masked pixels with that of the surrounding pixels. Otherwise an [11 pixel] one-dimensional median filter is used. The default value is: 1
	dispmedian		Set this keyword to a positive integer > 1 to subtract sampling flux along the dispersion axis. The value defines the width of the median window.
	writeall		Set this keyword to write all intermediate images in addition to the cleaned image and the cosmic-ray mask.
	showcrgui		Set this keyword to show a properties GUI of the cosmic-ray cleaning parameters before the algorithm is executed.
	nocrc		Set this keyword to force the use of the slower IDL-based cosmic-ray cleaning routine instead of the faster C-based routine.
	fnotify		Set this keyword to the name of a file that is written when the tool has finished; this can be a useful feature when using the p3d queue handler. In that case, set this keyword and check if the file exists before continuing. And do not forget to remove the file.
	font		Set this scalar string to the name of the font to be used with all widget components of this tool. It is advised to use a fixed font, the outcome is otherwise difficult to predict. Set the keyword to 'big' to use the following pre-defined font ed-medium-r-normal--18-120-100-100-c-90-iso8859-1'; this could be useful on a system using a 4k screen (with 3160x2560 pixels). Another option to specify a font is to set the environment variable p3d_font or the p3d IDL system variable !p3d_font. The default font used with the widgets is, for your information, printed to the screen when FONT is unset and the highest verbosity level is used (verbose = 3). On Linux machines, use a shell command such as 'xlsfonts' to list all available fonts.
	cmdline	X	This keyword is used then the tool is launched by the p3d server.
	gui	X	If this keyword is set, the routine is assumed to be called from the p3d GUI – meaning that there is no initialization of the p3d system variables.
	stawid	X	If set, various messages are written to the p3d GUI status line (this must be the widget id of that label widget).
	topwid	X	If set, error messages are displayed using DIALOG_MESSAGE, using this widget id as DIALOG_PARENT, instead of MESSAGE.
	logunit	X	Messages are saved to the file pointed to by this logical file unit, if it is defined.
	verbose		Set this parameter to a positive scalar integer to make p3d write some information on STDOUT about what is going on. The following four values are acknowledged: 0 Writes no information at all. This is the default. 1 Writes the more important information; regarding subroutine configurations, mostly. 2 Writes most information; includes a more verbose output than 1. 3 Writes all information, including information on the execution state of GUI subroutines. This may be a useful mode when debugging the code.
	quiet		Set this keyword to not print the disclaimer on the screen when this tool is started from the command line or from the shell.
	error	X	This scalar integer returns an error code if set. Any value different from zero means that an error has occurred.
	debug	X	An error handler is not setup if this keyword is set. Otherwise, the error handler is setup using the routine CATCH, which makes each subroutine exit quietly when a bug is encountered. Use this keyword when debugging p3d.
	cdebug		A keyword as DEBUG used with the called C routines; valid values are 0-6 where a higher number results in more output to STDERR.
	help	X	Set this keyword to show this routine documentation, and then exit.

Output parameters:

out

X

Upon a successful execution this variable contains the flat-field (data) when the routine exits.

Examples:

This routine can be called in four different ways (for use on WINDOWS platforms, replace the path character '/' with '\'):

i) Using the p3d GUI

There is no need to setup the input parameters, p3d handles this transparently.

ii) As a stand-alone tool from the IDL command line

In addition to your input flat-field images, it is necessary to pass the name of the (provided) instrument-specific parameter file, a master-bias image file or (at least) one of the bias prescan and overscan parameters (if you use optimal extraction), a trace-mask image file, and a dispersion-mask image file (for use when object data are normalized), as well as any additional arguments to p3d:

files = ['flatfile_1.fits.gz', 'flatfile_2.fits.gz']
parfile = 'larr2k.prm'
opath = 'Output'
masterbias = opath + '/masterbias.fits.gz'
tracemask  = opath + '/tracemask.fits.gz'
transmission = opath + '/transmission.fits.gz'
userparfile = opath + '/user_p3d.dat' ;; If you use this file
detector = 0 ;; Required with multi-detector configurations.
p3d_cflatf, files, parfile, masterbias=masterbias, $
    tracemask=tracemask, transmission=transmission, $
    userparfile=userparfile, opath=opath, detector=detector, $
    logfile=opath + 'dred.log', /verbose

iii) As a stand-alone tool from a terminal, using the IDL VM

In addition to your input flat-field images, it is necessary to pass the name of the (provided) instrument-specific parameter file, a master-bias image file or (at least) one of the bias prescan and overscan parameters (if you use optimal extraction), a trace-mask image file, and a dispersion-mask image file, as well as any additional arguments to p3d. The recommended approach is to use the provided shell script p3d_cflatf_vm.sh or the Python tool p3d_dispatch.py:

# Using the shell script, making use of the p3d queue system:
parfile=larr2k.prm
opath=Output
masterbias=${opath}/masterbias.fits.gz
tracemask=${opath}/tracemask.fits.gz
transmission=${opath}/transmission.fits.gz
userparfile=${opath}/user_p3d.dat # If you use this file
detector=0 # Required with multi-detector configurations.
p3d_cflatf_vm.sh \
    flatfile_1.fits.gz,flatfile_2.fits.gz,flatfile_3.fits.gz \
    $parfile masterbias=$masterbias tracemask=$tracemask \
    transmission=$transmission userparfile=$userparfile \
    opath=$opath detector=$detector logfile=$opath/dred.log /verbose

# Using the Python tool, without using the p3d queue system:
parfile=larr2k.prm
opath=Output
masterbias=${opath}/masterbias.fits.gz
tracemask=${opath}/tracemask.fits.gz
transmission=${opath}/transmission.fits.gz
userparfile=${opath}/user_p3d.dat # If you use this file
detector=0 # Required with multi-detector configurations.
p3d_dispatch.py --noqueue cflatf \
    flatfile_1.fits.gz,flatfile_2.fits.gz,flatfile_3.fits.gz \
    $parfile masterbias=$masterbias tracemask=$tracemask \
    transmission=$transmission userparfile=$userparfile \
    opath=$opath detector=$detector logfile=$opath/dred.log /verbose

Note: In this approach, it is necessary to specify all options using the full names of the keywords. Partially correct keyword names are not permitted, as they are inside IDL. Hence, as an example, you must type "p3d_cflatf_vm.sh tracemask=$tracemask" and -NOT- "p3d_cflatf_vm.sh tr=$tracemask".

Note: The tool launches in the current directory, which is why it is unnecessary to specify full paths if files reside in the current directory.

Note: When several files are specified in the first argument, they must be separated with either a comma or a semicolon.

Note: Parameters indicated with an 'X' in the list above are unavailable in this approach. This includes the two output variable parameters OFILENAME, and OUT.

iv) As a stand-alone tool from Python, using the IDL VM

This approach is analogous to iii), but is executed from the Python command line. Here is an example:

# Launch p3d_cflatf in a single IDL session, without using the
# p3d queue system:
from p3d_dispatch import *
files = 'flatfile_1.fits.gz,flatfile_2.fits.gz,flatfile_3.fits.gz'
parfile = 'larr2k.prm'
opath = 'Output'
masterbias = opath + '/masterbias.fits.gz'
tracemask  = opath + '/tracemask.fits.gz'
transmission = opath + '/transmission.fits.gz'
userparfile = opath + '/user_p3d.dat' # If you use this file
detector = 0 # Required with multi-detector configurations.
commandline = files + ' ' + parfile + ' masterbias=' + masterbias +
  ' tracemask=' + tracemask + ' transmission=' + transmission +
  ' userparfile=' + userparfile + ' opath=' + opath +
  ' detector=' + detector +
  ' logfile=' + opath + '/dred.log /verbose'
p3d_dispatch('cflatf', commandline, noqueue=True)

Creates a master-bias image from a set of raw bias images.

To reduce CCD image data properly, it is necessary to subtract the bias, or the zero-level value, of each pixel. The subtraction can be done by using information in the prescan and the overscan regions, or using an entire image. The latter approach is handled by this tool. The former approach is used through the bias?? keywords and user parameters in the five tools: p3d_cr, p3d_ctrace, p3d_cdmask, p3d_cflatf, and p3d_cobjex.

The purpose of this tool is to combine several raw bias images into one master-bias image, where the error in each pixel is smaller than in a single image. The image combination method can be selected between the available options using the "methodimcombine" parameter in the user-parameter file (see the master user-parameter file, and also p3d_misc_imcombine.pro).

While the recommended approach is to select, and use, a large number of bias images, it is possible to select only one file; in this case the only difference between the input bias image and the output master-bias image are a few added header keywords. While it is also not recommended, p3d offers the possibility to smooth a single image (using first an average and thereafter a median box car smoothing with a kernel of 5x5 pixels). All information of individual pixels is lost if this approach is used.

General information on how to learn more about this routine

Instrument parameter files (*.prm), which are required to run this tool, are found in the directory "!p3d_path/data/instruments/*/" (outside IDL check "${p3d_path}/data/instruments/*/").

A default (master) user-parameter file is available in "!p3d_path/data/master_userparfile.dat". Copy this file to your data-reduction output directory, rename the file to, for example, "user_p3d.dat" and edit it to suit your needs.

Examples on how to use this tool are provided in the EXAMPLES section below. Also see the tutorials in the directory "!p3d_path/doc/".

See the file README in the root of the p3d package for installation instructions, if p3d is not already installed on the computer.

Additional information, documentation, new versions, etc., can be found at the p3d project web site "https://p3d.sourceforge.io".

Requirements and format of the input and the output data

p3d only works with FITS-formatted images, i.e. fits files. Such files have a header that contains additional information about the observing conditions, the telescope, the instrument setup, and other useful information. Both input and output fits files can be compressed using gzip, to save storage space (also on Windows platforms); p3d handles such compressed files transparently.

This routine takes one, or several, raw bias images as input. The images are combined by p3d. Prescan and overscan regions must not be removed in the input data, they are removed by p3d during the reduction (when using the other routines). Input files of instruments using several readout ports must also not be combined in advance, p3d combines the images automatically.

In addition to the standard fits-file header keywords the master-bias image contains additional p3d-specific keywords:

Input parameters:

	filename		An array of strings that specifies the names of the raw bias image files, which are used when creating the master-bias image.
	parfile		A scalar string that specifies the name of a file with instrument-specific setup parameters. These files follow a strict format. Already properly formatted files are available for all configured instruments in the directories p3d_path/data/instruments/*/. It is sufficient to specify the file basename, without the full path of the p3d subdirectory; for example: "nvimos_hr.prm".

Keyword parameters:

	nodialog		The file dialog that allows a single file to be smoothed is not shown when this keyword is set. Smoothing is always avoided when this keyword is set. It is possible to use the user parameter 'biasnodialog' instead. The default value is: 1
	userparfile		A scalar string with the name of a file with user- defined parameters. The following parameters are read here from this file: 'biasnodialog' corresponds to NODIALOG. 'methodimcombine' see p3d_misc_imcombine.pro. 'detsec' see p3d_misc_detsec.pro.
	ofilename	X	This keyword returns the full name of the written master-bias image file.
	opath		A scalar string that specifies the path, where the output image is saved. The default value is: '.'
	opfx		A scalar string that specifies a prefix for the (automatically generated) output filename. The default value is: ''
	sfx		A scalar string that specifies the raw data filename suffix to use. The default value is: .fits
	detector		A scalar integer that specifies the currently selected detector; DETECTOR is a zero-based value. The default value is: 0
	compress		Set this keyword to compress all output files using gzip; the file suffix '.gz' is then appended to all output filenames. The default value is: 0
	nthreads		A scalar integer that specifies how many threads to use in the parallelized line-profile calculations. The calculation speed scales nearly linearly with the number of threads. The default is to use the maximum number available, but no more than 16: max = (!cpu.hw_ncpu < 16L). The default value is: max
	logfile		This keyword specifies the name of an optional log file when p3d_cmbias is launched as a separate program.
	loglevel		If this routine is launched as a separate program, and LOGFILE is used, this keyword specifies the log level; [1], 2, 3 (the more output, the higher the value). The default value is: 1
	fnotify		Set this keyword to the name of a file that is written when the tool has finished; this can be a useful feature when using the p3d queue handler. In that case, set this keyword and check if the file exists before continuing. And do not forget to remove the file.
	cmdline	X	This keyword is used when the tool is launched by the p3d server.
	gui	X	If this keyword is set, the routine is assumed to be called from the p3d GUI – meaning that there is no initialization of the p3d system variables.
	stawid	X	If set, various messages are written to the p3d GUI status line (this must be the widget id of that label widget).
	topwid	X	Error messages are displayed using DIALOG_MESSAGE instead of MESSAGE if this keyword is set, using this widget id as DIALOG_PARENT.
	logunit	X	Messages are saved to the file pointed to by this logical file unit if it is defined.
	verbose		Set this parameter to a positive scalar integer to make p3d write some information on STDOUT about what is going on. The following four values are acknowledged: 0 Writes no information at all; this is the default. 1 Writes the more important information; regarding subroutine configurations, mostly. 2 Writes most information; includes a more verbose output than 1. 3 Writes all information, including information on the execution state of GUI subroutines; this may be a useful mode when debugging the code.
	quiet		Set this keyword to avoid printing the disclaimer on the screen when this tool is started from the command line or from the shell.
	error	X	This scalar integer returns an error code if set. Any value different from zero means that an error has occurred.
	debug	X	An error handler is not setup when this keyword is set. Otherwise, the error handler is setup using the routine CATCH, which makes each subroutine exit quietly when a bug is encountered. Use this keyword when debugging p3d.
	cdebug		A keyword as DEBUG used with the called C routines; valid values are 0-6 where a higher number results in more output to STDERR.
	help	X	Set this keyword to show this routine documentation and then exit.

Output parameters:

out

X

Upon a successful execution, this variable contains the master-bias image when the routine exits.

Examples:

This routine can be called in four different ways (for use on WINDOWS platforms, replace the path character '/' with '\'):

i) Using the p3d GUI

There is no need to setup the input parameters, p3d handles this transparently.

ii) As a stand-alone tool from the IDL command line

In addition to the bias images, it is necessary to pass the name of the (provided) instrument-specific parameter file as well as any additional arguments to p3d:

files = ['biasfile_1.fits.gz', 'biasfile_2.fits.gz']
parfile = 'larr2k.prm'
opath = 'Output'
userparfile = opath + '/user_p3d.dat' ;; If you use this file
detector = 0 ;; Required with multi-detector configurations.
p3d_cmbias, files, parfile, userparfile=userparfile, opath=opath, $
    detector=detector, logfile=opath + 'dred.log', /verbose

iii) As a stand-alone tool from a terminal, using the IDL VM

In addition to the bias images, it is necessary to pass the name of the (provided) instrument-specific parameter file as well as any additional arguments to p3d. The recommended approach is to use the provided shell script p3d_cmbias_vm.sh or the Python tool p3d_dispatch.py:

# Using the shell script, without using the p3d queue system:
parfile=larr2k.prm
opath=Output
userparfile=${opath}/user_p3d.dat # If you use this file
detector=0 # Required with multi-detector configurations.
p3d_cmbias_vm.sh -s biasfile_1.fits.gz,biasfile_2.fits.gz,... \
    $parfile userparfile=$userparfile opath=$opath \
    detector=$detector logfile=$opath/dred.log /verbose

# Using the Python tool, using the p3d queue system:
parfile=larr2k.prm
opath=Output
userparfile=${opath}/user_p3d.dat # If you use this file
detector=0 # Required with multi-detector configurations.
p3d_dispatch.py cmbias biasfile_1.fits.gz,biasfile_2.fits.gz,... \
    $parfile userparfile=$userparfile opath=$opath \
    detector=$detector logfile=$opath/dred.log /verbose

Note: In this approach, it is necessary to specify all options using the full names of the keywords. Partially correct keyword names are not permitted, as they are inside IDL. Hence, as an example, you must type "p3d_cmbias_vm.sh userparfile=$userparfile" and -NOT- "p3d_cmbias_vm.sh user=$userparfile".

Note: The tool launches in the current directory, which is why it is unnecessary to specify full paths if files reside in the current directory.

Note: When several files are specified in the first argument, they must be separated with either a comma or a semicolon.

Note: Parameters indicated with an 'X' in the list above are unavailable in this approach. This includes the two output variable parameters OFILENAME, and OUT.

iv) As a stand-alone tool from Python, using the IDL VM

This approach is analogous to iii), but is executed from the Python command line. Here is an example:

# Launch p3d_cmbias in a single IDL session, without using the
# p3d queue system:
from p3d_dispatch import *
bias = 'biasfile_1.fits.gz,biasfile_2.fits.gz,biasfile_3.fits.gz'
parfile = 'larr2k.prm'
opath = 'Output'
userparfile = opath + '/user_p3d.dat' # If you use this file
detector = 0 # Required with multi-detector configurations.
commandline = bias + ' ' + parfile + ' userparfile=' + userparfile +
  ' opath=' + opath + ' nthreads=4' + ' detector=' + detector +
  ' logfile=dred.log loglevel=2 /verbose'
p3d_dispatch('cmbias', commandline, noqueue=True)

Creates an image with extracted, and optionally wavelength calibrated and flat-fielded, science-object spectra from a set of input image files.

The purpose of this tool is to extract science-object spectra. This extraction can be made in several different ways, optionally correcting for several different instrument-specific issues. The input images are at first combined, and the spectra are thereafter extracted using the tophat extraction method, or alternatively using an optimal extraction method. A trace-mask image is always required to find and trace the spectra; such an image is created with the tool p3d_ctrace. If optimal extraction is to be performed, it is also necessary to provide a line-profiles image; which is also created by the tool p3d_ctrace. After the spectra are extracted, they are (optionally) wavelength calibrated using a dispersion mask; which was created using p3d_cdmask. The final procedure is to normalize the spectra with an extracted flat-field image; which was created using p3d_cflatf.

The output-data files – consisting of the extracted spectrum image, the extracted error image, the bad-pixels mask image, the saturated-pixels mask image, the data-quality mask image, the normalized flat-field image, and the calculated scattered-light image – are by default written to the same file, in consecutive extensions. They are instead written to separate files if the keyword ALLINONE, or the user-parameter 'allinone', is unset. There is no real advantage to either approach; although, if separate files are used and the error file, the output bad-pixels mask image file, or the saturated-pixels mask image file are renamed, they will not be found by other tools.

The following sections describe the individual steps in more detail.

This tool calls p3d_cr if requested to do so, which cleans single input images of cosmic-ray hits; see the keyword /CRCLEAN. The outcome of the cleaning procedure depends both on properties of the input image as well as the algorithm parameters. Different parameters might produce a different outcome. There is no guarantee whatsoever that the cleaning removes only cosmic-ray hits. Check the output images to build your an opinion regarding the quality of the outcome.

Spectrum extraction

The spectrum-extraction method is selected using the 'methodextract' parameter in the user-parameter file; available options are 'tophat' and 'optimal'. The method of the optimal extraction is selected with the user parameter 'optexmethod'. A simple setup only requires these two parameters to be properly considered. Also, if you want to study the outcome, ensure that the keyword EXMONITOR is set (alternatively set the user parameter 'exmonitor'). With most instruments, there are, however, additional issues that might be necessary to consider, see the following two subsections.

Instrument flexure

Many instruments are affected by flexure, meaning that spectra move around on the detector, both during the exposure and in the time between science exposures and calibration exposures. The shift is often smaller than one pixel, but may still influence the outcome significantly (see, e.g., Sandin et al. 2010). Furthermore, the shift occurs both on the dispersion axis and on the cross-dispersion (spatial) axis. p3d allows a correction for shifts that occur between calibration and the science exposures, before spectra are extracted.

For shifts that occur on the cross-dispersion axis, see the keywords below related to, a so-called, recentering (RECENTERVAL and RECENTERLIMVAL) and the recentering section in the master user-parameter file. Also see the p3d documentation WIKI for a more thorough discussion with examples.

For shifts that occur on the dispersion axis, see the SKYALIGN, ONESKYOFFSET, and MAXSKYOFFSET keywords below, and also the sky-align section in the master user-parameter file. Also see the p3d documentation WIKI for a more thorough discussion with examples.

As an example, set the SKYALIGN keyword to either the name of a plain-text file listing the wavelengths of (accurately determined) telluric emission lines, or to an array of corresponding wavelengths. All telluric lines are fitted for all (working) spectra. The wavelength arrays of the dispersion mask are thereafter shifted with a weighted offset of the respective line fit from the pre-defined rest wavelength, see the routine p3d_wavecal_dispersion_correction for more details.

Laser offset, or beam-switched, data

An instrument may provide the option to place a second set of spectra offset by an integer number of pixels from the first (see the instrument keyword BS_NROWS). There are two options with such raw data: the offset spectra are either reduced separately from the nominal spectra and the results are saved to the extensions BSW_DATA and BSW_ERR or DATA and ERROR, respectively. Alternatively, set the user parameter 'bsw_px_subtraction' to have the offset spectra be subtracted from the nominal spectra already in the extraction stage.

Using the latter approach, the center pixels – defined by the user parameter 'bsw_px_center_width' – of each offset spectrum and pixel on the dispersion axis are subtracted from each nominal spectrum before the bias is subtracted. The contribution to the flux from the remaining parts of the offset-to-nominal cross-dispersion profile difference is added up after a usual extraction. The value of this parameter that is used in the extractionb is stored in the output file using the keyword BS_DPXSB.

The fraction of the light in the offset spectra that is subtracted from the nominal spectra can be adjusted with the user parameter 'bsw_px_multfactor'. The value can be set in the range 0 < bsw_px_multfactor < 2 and the default value is 1. The value used in the extraction is stored in the output file using the keyword BS_DPXMF.

Scattered light on the detector

Some instruments are more affected than others by scattered light on the detector; this scattered light is not the same as scattered light from the primary and secondary mirrors or atmospheric scattered light. The spectrograph scattered light can be estimated and removed before spectra are extracted – if there is enough "empty" space between the spectra on the detector. To activate the scattered-light subtraction, see the SCATTLIGHTSUBTRACT keyword below as well as the corresponding section in the master user-parameter file. Also see the p3d documentation WIKI for a more thorough discussion with examples.

About the wavelength-calibration procedure

The wavelength calibration is performed using a dispersion-mask image provided with the DISPMASK keyword. For each spectrum, the n + 1 parameters of the dispersion-mask file are used to calculate an n:th order polynomial (wavelength as function of pixel). All spectra are thereafter interpolated to use the same starting wavelength (crval) and pitch (cdelt). The default behavior is to use the minimum number of pixels that contains data for all spectra (npix). The wavelength l for pixel i is then: l = crval + cdelt * (i – crpix), where crpix is always set to 1 in p3d. Set the keywords CRVAL, CDELT, and NPIX to use values other than the default values, see below. Alternatively, set the corresponding lower-case strings in the master user-parameter file ('crval', 'cdelt', and 'npix'). The values can also be set interactively using the WAVEPROMPT keyword.

For multi-detector instruments, such as VIMOS, you most likely want to use the same wavelength array for all detectors. Default grism-dependent values are provided in the four instrument-parameter files of VIMOS.

For instruments where the first (x) axis is the dispersion axis, the following header keywords are used: crval1, cdelt1, crpix1, and ctype1. The header keywords will instead be: crval2, cdelt2, crpix2, and ctype2 if the second axis (y) is the dispersion axis. The keyword crpix is not queried; it is set to 1 to ensure that it is not used. The same keywords on the cross-dispersion axis are all removed from the header if they are defined; the following World-Coordinate-System keywords are also removed if they are available in the input data header: cunit, cd1_1, cd1_2, cd2_1, and cd2_2.

p3d performs an additional wavelength-related calibration if the correction due to flexure on the dispersion axis is used, see the "Instrument flexure" subsection above.

About the flat-fielding procedure

The normalization is performed using an extracted flat-field image provided with the FLATFIELD keyword. A normalized flat-field is created in two steps: at first spectra are extracted using the tool p3d_cflatf, thereafter the extracted flat-field image is normalized by this tool, just before it is used to normalize the extracted science spectra.

After the flat-field image has been normalized, it is wavelength calibrated using the same dispersion mask as the science-object data. This behavior can be changed if a separate dispersion mask was selected when running the tool p3d_cflatf.

Please see p3d_cflatf for all details on how to setup the flat-field procedure. That tool also describes how to use both a lamp flat-field and a twilight flat-field image.

Note: To achieve a consistent flat-fielding procedure, ensure that you use the same parameters – i.e. user-parameter file – when running p3d_cflatf and p3d_cobjex.

General information on how to learn more about this routine

The spectrum extraction procedure is described in excruciating detail in the p3d paper: Sandin et al. (2010), A&A, 515, 35. The flexure-correction procedures as well as the scattered-light subtraction are described in the SPIE paper Sandin et al. (2012), SPIE, 8451, 84510F, and also see the p3d documentation WIKI (see below).

Instrument parameter files (*.prm), which are required to run this routine, are found in the directory "!p3d_path/data/instruments/*/" (outside of IDL, check "${p3d_path}/data/instruments/*/").

A default (master) user-parameter file is available in "!p3d_path/data/master_userparfile.dat". Copy this file to your data-reduction output directory, rename the file to, for example, "user_p3d.dat" and edit it to suit your needs.

Examples on how to use this tool are provided in the EXAMPLES section below. Also see the tutorials in the directory "!p3d_path/doc/".

See the file README in the root of the p3d package for installation instructions, if p3d is not already installed on the computer.

Additional information, documentation, new versions, etc., can be found at the p3d project web site "https://p3d.sourceforge.io".

Requirements and format of the input and the output data

p3d only works with FITS-formatted images, i.e. fits files. Such files have a header that contains additional information about the observing conditions, the telescope, the instrument setup, and other useful information. Both input and output fits files can be compressed using gzip to save storage space (also on Windows platforms); p3d handles such compressed files transparently.

This tool takes one or several raw science-object images as input. The images are combined (by default), but p3d can also extract spectra for each file if the parameter 'combineimages_ob' is set to "no" (in the user-parameter file). Prescan and overscan regions must not be removed in the input data, they are removed by p3d during the reduction. Input files of instruments using several readout ports must also not be combined in advance, p3d combines the images automatically.

The output file always contains the pre-defined number of extracted spectra (with as many pixels on the cross-dispersion axis), which are optionally wavelength calibrated and normalized with a flat-field image. The number of pixels on the dispersion axis is the input size (where the prescan and the overscan regions have been removed). The spectra are ordered as in the input image – this is the so-called row-stacked spectrum (RSS) format. To create a spatial map of the intensities at any wavelength, using this RSS file, it is necessary to make use of a fiber position-table file, as the tool p3d_sv does. Please, use the fiber position-table file defined in the respective instrument-specific parameter file. Extracted data can also be saved using the Euro3D format (use the keyword SAVEE3D or the user parameter 'savee3d'), and data of instruments using square-shaped spatial elements are easily converted to cube format using the tool p3d_rss2cube. In addition to the standard fits-file header keywords, extracted spectrum images contain additional p3d-specific keywords:

The following two keywords are set if a fiber-flat field image is used and the NF2F keyword is set to normalize the fiber-flat field fiber-to-fiber:

The combined image contains the following three p3d-specific header keywords if several input files were specified that were thereafter combined by p3d, before extracting the spectra:

Saved, and optionally already combined, bias-subtracted raw-data images contain only one additional header keyword:

Input parameters:

	filename		An array of strings that specifies the names of the raw-data image files used when creating the extracted science object file.
	parfile		A scalar string that specifies the name of a file with instrument-specific setup parameters. These files follow a strict format. Already properly formatted files are available for all configured instruments in the directories p3d_path/data/instruments/*/. It is sufficient to specify the file basename, without the full path of the p3d subdirectory; for example: "nvimos_hr.prm".

Keyword parameters:

	group		An optional integer array with as many elements as the FILENAME argument. The elements of FILENAME with the same group number are combined if this keyword is set. Note! This keyword can only be used when p3d_cobjex is called as a stand-alone tool, i.e., it is inaccessible using the p3d GUI.
	masterbias		A scalar string with the name of a master-bias image file. This file is required if optimal extraction is used and none of the following four keywords is used: BIASPX, BIASPY, BIASOX, BIASOY. Note! This keyword must not be set if the prescan and the overscan regions should be used.
	tracemask		A scalar string with the name of a trace-mask image file.
	dispmask		A scalar string with the name of a dispersion-mask image file. Note! The flat-field image header must contain the entry IMDMK if it is to be calibrated with a separate dispersion-mask image. The filename stored in IMDMK is then used when applying the dispersion correction to the fiber-flat image.
	flatfield		A scalar string with the name of an extracted flat-field image.
	bpmask		A scalar string with the name of a bad-pixels mask file; this could be, for example, a cosmic-ray mask file. Note! This mask is only used if optimal extraction is switched on in the user-parameter file.
	crmask		A scalar string with the name of a cosmic-ray mask image file; this could be, for example, a bad-pixels mask image file. Note! This mask is only used if optimal extraction is switched on in the user-parameter file. Note! CRMASK in not used if CRCLEAN is set.
	biaspx		Set this keyword to use the x-axis prescan region when subtracting a bias. Specifically, a median value is calculated for each pixel on the y axis in the x-axis prescan region. The same array is used with all y-axis columns of bias subtracted data.
	biaspy		Set this keyword to use the y-axis prescan region when subtracting a bias. Specifically, a median value is calculated for each pixel on the x axis in the y-axis prescan region. The same array is used with all x-axis rows of bias subtracted data.
	biasox		Set this keyword to use the x-axis overscan region when subtracting a bias. Specifically, a median value is calculated for each pixel on the y axis in the x-axis overscan region. The same array is used with all y-axis columns of bias subtracted data.
	biasoy		Set this keyword to use the y-axis overscan region when subtracting a bias. Specifically, a median value is calculated for each pixel on the x axis in the y-axis overscan region. The same array is used with all x-axis rows of bias subtracted data.
	biasconstant		This keyword can be used in two ways: Set this keyword to use a constant value as a bias value; otherwise an array is used. All prescan and overscan regions specified using the keywords BIASPX, BIASPY, BIASOX, and BIASOY can be used when this keyword is set; otherwise, only one of them. If this keyword is set to a double type array or to a comma-separated string with values greater than one (when called from the shell), these values are used to create a bias. The number of elements must be the same as the number of blocks or extensions.
	savebiassub		Set this keyword to write the bias-subtracted raw-data image before spectrum extraction. The name of the output file is the same as the (combined) input file, with the added intermediate suffix '_subbias'. Also see the user parameter 'savebiassub'.
	waveprompt		This keyword determines if the the user should be queried for changing CRVAL, CDELT, and NPIX using an interactive interface (they can all be set with the user-parameter file). The default value is: 0
	drizzle		Unset this keyword to use a linear interpolation instead of the default one-dimensional drizzling algorithm when resampling data from pixels to wavelengths (p3d_wavecal_dispersion_correction). The default value is: 1
	resample_startpx		A scalar value that specifies the starting pixel in the calculation of an output wavelength array when resampling pixel values to wavelength values. The keyword can be set to 'first', 'middle' [default], or 'last', to use of the first, the middle, or the last output pixel, respectively. Alternatively, the keyword can be set to an integer within the same range. Negative values, and values higher than the maximum number of pixels, short of one, are truncated. The default value is: 'middle'
	originalerrors		Set this keyword to use the original, less correct, version of the error calculation applied during the resampling (with the linear interpolation scheme).
	skyalign		A scalar string that contains the name of a line-list file with sky-emission lines, or an array of floating-point values with the wavelengths of sky-emission lines (the unit is Angstrom [Å]); this keyword is used when re-aligning the wavelength array in the dispersion-mask image due to effects of flexure (p3d_wavecal_dispersion_correction).
	oneskyoffset		Set this keyword to use one median offset value on all wavelength arrays in the dispersion mask. Otherwise, each wavelength array in the dispersion mask is offset individually. Additionally, when this parameter is unset, the offset at each wavelength bin is weighted with the wavelength extent of the pixel relative to the corresponding extent at the selected sky-emission line (since uncalibrated wavelength arrays generally deviate from a linear function).
	maxskyoffset		A scalar floating-point value that specifies the imum allowed offset of sky-emission lines, from wavelength provided in the dispersion-mask ge; the unit is Angstrom [Å]. See _wavecal_dispersion_correction for more details. The default value is: 2.0
	savee3d		The output data are saved in the Euro3D (e3d) format in addition to the usual RSS format when this keyword is set.
	sbsw		This keyword returns a scalar integer, which non-zero value specifies the unbinned integer separation between interleaved sets of spectra.
	userparfile		A scalar string with the name of a file with user-defined parameters. The following parameters are read here from this file: 'keywordsfile' The name of the file with instrument-specific header keywords. 'combineimages_ob' To specify if input images are to be combined or not. 'compress' Corresponds to COMPRESS. 'savee3d' Corresponds to SAVEE3D. 'a2' To use A3 instead of A4. 'a3' To use A3 instead of A4. 'a5' To use A3 instead of A4. 'letter' To use US Letter instead of A4 with PostScript and PDF output. 'legal' To use US Letter instead of A4 with PostScript and PDF output. 'tabloid' To use US Tabloid instead of A4 with PostScript and PDF output. 'psusletter' To use US Letter instead of A4 with PostScript and PDF output. 'opfx' Corresponds to OPFX. 'sfx' Corresponds to SFX. 'icsfx' To specify the data output file intermediate suffix of combined files. 'obsfx' To specify the data output file intermediate suffix of the extracted spectrum image. 'mosfx' To specify the data output file intermediate suffix of the created bad pixels-mask image. 'mmsfx' To specify the data output file intermediate suffix of the created saturated pixels-mask image. 'scattlightsubtract' Corresponds to SCATTLIGHTSUBTRACT. 'sls_writefile' To write an image with the scattered-light component. 'methimcombine_ob' To specify the method that is used to combine input images. 'allinone' Corresponds to ALLINONE. 'writesatmask' Corresponds to SATMASK. 'ff_save_final_image' Corresponds to SAVEFINALFLAT. 'satlimit' To define the limit of saturated values. 'nthreads_cr' Corresponds to CRNTHREADS. 'nthreads' Corresponds to NTHREADS. 'drizzle' Corresponds to DRIZZLE. 'resample_startpx' Corresponds to RESAMPLE_STARTPX. 'originalerrors' Corresponds to ORIGINALERRORS. 'spnum' Number of spectra. 'norders' Number of spectra, altern. 'deadfibersfile' To specify a dead-fibers file. 'crclean' Corresponds to CRCLEAN. 'savebiassub' Corresponds to SAVEBIASSUB. 'biaspx' Corresponds to BIASPX. 'biaspy' Corresponds to BIASPY. 'biasox' Corresponds to BIASOX. 'biasoy' Corresponds to BIASOY. 'biasconstant' Corresponds to BIASCONSTANT. 'nocrmask' Corresponds to NOCRMASK. 'skyalign' Corresponds to SKYALIGN. 'oneskyoffset' Corresponds to ONESKYOFFSET. 'maxskyoffset' Corresponds to MAXSKYOFFSET. 'noobcheck' Corresponds to NOOBCHECK. 'exmonitor' Corresponds to EXMONITOR. 'recenterprofiles' see RECENTERVAL. 'gapfile' To read the spectrum gaps file; used when recentering profiles. 'methodextract' see p3d_misc_extract_method. 'smowidth_ff' To smooth the flat-field image, see the master user-parameter file. 'deg_polyfit_ff' To replace the flat-field image with a polynomial fit, see the master user-parameter file. 'ff_originalnorm' see the master user-parameter file. 'ff_normf2f' To use a fiber-to-fiber transmission array, see the master user-parameter file. 'ff_normdisp' see the master user-parameter file. 'ff_lowerpx' see the master user-parameter file. 'ff_upperpx' see the master user-parameter file. 'crval' To set the wavelength of the bluemost pixel. Can also be suffixed with an underscore followed by the grating id keyword (see the parameter GRAT_ID in the instrument-keywords file). 'cdelt' To set the pitch. Can also be suffixed with an underscore followed by the grating id keyword (see the parameter GRAT_ID in the instrument-keywords file). 'npix' To set the number of pixels on the dispersion axis in the spectrum. Can also be suffixed with an underscore followed by the grating id keyword (see the parameter GRAT_ID in the instrument-keywords file). Additionally, the following parameters are used: 'laimagemethod' Corresponds to IMAGEMETHOD. 'laimageclean' Corresponds to IMAGECLEAN. 'ladispmedian' Corresponds to DISPMEDIAN. 'crwriteall' Corresponds to WRITEALL. 'nocrc' Corresponds to NOCRC. 'sigclip' Corresponds to SIGCLIP. 'objlim' Corresponds to OBJLIM. 'ratlim' Corresponds to RATLIM. 'crfwhm' Corresponds to FWHM. 'crgausskernelsize' Corresponds to GAUSSKERNELSIZE. 'fwhm_tr' Used to set the width of the Gaussian profile when crfwhm is unavailable and IMAGEMETHOD is unset. 'sigfrac' Corresponds to SIGFRAC. 'growradius' Corresponds to GROWRADIUS. 'maxiter' Corresponds to MAXITER. 'cleansfx' Corresponds to CLEANSFX. 'masksfx' Corresponds to MASKSFX. 'laplsfx' Corresponds to LAPLSFX. 'poflsfx' Corresponds to POFLSFX. 'siprsfx' Corresponds to SIPRSFX. 'laprsfx' Corresponds to LAPRSFX. 'finesfx' Corresponds to FINESFX. 'wallsfx' Corresponds to WALLSFX. Note! If any of the corresponding keywords is specified, that value takes precedence.
	ofilename	X	This keyword returns the full name of the created extracted science object file.
	exmonitor		If this keyword is set, a line-profiles viewer is shown after all fitting is done when using optimal extraction. The default value is: 1
	compress		Set this keyword to compress all output files using gzip; the file suffix '.gz' is appended to all output filenames. The default value is: 0
	bsw_ofilename	X	n/a.
	opath		A scalar string that specifies the path where the output data are saved. The default value is: '.'
	opfx		A scalar string that specifies a prefix for the (automatically generated) output filename. The default value is: ''
	sfx		A scalar string that specifies the raw data filename suffix to use. The default value is: .fits
	detector		A scalar integer that specifies the currently selected detector; DETECTOR is a zero-based value. The default value is: 0
	logfile		This keyword specifies the name of an optional log file when p3d_cobjex is launched as a separate program.
	loglevel		If this routine is launched as a separate program, and LOGFILE is used, this keyword specifies the log level; [1], 2, 3 (the higher the value, the more output). The default value is: 1
	cinv		Set this keyword to plot white lines on a black background instead of the default, which is to plot black lines on a white background.
	recenterval		Set this keyword or RECENTERLIMVAL to activate the use of profile recentering on science images (recenterprofiles). This keyword configures several of the recentering options. If RECENTERVAL is a: string the telluric line mode is used ('recenterusetelluric'). The default file is used when the keyword is set to 'telluric' while any other string is interpreted as a filename ('recentertellines'). value, -2.0 <= value <= 2.0 the value is used as a preset offset (recenteroffset). other values are used as a median-filter width with data; the used width is the integer of the absolute value.
	recenterlimval		Set this keyword or RECENTERVAL to activate the use of profile recentering on science images ('recenterprofiles'). This keyword allows to filter out all spectra that have a lower maximum count than specified with this value [ADU], before calculating an offset using the median-filter method (when it is used). Note! This keyword is only considered if the user parameter 'recenterdpixpos' contains a single element.
	nf2f		Spectra are normalized fiber-to-fiber with a one-dimensional array if this keyword is set. The array is selected using the filename defined in the IMTRM header keyword of the flat-field image file (FLATFIELD). If the header keyword IMTRM is unavailable, and NF2F is set, data are normalized using a calculated transmission array of the flat-field image. In this case, data must not be normalized with the average spectrum (see the user parameter 'ff_normdisp'). For example, in the standard setup, if TRANSMISSION was set to the transmission-array file of a twilight flat while running p3d_cflatf, data are corrected for fiber-to-fiber transmission using that twilight-flat data here (see p3d_tool_flatfield), while the spectrum normalization is done using the flat-field image selected using FLATFIELD.
	pcutlow		A scalar floating-point value that specifies the extent of the total number of pixels on the dispersion axis removed from each spectrum at the lower (blue) end before calculating a fiber-to-fiber transmission array. This value is specified in per cent. Also see the user parameter 'ff_lowerpx'; This parameter os only used if NF2F is set. The default value is: 5.0
	pcuthigh		A scalar floating-point value that specifies the extent of the total number of pixels on the dispersion axis removed from each spectrum at the upper (red) end before calculating a fiber-to-fiber transmission array. This value is specified in per cent. Also see the user parameter 'ff_upperpx'; this parameter is only used if NF2F is set. The default value is: 5.0
	scattlightsubtract		Set this keyword to first calculate and thereafter subtract a scattered-light image. Such an image is calculated by smoothing and interpolating the raw-data image, using regions on the CCD that lie far enough from nearby spectra; considering only the cross-dispersion axis. The scattered-light image is subtracted from the masterbias-subtracted raw image before (calculating the error image and) spectra are extracted. Note! This procedure only works with data where there is some "empty" space left near the edges, and preferrably also between spectrum traces on the CCD, on the cross-dispersion axis, which can be used to measure the extent of scattered light.
	savefinalflat		Set this keyword to write the final normalized flat-field image to the disk; this is otherwise not done. Also see the user parameter 'ff_save_final_image'.
	satmask		Set this keyword to write saturated-pixels mask images to the disk. Such images indicate if any saturated pixels in the input images were used when calculating a spectrum image.
	crnthreads		A scalar integer that specifies how many threads are used when cleaning single input images of cosmic-ray hits. If this keyword is not set, NTHREADS is used instead. If nthreads was not set either, the default is to use the maximum number of available threads minus one, but the maximum number is 16: (!cpu.hw_ncpu – 1) < 16. The default value is: max – 1
	nthreads		A scalar integer that specifies how many threads to use in the parallelized line profile calculations. The calculation speed scales nearly linearly with the number of threads. The default is to use the maximum number available, but no more than 16: max = !cpu.hw_ncpu < 16. The default value is: max
	allinone		Unset this keyword to write the output data to separate files, instead of writing the extracted spectra to the first (DATA) extension, its error to the ERROR extension, the bad-pixels mask image to the MASK extension, the saturated-pixels mask image to the SATMASK extension, the data-quality mask image to the QUALITY extension, the normalized extracted flat-field image to the NORMFLAT extension, and the calculated scattered-light image to the SCATTLIG extension. The normalized flat-field image is only written if the keyword SAVEFINALFLAT or the user parameter 'ff_save_final_image' are set. The scattered-light image is only written if the user parameter 'sls_writefile' is set. The default value is: 1
	crclean		Set this keyword to clean all uncombined input images of cosmic-ray hits before the extraction procedure is started; either a master-bias image or prescan and overscan regions must be used to provide the bias level to use this functionality.
	nocrmask		Set this keyword to avoid using the cosmic-ray hit mask image, which is created when CRCLEAN is set, when extracting the spectra optimally. Normally, it is recommended to keep this keyword unset; it is available to allow the user the ultimate choice. The following keywords are used to configure the cosmic-ray cleaning procedure; they are only used if CRCLEAN is set; see p3d_cr and p3d_tool_crays_lapycosmic for more details:
	sigclip		A scalar decimal value that specifies the clipping sigma.
	objlim		A scalar decimal value that specifies the object limit value; this value is only used if IMAGEMETHOD is set.
	ratlim		A scalar decimal value that specifies the ratio limit value; this value is only used if IMAGEMETHOD is unset.
	crfwhm		A decimal scalar value, or a two-element array, that specifies the full width at half maximum of the gaussian kernel used to convolve the raw data. This keyword corresponds to FWHM in p3d_cr. This value is only used if IMAGEMETHOD is unset.
	gausskernelsize		A scalar integer that specifies the size of the Gaussian kernel used when convolving the input image with a Gaussian kernel using the PyCosmic algorithm. This value is by default set to 2 * q + 1, where q is the maximum value of the x and the y pixel where a one-dimensional Gaussian reaches 15% of its maximum value (1.0); this condition makes use of the full-width-at-half-maximum value of FWHM. The value must be odd, and the maximum value is in any case 19. Be aware that convolution times increase strongly with the size of the Gaussian kernel.
	sigfrac		A scalar decimal value that specifies a scale factor for pixels surrounding a cosmic-ray masked pixel; this parameter is only used if GROWRADIUS is non-zero. The default value is: 0.5
	growradius		Set this keyword to grow all cosmic-ray hit pixels by a frame that is this many pixels wide; this is done to check if the surrounding pixels are affected by cosmic-ray hits as well. The default value is 1, the values 0 and 2 are also accepted. Surrounding pixels are checked if they are cosmic-ray affected using the first criterion S' > SIGFRAC * SIGCLIP. The default value is: 1
	maxiter		A scalar integer value that specifies the maximum number of iterations used when identifying cosmic-ray hits in each image.
	imagemethod		Set this keyword to use the original algorithm of P. van Dokkum (L.A. Cosmic), which was developed for cosmic-ray detection in images. The default is otherwise to use the IFS-modified algorithm with IMAGEMETHOD unset.
	imageclean		Set this keyword to use the original [5 * 5 px²] two-dimensional median filter when replacing the flux in masked pixels with that of the surrounding pixels. An [11 px] one-dimensional median filter is used otherwise. The default value is: 1
	dispmedian		Set this keyword to a positive integer > 1 to subtract sampling flux along the dispersion axis. The value defines the width of the median window.
	writeall		Set this keyword to write all intermediate images in addition to the cleaned image and the cosmic-ray mask.
	showcrgui		Set this keyword to show a properties GUI of the cosmic-ray cleaning parameters before the algorithm is executed.
	nocrc		Set this keyword to force the use of the slower IDL-based cosmic-ray cleaning routine instead of the faster C-based routine.
	noobcheck		Set this keyword to avoid checking the OB header keyword; this only applies to the instruments that use it. To see if it applies to your instrument, check if the OBID parameter is defined in the instrument keywords file.
	nooptimal	X	Set this keyword to force tophat extraction; this is used by p3d_autoreduce to force the fastest reduction.
	fnotify		Set this keyword to the name of a file that is written when the tool has finished; this can be a useful feature when using the p3d queue handler. In that case, set this keyword and check if the file exists before continuing. And do not forget to remove the file.
	font		Set this scalar string to the name of the font to be used with all widget components of this tool. It is advised to use a fixed font, the outcome is otherwise difficult to predict. Set the keyword to 'big' to use the following pre-defined font ed-medium-r-normal--18-120-100-100-c-90-iso8859-1'; this could be useful on a system using a 4k screen (with 3160x2560 pixels). Another option to specify a font is to set the environment variable p3d_font or the p3d IDL system variable !p3d_font. The default font used with the widgets is, for your information, printed to the screen when FONT is unset and the highest verbosity level is used (verbose = 3). On Linux machines, use a shell command such as 'xlsfonts' to list all available fonts.
	cmdline	X	This keyword is used then the tool is launched by the p3d server.
	gui	X	If this keyword is set, the routine is assumed to be called from the p3d GUI – meaning that there is no initialization of the p3d system variables.
	stawid	X	If set, various messages are written to the p3d GUI status line (this must be the widget id of that label widget).
	topwid	X	If set, error messages are displayed using DIALOG_MESSAGE, using this widget id as DIALOG_PARENT, instead of MESSAGE.
	logunit	X	Messages are saved to the file pointed to by this logical file unit, if it is defined.
	verbose		Set this parameter to a positive scalar integer to make p3d write some information on STDOUT about what is going on. The following four values are acknowledged: 0 Writes no information at all. This is the default. 1 Writes the more important information; regarding subroutine configurations, mostly. 2 Writes most information; includes a more verbose output than 1. 3 Writes all information, including information on the execution state of GUI subroutines. This may be a useful mode when debugging the code.
	quiet		Set this keyword to not print the disclaimer on the screen when this tool is started from the command line or from the shell.
	error	X	This scalar integer returns an error code if set. Any value different from zero means that an error has occurred.
	debug	X	An error handler is not setup if this keyword is set. Otherwise, the error handler is setup using the routine CATCH, which makes each subroutine exit quietly when a bug is encountered. Use this keyword when debugging p3d.
	cdebug		A keyword as DEBUG used with the called C routines; valid values are 0-6 where a higher number results in more output to STDERR.
	help	X	Set this keyword to show this routine documentation, and then exit.

Output parameters:

	out	X	Upon a successful execution this variable contains the extracted data when the routine exits.
	bswout	X	Upon a successful execution this variable contains the extracted beam switch data when the tool finishes.

Examples:

This routine can be called in four different ways (for use on WINDOWS platforms, replace the path character '/' with '\'):

i) Using the p3d GUI

There is no need to setup the input parameters, p3d handles this transparently.

ii) As a stand-alone tool from the IDL command line

In addition to your input science-object images, it is necessary to pass the name of the (provided) instrument-specific parameter file, a master-bias image file or (at least) one of the bias prescan and overscan parameters (if you use optimal extraction), a trace-mask image file, a dispersion-mask image file (if used), and an extracted flat-field image file (if used), as well as any additional arguments to p3d:

files = ['objectfile_1.fits.gz', 'objectfile_2.fits.gz']
parfile = 'larr2k.prm'
opath = 'Output'
masterbias  = opath + '/masterbias.fits.gz'
tracemask   = opath + '/tracemask.fits.gz'
flatfield   = opath + '/flatfield.fits.gz'
userparfile = opath + '/user_p3d.dat' ;; If you use this file
detector = 0 ;; Required with multi-detector configurations.
p3d_cobjex, files, parfile, masterbias=masterbias, $
    tracemask=tracemask, flatfield=flatfield, $
    userparfile=userparfile, opath=opath, detector=detector, $
    logfile=opath + 'dred.log', /verbose

iii) As a stand-alone tool from a terminal, using the IDL VM

In addition to your input science-object images, it is necessary to pass the name of the (provided) instrument-specific parameter file, a master-bias image file or (at least) one of the bias prescan and overscan parameters (if you use optimal extraction), a trace-mask image file, a dispersion-mask image file (if you use one), a flat-field image file (if you use one), as well as any additional arguments to p3d. The recommended approach is to use the provided shell script p3d_cobjex_vm.sh or the Python tool p3d_dispatch.py:

# Using the shell script, making use of the p3d queue system:
parfile=larr2k.prm
opath=Output
masterbias=${opath}/masterbias.fits.gz
tracemask=${opath}/tracemask.fits.gz
flatfield=${opath}/flatfield.fits.gz
userparfile=${opath}/user_p3d.dat # If you use this file
detector=0 # Required with multi-detector configurations.
p3d_cobjex_vm.sh \
    objectfile_1.fits.gz,objectfile_2.fits.gz,objectfile_3.fits.gz \
    $parfile masterbias=$masterbias tracemask=$tracemask \
    flatfield=$flatfield userparfile=$userparfile \
    opath=$opath detector=$detector logfile=$opath/dred.log \
    /verbose

# Using the Python tool, without using the p3d queue system:
parfile=larr2k.prm
opath=Output
masterbias=${opath}/masterbias.fits.gz
tracemask=${opath}/tracemask.fits.gz
flatfield=${opath}/flatfield.fits.gz
userparfile=${opath}/user_p3d.dat # If you use this file
detector=0 # Required with multi-detector configurations.
p3d_dispatch.py --noqueue cobjex \
    objectfile_1.fits.gz,objectfile_2.fits.gz,objectfile_3.fits.gz \
    $parfile masterbias=$masterbias tracemask=$tracemask \
    flatfield=$flatfield userparfile=$userparfile \
    opath=$opath detector=$detector logfile=$opath/dred.log \
    /verbose

Note: In this approach, it is necessary to specify all options using the full names of the keywords. Partially correct keyword names are not permitted, as they are inside IDL. Hence, as an example, you must type "p3d_cobjex_vm.sh tracemask=$tracemask" and -NOT- "p3d_cobjex_vm.sh tr=$tracemask".

Note: The tool launches in the current directory, which is why it is unnecessary to specify full paths if files reside in the current directory.

Note: When several files are specified in the first argument, they must be separated with either a comma or a semicolon.

Note: Parameters indicated with an 'X' in the list above are unavailable in this approach. This includes the three output variable parameters OFILENAME, OUT, and BSWOUT.

iv) As a stand-alone tool from Python, using the IDL VM

This approach is analogous to iii), but is executed from the Python command line. Here is an example:

# Launch p3d_cobjex in a single IDL session, without using the
# p3d queue system:
from p3d_dispatch import *
parfile = 'larr2k.prm'
opath = 'Output'
masterbias = opath + '/masterbias.fits.gz'
tracemask  = opath + '/tracemask.fits.gz'
flatfield  = opath + '/flatfield.fits.gz'
userparfile = opath + '/user_p3d.dat' # If you use this file
detector = 0 # Required with multi-detector configurations.
files = 'objectfile_1.fits.gz,objectfile_2.fits.gz,' +
        'objectfile_3.fits.gz'
commandline = files + ' ' + parfile + ' masterbias=' + masterbias +
  ' tracemask=' + tracemask + ' flatfield=' + flatfield +
  ' userparfile=' + userparfile + ' opath=' + opath +
  ' detector=' + detector +
  ' logfile=' + opath + '/dred.log /verbose'
p3d_dispatch('cobjex', commandline, noqueue=True)

Cleans one or several raw spectrum images of fiber-fed Integral-Field Spectroscopy instruments of cosmic-ray hits.

This routine is a wrapper for the p3d IFS-capable version of the PyCosmic (Husemann et al. 2012, hereafter HKS12) and the L.A. Cosmic (van Dokkum 2001, hereafter vD01) algorithms. These routines clean single (uncombined) raw-data images of cosmic-ray hits. Requirements include a master-bias image, or (at least) one of the prescan and the overscan parameters, which is subtracted from the data before the cleaning algorithm is applied. All algorithm parameters are collected here, and data are also prepared and saved here. The algorithms themselves are placed in p3d_tool_crays_lapycosmic.

The most complete resource regarding the two supported algorithms – including a detailed comparison – is HKS12; this reference also compares the, so-called, DCR algorithm of Pych (2004). The default algorithm is PyCosmic; set the keyword IMAGEMETHOD or the user parameter 'laimagemethod' to use L.A. Cosmic instead. As is illustrated by HKS12, undersampled data are better cleaned with the L.A. Cosmic algorithm, in other cases it appears that PyCosmic does a better job.

To reproduce the results of HKS12 when using PyCosmic, it is necessary to set the grow-radius keyword GROWRADIUS or the user parameter 'growradius' to 0; the default value is 1. (This option is for data-reduction connaisseurs, as results appear to be very similar when using a non-zero grow radius.) With a zero grow radius, the outcome should be very similar, but not identical; in particular, single pixels with the lowest possible signal-to-noise are masked differently (for example where the calculated value is just above SIGCLIP). An algorithm difference between PyCosmic as implemented here and the Python-based routine, which might be the cause of differences between masks, is that the former (latter) performs one two-dimensional (two consecutive one-dimensional) convolutions with a Gaussian kernel.

Regardless of the selected algorithm, it is important to check that the cosmic-ray hit cleaning procedure only removes cosmic-ray hits, and does not affect telluric lines or object emission lines. It is often enough to compare the cleaned image with the cosmic-ray mask and the raw image. The keyword WRITEALL can be used to save all intermediate products, which allow a more detailed study of the quality of the cleaning procedure.

The output-data files consist of the cosmic-ray cleaned image and the cosmic-ray mask image. These are by default written to the same file, in consecutive extensions. They are instead written to separate files if the keyword ALLINONE or the user-parameter 'allinone' is unset. There is no real advantage to either approach; although, if separate files are used and the cosmic-ray mask image file is renamed it will not be found by the other p3d tools.

This tool can be used in two ways:

• Run this tool directly on raw-data images.

  Note: Regarding the PMAS 4kx4k CCD or other (future) multi-block instruments: the routine must be used with uncombined data; this also applies to the master-bias image.

  Note: This approach is (currently) unavailable with instruments that store data of different detectors in different extensions of the fits-file image. This applies to, for example, GMOS-S and GMOS-N.

• Set the CRCLEAN keyword when executing the tools p3d_ctrace, p3d_cdmask, p3d_cflatf, and p3d_cobjex. The selected algorithm is then applied to the (combined [if required]) image (format) that p3d works with. All algorithm-critical keywords of this tool are also available from the respective calling tool.

  Note: If several images are fed to the respective tool, they will at first be combined using, for example, a median or an average; unless the GROUP keyword is used to prevent this (see p3d_cobjex). A resulting combined image will not be cleaned of cosmic-ray hits, even if CRCLEAN is set.

Warning: It may be important to check the quality of the cleaned outcome. All output images that are created through the use of the WRITEALL keyword, require a lot of disk space; probably hundreds of megabytes and perhaps even gigabytes.

Speed tests

Here are results of speed tests on some VIMOS data, using 1-8 threads on a 4-core machine with hyperthreading. In this case, the PyCosmic algorithm was used; speeds ought to be slightly longer with the L.A. Cosmic algorithm. All values in columns 2-6 are in seconds. The speedup //Sp// = //T1// / //Tp//, and the efficiency //Ep// = //Sp// / //p//, where //p// is the number of threads. Values are also compared to the IDL routine, //Ip// = //T//(IDL) / //Tp//.

Quadrant: 1/4 2/4 3/4 4/4 mean Sp Ep Ip IDL 1 th. 58.069 57.724 57.847 57.849 57.872 1.26 1.26 1.00 C 1 th. 73.032 72.886 72.243 72.624 72.696 1.00 1.00 0.80 C 2 th. 39.693 39.824 39.334 39.404 39.564 1.84 0.92 1.46 C 3 th. 27.804 27.896 27.722 27.783 27.801 2.61 0.87 2.08 C 4 th. 23.882 24.138 23.776 24.142 23.985 3.03 0.76 2.41 C 5 th. 24.410 24.165 24.146 24.033 24.189 3.01 0.60 2.39 C 6 th. 21.315 21.309 21.395 21.380 21.350 3.40 0.57 2.71 C 7 th. 18.987 18.647 18.796 18.891 18.830 3.86 0.55 3.07 C 8 th. 24.556 24.279 24.272 24.152 24.315 2.99 0.37 2.38

The table shows a speedup far from linear. In this case, it seems to be sufficient to use 4 threads if the number of threads is the limiting factor, and 7 threads if the execution time of the highest priority.

References

van Dokkum, P.G. 2001, PASP, 113, 1420, "Cosmic-Ray Rejection by Laplacian Edge Detection" (vD01)

Pych, W. 2004, PASP, 116, 148, "A Fast Algorithm for Cosmic-Ray Removal from Single Images"

Husemann, B., Kamann, S., Sandin, C., Sánchez, S., Mast, D. and Garcia-Benito, R. 2012, A&A, 545, A137, "PyCosmic: a robust method to detect cosmics in CALIFA and other fiber-fed integral-field spectroscopy datasets" (HKS12)

Links

The PyCosmic code of HKS12 can be found here: http://pycosmic.sf.net

The original L.A. Cosmic code(s) of vD01 can be found here: http://www.astro.yale.edu/dokkum/lacosmic/

The DCR code of Pych (2004) can be downloaded here: http://users.camk.edu.pl/pych/DCR/index.html

General information on how to learn more about this routine

Instrument parameter files (*.prm), which are required to run this routine, are found in the directory "!p3d_path/data/instruments/*/" (outside of IDL, check "${p3d_path}/data/instruments/*/").

A default (master) user-parameter file is available in "!p3d_path/data/master_userparfile.dat". Copy this file to your data-reduction output directory, rename the file to, for example, "user_p3d.dat" and edit it to suit your needs.

Examples on how to use this routine are provided in the EXAMPLES section below.

See the file README in the root of the p3d package for installation instructions, if p3d is not already installed on the computer.

Additional information, documentation, new versions, etc., can be found at the p3d project web site "https://p3d.sourceforge.io".

Requirements and format of the input and the output data

p3d only works with FITS-formatted images, i.e. fits files. Such files have a header that contains additional information about the observing conditions, the telescope, the instrument setup, and other useful information. Both input and output fits files can be compressed using gzip, to save storage space (also on Windows platforms); p3d handles such compressed files transparently.

This tool takes one, or several, raw images, or (block-)combined images spectrum images as input, along with a matching master-bias image.

In addition to the standard fits-file header keywords, the resulting image contains additional p3d-specific keywords:

Input parameters:

	image		This variable contains the images to be cleaned of cosmic-ray hits. The value can be specified in different ways: A two-dimensional array (scalar image) or a three-dimensional array (array of images), where the array is counted in the first dimension. The array must be of floating-point type. A scalar string or an array of strings that contain(s) the FITS-filename(s) of the images; the image files can be compressed using gzip or compress – this must be reflected in the filename suffix, i.e., each file must, if it is compressed, end with ".gz" or ".Z".
	parfile		A scalar string that specifies the name of a file with instrument-specific setup parameters. These files follow a strict format. Already properly formatted files are available for all configured instruments in the directories p3d_path/data/instruments/*/. It is sufficient to specify the file basename, without the full path of the p3d subdirectory; for example: "nvimos_hr.prm".

Keyword parameters:

	masterbias		A scalar string with the name of a master-bias image file. This file is required if optimal extraction is used and none of the following four keywords is used: BIASPX, BIASPY, BIASOX, or BIASOY. The value can be specified in two different ways: as a two-dimensional array (scalar image) of floating-point type. as a scalar string that contains the name of the master-bias image file. Note! A master-bias image can of course be neither subtracted nor added if it is not provided. Note! This keyword must not be set if the prescan and the overscan regions should be used to estimate the bias level.
	biasadd		Set this keyword to add the bias image to the cleaned bias-subtracted image before saving it. The default value is: 1
	biaspx		Set this keyword to use the x-axis prescan region when subtracting a bias. Specifically, a median value is calculated for each pixel on the y-axis in the x-axis prescan region. The same array is used with all y-axis columns of data that are bias subtracted.
	biaspy		Set this keyword to use the y-axis prescan region when subtracting a bias. Specifically, a median value is calculated for each pixel on the x-axis in the y-axis prescan region. The same array is used with all x-axis rows of data that are bias subtracted.
	biasox		Set this keyword to use the x-axis overscan region when subtracting a bias. Specifically, a median value is calculated for each pixel on the y-axis in the x-axis overscan region. The same array is used with all y-axis columns the data that are bias subtracted.
	biasoy		Set this keyword to use the y-axis overscan region when subtracting a bias. Specifically, a median value is calculated for each pixel on the x-axis in the y-axis overscan region. The same array is used with all x-axis rows of data that are bias subtracted.
	biasconstant		This keyword can be used in two ways: Set this keyword to use a constant value as a bias value; otherwise an array is used. All prescan and overscan regions specified using the keywords BIASPX, BIASPY, BIASOX, and BIASOY can be used when this keyword is set; otherwise, only one of them. If this keyword is set to a double type array or to a comma-separated string with values greater than one (when called from the shell), these values are used to create a bias. The number of elements must be the same as the number of blocks or extensions.
	prescanx	X	This keyword is used to define x-axis prescan regions when IMAGE is a floating-point-type-value array. In this case, and if this keyword is used through BIASPX, the keyword contains the x-axis prescan array as tags 'd1'–'dn', where //n// is short for the number of blocks. The array of the tag 'use' is set to 1 whenever data are present, otherwise 0.
	prescany	X	This keyword is used to define y-axis prescan regions when IMAGE is a floating-point-type-value array. In this case, and if this keyword is used through BIASPY, the keyword contains the y-axis prescan array as tags 'd1'–'dn', where //n// is short for the number of blocks. The array of the tag 'use' is set to 1 whenever data are present, otherwise 0.
	overscanx	X	This keyword is used to define x-axis overscan regions when IMAGE is a floating-point-type-value array. In this case, and if this keyword is used through BIASOX, the keyword contains the x-axis overscan array as tags 'd1'–'dn', where //n// is short for the number of blocks. The array of the tag 'use' is set to 1 whenever data are present, otherwise 0.
	overscany	X	This keyword is used to define y-axis overscan regions when IMAGE is a floating-point-type-value array. In this case, and if this keyword is used through BIASOY, the keyword contains the y-axis overscan array as tags 'd1'–'dn', where //n// is short for the number of blocks. The array of the tag 'use' is set to 1 whenever data are present, otherwise 0.
	sigclip		A scalar floating-point type value that specifies the clipping sigma, see p3d_tool_crays_lapycosmic. The default value is: 5.0
	objlim		A scalar floating-point type value that specifies the object limit value, see p3d_tool_crays_lapycosmic; this value is only used if IMAGEMETHOD is set. The default value is: 2.0
	ratlim		A scalar floating-point type value that specifies the ratio limit value, see p3d_tool_crays_lapycosmic.pro; this value is only used if IMAGEMETHOD is unset. The default value is: 1.2
	fwhm		A floating-point type scalar value, or a two-element value, that specifies the full width at half maximum of the gaussian kernel used to convolve the raw data. If only one value is specified, it is used with both axes. These two values are divided with the two CCD bin numbers. This keyword is only used if IMAGEMETHOD is unset. The default value is: fwhm_tr, fwhm_tr
	gausskernelsize		A scalar integer that specifies the size of the Gaussian kernel used when convolving the input image with a Gaussian using the PyCosmic algorithm. The default value is 2 * //q// + 1, where //q// is the maximum value of the //x// and the //y// pixel where a one-dimensional Gaussian reaches 15 % of its maximum value (1.0); this condition makes use of FWHM. The value must be odd, and the maximum value is in any case 19. Be aware that convolution times increase strongly with the size of the Gaussian kernel.
	sigfrac		A scalar floating-point type value that specifies a scale factor for pixels surrounding a cosmic-ray masked pixel. This parameter is only used if GROWRADIUS is non-zero. The default value is: 0.5
	growradius		Set this scalar-integer keyword to grow all cosmic-ray hit pixels by a frame this many pixels wide; this is done to check if surrounding pixels are affected by cosmic-ray hits as well. The default value is 1; 0 and 2 are also accepted. The first criterion S' > SIGCLIP is replaced with S' > SIGFRAC * SIGCLIP when surrounding pixels are checked if they are cosmic-ray affected. To reproduce the PyCosmic-specific results of HKS12, it is necessary to set GROWRADIUS = 0. The default value is: 1
	maxiter		A scalar integer that specifies the maximum number of iterations used when identifying cosmic-ray hits in each image, see p3d_tool_crays_lapycosmic. The default value is: 4
	imagemethod		Set this keyword to use the original algorithm of P. van Dokkum (L.A. Cosmic), which was developed for cosmic-ray detection in images. The default is otherwise to use the IFS-modified algorithm with IMAGEMETHOD unset.
	imageclean		Set this keyword to use the original [5 * 5 px²] two-dimensional median filter when replacing the flux in masked pixels with that of the surrounding pixels. otherwise an [11 px] one-dimensional median filter is used. The default value is: 1
	dispmedian		Set this keyword to a positive integer > 1 to subtract sampling flux along the dispersion axis. The value defines the width of the median window.
	writeall		Set this keyword to write all intermediate images in addition to the cleaned image and the cosmic-ray mask.
	sfx		A scalar string that specifies the output filename suffix without any file-compression suffix. The default value is: '.fits'
	cleansfx		A scalar string that specifies the output filename suffix appended to saved cleaned images. The default value is: '_crcl'
	masksfx		A scalar string that specifies the output filename suffix appended to saved cosmic-ray masks. The default value is: '_crmk'
	laplsfx		A scalar string that specifies the output filename suffix appended to saved //L^+// images; this is only used when WRITEALL is set. The default value is: '_crlp'
	poflsfx		A scalar string that specifies the output filename suffix appended to saved //S// images; this is only used when WRITEALL is set. The default value is: '_crpf'
	siprsfx		A scalar string that specifies the output filename suffix appended to saved //S'// = //L^+// / (2 * //N//) images; this is only used when WRITEALL is set. The default value is: '_crsp'
	laprsfx		A scalar string that specifies the output filename suffix appended to saved //L^+// (IMAGE / conv(IMAGE)) images; this is only used when WRITEALL is set and IMAGEMETHOD is unset. The default value is: '_crlr'
	finesfx		A scalar string that specifies the output filename suffix appended to saved fine-structure images; this is only used when WRITEALL is set and IMAGEMETHOD is set. The default value is: '_crfs'
	wallsfx		A scalar string that specifies the output filename suffix appended to saved multi-extension images; this is only used when WRITEALL is set. The default value is: '_crwa'
	compress		Set this keyword to compress all output files using gzip; the file suffix '.gz' is appended to all output filenames. The default value is: 0
	nthreads		A scalar integer that specifies how many threads re used when cleaning single input images of osmic-ray hits. The calculation speed scales early linearly with the number of threads. The efault is to use the maximum number of available hreads minus one, but the maximum number s 16: (!cpu.hw_ncpu – 1) > 1 < 16. The default value is: max-1
	showgui		Set this keyword to show a properties GUI before the algorithm is launched. The GUI allows an adjustment of the parameters critical to the algorithm.
	noC		Set this keyword to force the use of the slower IDL-based routine instead of the faster compiled C-based routine.
	gain		A floating-point type value with as many elements as IMAGE. The unit of GAIN must be electrons per analog-to-digital unit [e- / ADU]. The value is retrieved from the respective file if it is not provided (using IMAGE when it is a string, or FILENAMES when IMAGE is of floating-point type).
	ron		A floating-point type with as many elements as IMAGE. The unit of RON must be analog-to-digital units [ADU]. The value is retrieved from the respective file if it is not provided (using IMAGE when it is a string, or FILENAMES when IMAGE is of floating-point type).
	filenames	X	The output filenames are created using the strings in this array if IMAGE is not a string array. The same files are used to retrieve the FITS header arrays.
	opath		A scalar string that specifies the path where the output data are saved. The default value is: '.'
	opfx		A scalar string that specifies a prefix for the (automatically generated) output filename. The default value is: ''
	detsec	X	A four-element by //n//-element integer array that specifies the detector region to use on the CCD. DETSEC must be present when the number of blocks is greater than one.
	userparfile		A scalar string with the name of a file with user- defined parameters. The following parameters are read here from this file: 'keywordsfile' The name of the file with instrument-specific header keywords. 'compress' Corresponds to COMPRESS. 'allinone' Corresponds to ALLINONE. 'nthreads_cr' Corresponds to NTHREADS. 'nthreads' Corresponds to NTHREADS; this parameter is only read if 'nthreads_cr' is not found. 'biaspx' Corresponds to BIASPX. 'biaspy' Corresponds to BIASPY. 'biasox' Corresponds to BIASOX. 'biasoy' Corresponds to BIASOY. 'biasconstant' Corresponds to BIASCONSTANT. 'laimagemethod' Corresponds to IMAGEMETHOD. 'laimageclean' Corresponds to IMAGECLEAN. 'ladispmedian' Corresponds to DISPMEDIAN. 'crwriteall' Corresponds to WRITEALL. 'nocrc' Corresponds to NOC. 'sigclip' Corresponds to SIGCLIP. 'objlim' Corresponds to OBJLIM. 'ratlim' Corresponds to RATLIM. 'crfwhm' Corresponds to FWHM. 'crgausskernelsize' Corresponds to GAUSSKERNELSIZE. 'fwhm_tr' Used to set the width of the Gaussian profile when crfwhm is unavailable and IMAGEMETHOD is unset. 'sigfrac' Corresponds to SIGFRAC. 'growradius' Corresponds to GROWRADIUS. 'maxiter' Corresponds to MAXITER. 'opfx' Corresponds to OPFX. 'cleansfx' Corresponds to CLEANSFX. 'masksfx' Corresponds to MASKSFX. 'laplsfx' Corresponds to LAPLSFX. 'poflsfx' Corresponds to POFLSFX. 'siprsfx' Corresponds to SIPRSFX. 'laprsfx' Corresponds to LAPRSFX. 'finesfx' Corresponds to FINESFX. 'wallsfx' Corresponds to WALLSFX. 'a2' To use A3 instead of A4. 'a3' To use A3 instead of A4. 'a5' To use A3 instead of A4. 'letter' To use US Letter instead of A4 with PostScript and PDF output. 'legal' To use US Letter instead of A4 with PostScript and PDF output. 'tabloid' To use US Tabloid instead of A4 with PostScript and PDF output. 'psusletter' To use US Letter instead of A4 with PostScript and PDF output. Note! If any of the corresponding keywords is specified, that value takes precedence.
	allinone		Unset this keyword to write the output data to separate files, instead of writing the cosmic-ray cleaned image to the first (DATA) extension and the cosmic-ray mask image to the MASK extension. The default value is: 1
	daxis	X	A scalar integer that specifies the dispersion dimension in the input image(s). The default value is: 1
	logfile		This keyword specifies the name of an optional log file when p3d_cobjex is launched as a separate program.
	loglevel		If this routine is launched as a separate program, and LOGFILE is used, this keyword specifies the log level; [1], 2, 3 (the higher the value, the more output). The default value is: 1
	fnotify		Set this keyword to the name of a file that is written when the tool has finished; this can be a useful feature when using the p3d queue handler. In that case, set this keyword and check if the file exists before continuing. And do not forget to remove the file.
	font		Set this scalar string to the name of the font to be used with all widget components of this tool. It is advised to use a fixed font, the outcome is otherwise difficult to predict. Set the keyword to 'big' to use the following pre-defined font ed-medium-r-normal--18-120-100-100-c-90-iso8859-1'; this could be useful on a system using a 4k screen (with 3160x2560 pixels). Another option to specify a font is to set the environment variable p3d_font or the p3d IDL system variable !p3d_font. The default font used with the widgets is, for your information, printed to the screen when FONT is unset and the highest verbosity level is used (verbose = 3). On Linux machines, use a shell command such as 'xlsfonts' to list all available fonts.
	cmdline	X	This keyword is used then the tool is launched by the p3d server.
	gui	X	If this keyword is set, the routine is assumed to be called from the p3d GUI – meaning that there is no initialization of the p3d system variables.
	subroutine	X	If set, parts of the routine opening message will not be written to the log file.
	stawid	X	If set, various messages are written to the p3d GUI status line (this must be the widget id of that label widget).
	topwid	X	If set, error messages are displayed using DIALOG_MESSAGE, using this widget id as DIALOG_PARENT, instead of MESSAGE.
	logunit	X	Messages are saved to the file pointed to by this logical file unit, if it is defined.
	verbose		Set this parameter to a positive scalar integer to make p3d write some information on STDOUT about what is going on. The following four values are acknowledged: 0 Writes no information at all. This is the default. 1 Writes the more important information; regarding subroutine configurations, mostly. 2 Writes most information; includes a more verbose output than 1. 3 Writes all information, including information on the execution state of GUI subroutines. This may be a useful mode when debugging the code.
	quiet		Set this keyword to not print the disclaimer on the screen when this tool is started from the command line or from the shell.
	error	X	This scalar integer returns an error code if set. Any value different from zero means that an error has occurred.
	debug	X	An error handler is not setup if this keyword is set. Otherwise, the error handler is setup using the routine CATCH, which makes each subroutine exit quietly when a bug is encountered. Use this keyword when debugging p3d.
	cdebug		A keyword as DEBUG used with the called C routines; valid values are 0-6 where a higher number results in more output to STDERR.
	help	X	Set this keyword to show this routine documentation, and then exit.

Examples:

This routine can be called in three different ways (for use on WINDOWS platforms, replace the path character '/' with '\'):

i) As a stand-alone tool from the IDL command line

In addition to the raw spectrum images, it is necessary to pass the name of the (provided) instrument-specific parameter file, as well as any additional arguments to p3d_cr:

files = ['file_1.fits.gz', 'file_2.fits.gz']
parfile=  'nvimos_hr.prm'
opath = 'Output'
userparfile = opath + '/user_p3d.dat' ;; If you use this file
p3d_cr, files, parfile, userparfile=userparfile, $
    opath=opath, logfile=opath+'dred.log', /verbose

ii) As a stand-alone tool from a terminal, using the IDL VM

In addition to the raw spectrum images, it is necessary to pass the name of the (provided) instrument-specific parameter file as well as any additional arguments to p3d_cr. The recommended approach is to use the provided shell script p3d_cr_vm.sh or the Python tool p3d_dispatch.py:

# Using the shell script, without using the p3d queue system:
files=file_1.fits.gz,file_2.fits.gz
parfile=nvimos_hr.prm
opath=Output
userparfile=${opath}/user_p3d.dat # If you use this file
p3d_cr_vm.sh -s $files $parfile userparfile=$userparfile \
    opath=$opath logfile=$opath/dred.log /verbose

# Using the Python tool, using the p3d queue system:
files=file_1.fits.gz,file_2.fits.gz
parfile=nvimos_hr.prm
opath=Output
userparfile=${opath}/user_p3d.dat # If you use this file
p3d_dispatch.py cr $files $parfile userparfile=$userparfile \
    opath=$opath logfile=$opath/dred.log /verbose

Note: In this approach, it is necessary to specify all options using the full names of the keywords. Partially correct keyword names are not permitted, as they are inside IDL. Hence, as an example, you must type "p3d_cr_vm.sh userparfile=$userparfile" and -NOT- "p3d_cr_vm.sh user=$userparfile".

Note: The tool launches in the current directory, which is why it is unnecessary to specify full paths if files reside in the current directory.

Note: When several files are specified in the first argument, they must be separated with either a comma or a semicolon.

Note: Parameters indicated with an 'X' in the list above are unavailable in this approach. This includes the output variable parameter OIMAGE.

iii) As a stand-alone tool from Python, using the IDL VM

This approach is analogous to iii), but is executed from the Python command line. Here is an example:

# Launch p3d_cr in a single IDL session, without using the
# p3d queue system:
from p3d_dispatch import *
files = 'file_1.fits.gz,file_2.fits.gz'
parfile = 'nvimos_hr.prm'
opath = 'Output'
userparfile = opath + '/user_p3d.dat' # If you use this file
commandline = files + ' ' + parfile' +
  ' userparfile=' + userparfile + ' opath=' + opath + ' nthreads=4' +
  ' detector=' + detector + ' logfile=dred.log loglevel=2 /verbose'
p3d_dispatch('cr', commandline, noqueue=True)

Create simulated data that can be reduced with the reduction tools.

The purpose of this tool is to create simulated data that are saved to "raw" files, which can then be reduced with the p3d reduction tools. The simulated data consist of bias images, continuum images, arc images, and object images.

The simulated raw data are created in a number of steps. The short description of the procedure is as follows:

• The configuration is chosen using the instrument-parameter file. The same configuration defines the number of spectra, their separation, and similar properties. The wavelength range is selected from the options available in the instrument gratings file using the keyword GRATINGSELECT.

  Note: The number of spectra can, for some setups, also be set using the keyword SPNUM. The spectrum separation using the keyword DIST, and the spectrum separation using the keyword GAPSNUM.

• Continuum, arc, and object spectra are loaded or configured and interpolated to match the selected wavelength range. The object point source spectrum is additionally distributed on the integral field unit (IFU) using a two-dimensional Gaussian profile, which simulates seeing effects. The created continuum and arc spectra are used with all spatial elements.

• The individual spectra are copied to an output image one by one, accounting for effects of dispersion and cross-dispersion profiles through convolution with specified or default kernels.

  The convolution kernels can be provided for any number of wavelengths on the dispersion axis, using one kernel with all spectra or one kernel per spectrum. Furthermore, the convolution is either done for both the dispersion axis and the spatial axis simultaneously using two-dimensional kernels (KERNEL), or for either axis separately using one-dimensional kernels (DKERNEL and CKERNEL).

  A separate keyword (TRPOSPAR) is used to specify a field distorsion (cross-dispersion axis pixel offsets) from a straight spectrum for each spectrum.

  Two other, optional, keywords (DISPOFFSET and DISPMASK) are used to specify a pixel offset on the dispersion axis for each spectrum (DISPOFFSET) or a prepared dispersion mask, which defines pixel offsets on the dispersion axis for all pixels of each spectrum.

• Photon noise and readout noise are added to the respective image before they are all saved to raw data files.

A more detailed description of each step is given in the consecutive sections.

Select the instrument configuration

It is not possible to select a random instrument setup of p3d to create simulation data; because, the instrument must be provided with an auxiliary setup file that describes instrument-specific header keywords. The specific and required file parameter is 'keywordsfile_p' that must be set in the instrument-parameter file.

The basic instrument setup is defined in the instrument-parameter file. Properties such as the number of spectra in the simulated IFU, the fiber position table and basic properties of the resulting raw CCD image are fixed in this file. However, properties marked with an asterisk can be changed with a user-parameter file and can thus be set with more freedom.

The basic spectrum configuration

Use the keyword DETECTOR to choose between already configured setups. The chosen setup fixes the number of spectra, their separation, and how the spectra are spatially distributed on the IFU. All setups are provided in the one same instrument parameter file.

Note: The number of spectra can, for some setups, also be set with the keyword SPNUM. The spectrum separation using the keyword DIST, and the spectrum separation using the keyword GAPSNUM.

Select the wavelength range

The simulated wavelength range is defined in the file pointed to with the instrument parameter 'gratingsfile_dm'. The wavelength range is selected among the options that are available in that file; where the values are assumed to be specified in the Angstrom (Å) unit.

Spectrum setup

Spectra for the different products are setup slightly differently depending on the required outcome.

Continuum spectrum

The same continuum spectrum is used with all spectra. It is used to create a continuum raw image that is used to trace the spectra in the simulated CCD images using p3d_ctrace.

Using the keyword CONT_SPECTRUM, the spectrum can be provided as a two-column list of wavelengths and fluxes in a plain-text file or a binary-table FITS file. The file is in both these cases read using the routine p3d_misc_fluxcal_read_file_standard_star (see that file for requirements on the data); associated and possibly useful keywords in reading the file are CONT_CATALOG, CONT_FLUXUNIT, and CONT_WAVEUNIT.

The spectrum is rescaled from any physical units to electron counts using the keyword CONT_SCALE. Specifically, the median of the continuum spectrum is set to the value of CONT_SCALE, which default value is 10000.0 e-.

Alternatively, a constant-value spectrum is used when no continuum spectrum is defined.

Note: Use CONT_SCALE to find a value that results in spectra that are not saturated in the written raw data files. The maximum analog-to-digital unit count per pixel is 65535 (2 ^ 16 – 1; this is the unit after the electron count is multiplied with the block-specific gain before it is saved to the output file).

Object spectrum

The current options to setup an object spectrum are the following:

• Point source

A point source spectrum is smeared out on the simulated CCD image using a two-dimensional Gaussian profile. Some spectra will consequently contain more of the object signal than others.

Using the keyword PSF_SPECTRUM, the spectrum can be provided as a two-column list of wavelengths and fluxes in a plain-text file or a binary-table FITS file. The file is in both these cases read using the routine p3d_misc_fluxcal_read_file_standard_star (see that file for requirements on the data); associated and possibly useful keywords in reading the file are PSF_CATALOG, PSF_FLUXUNIT, and PSF_WAVEUNIT.

The spectrum is rescaled from any physical units to counts using the keyword PSF_SCALE. Specifically, the median of the object spectrum is set to the value of the scalar value of PSF_SCALE, which default value is 10000.0 e-.

Note: The rescaling could result in extremely bright emission lines if you are using an emission-line spectrum without a continuum. In this case, consider using the alternative option instead (this option is described next).

Alternatively, set the keyword PSF_SPECTRUM to a list of wavelengths of emission lines and simultaneously specify the intensity of each emission line as a list of values using the keyword PSF_SCALE. The line width of the resulting emission-line spectrum is initially 0.

Note: Choose PSF_SCALE in such a way that the spectra are not saturated in the written raw data files. The maximum analog-to-digital unit count per pixel is 65535 (2 ^ 16 – 1; this is the unit after the electron count is multiplied with the block-specific gain before it is saved to the output file).

The parameters of the two-dimensional Gaussian profile are set using the three keywords PSF_FWHM (the seeing, basically), PSF_CENTROID (the spatial position on the simulated IFU, which is defined in the fiber position table), and PSF_XY_CORREL (the angle of an elliptical Gaussian profile). The IFU is subsampled to a higher resolution using the keyword SUBSAMPLE before the fraction of the intensity per spatial element and individual spectrum is calculated.

Arc spectrum

The arc line data (arc spectrum) are used to create an arc image used to create a dispersion mask during the reductions using p3d_cdmask.

Using the keyword ARCLINES, the emission lines can be provided as a two-column list of wavelengths (Å) and intensities (counts) in a plain-text file. Alternatively, ARCLINES can be set to a list of wavelengths for all emission lines, with the respective intensity specified using ARC_SCALE (electron counts).

Creating the simulated raw image data

The properties of the simulated raw image data including the initial spectrum positions are defined by the following instrument parameters:

• 'detsize': Defines the size of the created raw CCD image.

• 'daxis': Defines the dispersion axis.

• 'gapfile': Defines a file that specifies the geometrical properties of spectrum gaps on the CCD. The same file may contain the 'startpx' parameter that defines the position on the cross-dispersion axis of the CCD where the first spectrum is placed.

  Attention: Also see the keyword GAPSNUM.

• 'dist_tr': Defines the spectrum separation; this parameter is specified for unbinned data. The position of each spectrum is calculated from the gaps array, the associated position of the first spectrum and the spectrum separation.

  Attention: Also see the keyword DIST.

• 'dist_random_tr': Defines a random offset in the spectrum separation; this parameter is specified for unbinned data.

The initial raw image is immediately subsampled on the dispersion axis (DSUBSAMPLE) to allow convolution on the same axis at a finer resolution.

Combined dispersion-axis and spatial-axis kernel

For each spectrum and pixel on the dispersion axis, the two-dimensional kernel is linearly interpolated from the two closest kernels provided with the file of the keyword KERNEL; notably, the kernel is subsampled on the dispersion (cross-dispersion) axis using the value in the header keyword DSUBSAMP (CSUBSAMP), which corresponds to the keyword DSUBSAMPLE (CSUBSAMPLE).

Before a spectrum is convolved with the respective kernel, the latter is shifted on the two axes:

• The kernel can be shifted on the dispersion axis for each pixel individually using a dispersion offset (DISPOFFSET) or a mask (DISPMASK).

• It is possible to specify cross-dispersion pixel offsets in the form of a curve for each spectrum; the offset is specified using the keyword TRPOSPAR, which must be either an array with polynomial coefficients or a FITS file.

  Array of polynomial coefficients: see keyword description.

  File: The FITS file must contain a two-dimensional image in its zeroth extension. The image contains //P// columns, defining a polynomial of order //P – 1//, and //nspec// rows, where //nspec// must equal the number of spectra.

  The kernel is shifted on the cross-dispersion axis for each pixel individually by the specified offset.

After the subsampled spectrum is convolved with the shifted kernels, it is rebinned to remove the subsampling and copied to the output image.

Attention: When two-dimensional kernels are shifted, it is important that the kernels are padded with enough pixels that are set to zero; otherwise, the profiles will be truncated. Either pad the kernels before using them here or use the keyword ZPAD and it is done here.

Dispersion-axis kernel

For each spectrum and pixel on the dispersion axis, the dispersion kernel is linearly interpolated from the two closest kernels provided with the file of the keyword DKERNEL; notably, the kernel is subsampled using the factor DSUBSAMPLE.

Before the spectrum is convolved with the kernel, the latter is optionally shifted using the dispersion offset (DISPOFFSET) or the dispersion mask (DISPMASK).

Cross-dispersion-axis kernel

For each spectrum and pixel on the dispersion axis, the cross-dispersion kernel is linearly interpolated from the two closest kernels provided with the file of the keyword CKERNEL; notably, the kernel is subsampled using the factor CSUBSAMPLE.

It is possible to specify pixel offsets in the form of a curve for each spectrum; the offset is specified using the keyword TRPOSPAR, which must be either an array of polynomial coefficients or the name of a fits file containing a two-dimensional image in the zeroth extension. The image contains //P// columns, defining a polynomial of order //P – 1//, and //nspec// rows, where //nspec// must equal the number of spectra.

The cross-dispersion axis kernel (CKERNEL) is shifted by the pixel offset, and multiplied with the resulting spectrum. Thereafter, the profile is rebinned to remove the subsampling and copied to the output image.

Simulation of spherical aberrations

Spherical aberrations are simulated by convolving the simulated spectrum image with a Gaussian profile.

The FWHM of the Gaussian profile is set with the instrument parameter 'aberration_fwhm' or, alternatively, the keyword ABERRATION_FWHM. The allowed range of values is 0.5 <= ABERRATION_FWHM <= 5.0; the feature is deactivated when the value is 0.

Write simulated data to output files

Four kinds of simulated image data are written consecutively to separate FITS files:

• Object image; file suffix "_obj-N.fits, where N is an integer that indicates which image with added noise is written (1 <= N <= NUM_OBJ).

• Continuum image; file suffix "_cont-N.fits, where N is an integer that indicates which image with added noise is written (1 <= N <= NUM_CONT).

• Arc image; file suffix "_arc-N.fits, where N is an integer that indicates which image with added noise is written (1 <= N <= NUM_ARC).

• Bias image; file suffix "_bias-N.fits, where N is an integer that indicates which image with added noise is written (1 <= N <= NUM_BIAS).

The respective image is split into blocks, which are also flipped, as indicated in the instrument-parameter file; the used image sections, also indicating prescan and overscan regions, are specified in the instrument preset keywords file along with block-specific gain and readout noise values.

The instrument preset keywords file also defines the content of the zeroth and consecutive header extensions

The bias level (BIAS_LEVEL), Poisson shot noise (normal distribution), as well as readout noise (uniform distribution) are added to the image before it is converted to the IDL unsigned integer-type (maximum value is 65535) and written to the output file.

Arc spectrum – example data

The plain-text arc spectrum file 'mosms_arc_spec.dat' mentioned below in the EXAMPLES section could contain the following arc-line entries (it is as mentioned just an example and any content that makes sense would do).

;; Contents of: mosms_arc_spec.dat
4046.563  1000.
4077.831  1000. 
4358.328  2000.
4916.07   1000.
5037.75   1000.
5330.78   1000.
5342.185  1000.
5400.562  1000.
5748.299  1000.
5764.418  1000.
5820.148  1000.
5852.4878 1200.
5881.895  1200.
5944.8342 1000.
5975.534  1000.
6029.9971 1000.
6074.3377  100.
6096.1630  100.
6143.0623  100.
6163.5939 1000.
6217.2813 1000.
6266.4950 1000.
6304.7892 1000.
6334.4279 2000.
6382.9914 2000.
6402.246  1000.
6506.5279 1000.
6532.8824 1000.
6598.9529 1000.
6678.2764 1000.
6717.0428 1000.
6929.468  1000.
7024.050  1000.
7032.4127 1000.
7173.939  1500.
7245.167  1000.
7438.899  1000.
7488.871  1000.
7535.775  1500.
7943.181  1000.
8082.458  1000.

Links

The spectrum file used in the examples section below can be downloaded from the following web site: http://www.eso.org/sci/observing/tools/standards/spectra.html

General information on how to learn more about this routine

Instrument parameter files (*.prm), which are required to run this tool, are found in the directory "!p3d_path/data/instruments/*/" (outside IDL check "${p3d_path}/data/instruments/*/"). Speficially, this tool requires the presence of the file "<instrument>_keywords_preset.dat".

A default (master) user-parameter file is available in "!p3d_path/data/master_userparfile.dat". Copy this file to your data-reduction output directory, rename the file to, for example, "user_p3d.dat" and edit it to suit your needs.

Examples on how to use this tool are provided in the EXAMPLES section below.

See the file README in the root of the p3d package for installation instructions, if p3d is not already installed on the computer.

Additional information, documentation, new versions, etc., can be found at the p3d project web site "https://p3d.sourceforge.io".

Requirements and format of the input and the output data

p3d only works with FITS-formatted images, i.e. fits files. Such files have a header that contains additional information about the observing conditions, the telescope, the instrument setup, and other useful information. Both input and output fits files can be compressed using gzip, to save storage space (also on Windows platforms); p3d handles such compressed files transparently.

This routine creates simulated raw data files, using a few existing files as input, if needed. The tool splits the simulated images into blocks that are written to separate extensions, adding prescan and overscan information. The written files are meant to be used as input to the other data-reduction tools of p3d.

Input parameters:

parfile

A scalar string that specifies the name of a file with instrument-specific setup parameters. These files follow a strict format. Already properly formatted files are available for all configured instruments in the directories p3d_path/data/instruments/*/. It is sufficient to specify the file basename, without the full path of the p3d subdirectory; for example: "mosms.prm".

Keyword parameters:

	psf_fwhm		- A two-element decimal type array that specifies the seeing (full width at half maximum) of the spatial two-dimensional Gaussian used with the PSF source; one value for the x- and the y-dimension, respectively. The unit is arcsec for spatial element shapes other than squares, and in that case the unit is number of spatial element (sides). The default is [2.0, 2.0]. The default value is: 2.0, 2.0
	psf_centroid		- A two-element decimal type array that specifies the center position of the PSF on the IFU; the first value specifies the x-axis center position and the second value the y-axis center position. The unit is arcsec for spatial element shapes other than squares, and in that later case the unit is number of spatial element (sides), where [0, 0] is at the lower left of the IFU. The default is to center the PSF on the IFU. The default value is: centered
	psf_xy_correl		- A scalar decimal-type value that specifies an angle for elliptical PSFs; 0 <= PSF_XY_CORREL <= 1.0. The default value is: 0.0
	psf_spectrum		Contains either a scalar string with the name of a plain-text or FITS file with a spectrum of a single source, or a decimal-type array with emission line wavelengths (which fluxes are specified using PSF_SCALE). In either case, the spectrum is interpolated to the wavelength array used to create the raw data.
	psf_catalog		A scalar string that is passed on as the keyword CATALOG to the routine that reads the spectrum file, p3d_misc_fluxcal_read_file_standard_star.
	psf_fluxunit		A scalar string that is passed on as the keyword FLUXUNIT to the routine that reads the spectrum file, p3d_misc_fluxcal_read_file_standard_star.
	psf_waveunit		A scalar string that is passed on as the keyword WAVEUNIT to the routine that reads the spectrum file, p3d_misc_fluxcal_read_file_standard_star.
	psf_scale		A scalar or array of decimal values that is used as follows depending on how PSF_SPECTRUM is specified: The provided data is a plain-text or a binary table FITS file. The median value of the spectrum is scaled to the value of PSF_SCALE. The default behavior is to scale the object spectrum so that the median value equals 10000 e-. The provided data are emission line wavelengths. The intensity of each wavelength is specified with the values in PSF_SCALE, which must have the same number of wavelengths. The default value is: 10000.0
	cont_spectrum		A scalar string that contains either the name of a plain-text or FITS file with a continuum spectrum that is used to create the raw continuum image (that is then used to trace the spectra in the reduction step). The default is otherwise to use a constant value for the spectrum (at the value of CONT_SCALE). The default value is: constant continuum at 10000.0 e-
	cont_catalog		A scalar string that is passed on as the keyword CATALOG to the routine that reads the continuum spectrum file, p3d_misc_fluxcal_read_file_standard_star.
	cont_fluxunit		A scalar string that is passed on as the keyword FLUXUNIT to the routine that reads the continuum spectrum file, p3d_misc_fluxcal_read_file_standard_star.
	cont_waveunit		A scalar string that is passed on as the keyword WAVEUNIT to the routine that reads the continuum spectrum file, p3d_misc_fluxcal_read_file_standard_star.
	cont_scale		A scalar or array of decimal values that is used as follows. The provided continuum spectrum data are a plain-text or a binary-table FITS file. The median value of the continuum spectrum is scaled to the value of CONT_SCALE. The default behavior is to scale the spectrum so that the median value equals 10000 e-. The default value is: 10000.0
	arclines		Must contain either a scalar string with the name of a plain-text file with two columns (wavelengths in Angstrom and intensities in counts) or a decimal-type array with emission line wavelengths (which intensities are specified using ARC_SCALE). In either case, the emission lines are interpolated to the wavelength array used to create the raw data arc spectrum.
	arc_scale		An array of decimal values that are used as arc-line intensities with the values in ARCLINES when that keyword is set to a list of wavelengths.
	spnum		An optional keyword that allows to set the number of spectra freely; this keyword is only used when the instrument parameter 'spnum_free' is set to "yes" and it is set to an integer with a value between the minimum and maximum allowed numbers specified in the instrument parameter file ('spnum_min' and 'spnum_max', respectively). The use of this keyword overrides the instrument parameter 'spnum'. Note! It is probably wise to also set the keywords DIST and GAPSNUM when this keyword is used.
	dist		An optional keyword that allows to set the value of the constant separation between spectra. The use of this keyword overrides the instrument parameter 'dist_tr'.
	gapsnum		An optional keyword that allows to set the constant number of spectra in a group of spectra freely (GAPSNUM[0]; value >= 1), the optional constant number of "empty" spectrum gaps is set with the second value (GAPSNUM[1]; value >= 1), and the optional starting pixel on the cross-dispersion axis (GAPSNUM[2]; value >= 0.0). The first two values are used as integers and the third value is a decimal value. The use of this keyword overrides the instrument parameter 'gapsfile'.
	aberration_fwhm		A scalar decimal value that specifies the FWHM of a Gaussian profile that is used to convolve the simulated images after they are created; this procedure simulates effects of spherical aberration. The allowed range of values is 0.5 <= ABERRATION_FWHM <= 5.0. The feature is deactivated when aberration_FWHM == 0.0. The default is to use the value of the instrument parameter 'aberration_fwhm', when such a value is available.
	subsample		A scalar integer that specifies how many pixels there are in the fine-resolution image per spatial element (square-shaped elements) or per main dimension (circular and hexagon-shaped elements). The default value is 10, and SUBSAMPLE >= 1. The default value is: 10
	kernel		A scalar string with the name of a fits file, which shall contain a two- to four-dimensional array with sets of two-dimensional kernels that are used across both the dispersion axis and the cross-dispersion axis of the CCD for multiple wavelengths and one or all spectra. The kernel data are provided in the DATA extension. Each kernel is specified as a two-dimensional array in the first two dimensions; the dispersion (cross-dispersion) axis is specified with the header keyword AXISDISP (AXISSPAT). The same kernel is used at all wavelengths if the third dimension contains one element. And similarly, the same kernel is used for all spectra if the fourth dimension contains one element. Otherwise, the fourth dimension must contain as many elements as there are spectra. The center wavelength of each kernel is specified with an array of values with as many elements as there are wavelength elements in KERNEL (third dimension); this array must also be available in the same file (see below). Kernels at intermediate wavelengths are linearly interpolated from the provided kernels. Kernels at wavelengths lower (higher) than the minimum (maximum) value in the wavelength array use the respective kernel without any interpolation. The kernels are assumed to be specified at the same wavelength for all spectra. All kernels are assumed to be subsampled on the dispersion axis according to the header keyword DSUBSAMP (corresponding to the keyword DSUBSAMPLE). Similarly, the kernels are assumed to be subsampled on the cross-dispersion axis according to the header keyword CSUBSAMP (corresponding to the keyword CSUBSAMPLE). Both subsampling factors must be set to odd positive integers. The number of elements in a kernel when it is not subsampled must be odd (to have one center element). The center-wavelength array is provided in the extension named ARRAY, as a one-dimensional floating-point type array with a unique list of monotonically increasing wavelengths that specify the center wavelength of each kernel provided in KERNEL.
	zpad		A scalar integer that pads kernels provided with the kernel keyword (KERNEL). The value must be set to 0 <= ZPAD <= 10 and is multiplied with the respective subsampling factor DSUBSAMP(LE) and CSUBSAMP(LE) to get the actual padding width. The default value is: 0
	dkernel		A scalar string with the name of a fits file, which shall contain a (one-,) two- or three-dimensional array with sets of one-dimensional kernels that are used across the dispersion axis of the CCD for multiple wavelengths and one or all spectra. Each kernel is specified as a one-dimensional array in the first dimension. The same kernel is used at all wavelengths if DKERNEL is specified as a one-dimensional array or if the second dimension contains one element. The same approach applies to the third dimension, which could be left out altogether to use the same kernel with all spectra. Otherwise, the third dimension must contain as many elements as there are spectra. Multiple kernels can be specified for a single spectrum by stacking them in the second dimension. The center wavelength of each kernel is specified with an array of values with as many elements as there are wavelength elements in DKERNEL (second dimension); this array must also be available in the same file (see below). Kernels at intermediate wavelengths are linearly interpolated from the provided kernels. Kernels at wavelengths lower (higher) than the minimum (maximum) value in the wavelength array use the respective kernel without any interpolation. The kernels are assumed to be specified at the same wavelength for all spectra. The kernel are provided in the DATA extension. All kernels are assumed to be subsampled according to the header keyword DSUBSAMP (which must be a positive odd integer) and the number of elements in a kernel when it is not subsampled must be odd (to have one center element). The center-wavelength array is provided in the extension named ARRAY, as a one-dimensional floating-point type array with a unique list of monotonically increasing wavelengths that specify the center wavelength of each kernel provided in DKERNEL. Note: DKERNEL is only used if KERNEL is unset.
	dzpad		A scalar integer that pads kernels provided with the kernel keyword (DKERNEL) as well as default kernels with zeroes. The value must be set to 0 <= DZPAD <= 10 and is multiplied with the subsampling factor DSUBSAMP(LE) to get the actual padding width. The default value is: 0
	dsubsample		A scalar integer that specifies the subsampling factor used on the dispersion axis; the value must be odd and greater than or equal to 1. Note: DSUBSAMPLE is only used with the default profiles when both KERNEL and DKERNEL are unset. The default value is: 1
	dispmask		A scalar string with the name of an existing file that contains a dispersion mask. The data must be available in the PARAMS or zeroth extension as a two-dimensional decimal-value array that contains polynomial coefficients for each spectrum; for each spectrum, the polynomial converts pixel values (x) to wavelengths (y). The dispersion mask must be setup to use the same pixel subsampling as is used with DSUBSAMPLE or DKERNEL. The first pixel (index 0) must be the wavelength of the lower range minus half the kernel width (DKERNEL). The last pixel must extend to – at least – the last pixel of the selected wavelength range, plus half the kernel width (DKERNEL). As an example, assuming the CCD is 2000 px wide on the dispersion axis and the dispersion kernel is 31px wide, the dispersion mask must be setup using (31 – 1) / 2 + 2000 + (31 – 1) / 2 = 2030 px. Hence, the first pixel in the dispersion mask is used with the lowest pixel in the kernel when the kernel is centered on the first pixel in the CCD image. Note: It is important that the dispersion axis in the dispersion mask is the same as in the simulated data! Note! The keyword DISPOFFSET is ignored if this keyword is used.
	dispoffset		An array with decimal values that specifies a pixel offset for the first pixel on the dispersion axis for each spectrum; this keyword is easier to use than DISPMASK for cases where there is a pixel offset and all spectrum pixels have the same wavelength extent. This keyword is ignored when DISPMASK is set. The values of the array are the coefficients of a polynomial as used by the IDL routine POLY. For example, to create a n offset array for //n// spectra with a parabolic shape where the minimum offset is at //n*2/3// and the offset is //sa0// pixels, use the following array: x = long(n * 2d0 / 3d0) DISPOFFSET = – [1d0, -2d0/x0, 1d0/x0^2] * sa0
	trpospar		A decimal array or a scalar string that defines the name of an optional FITS file that contains polynomial coefficients describing the curvature of each spectrum on the cross-dispersion axis. In case this variable is a decimal array, the values of the array are the coefficients of a polynomial as used by the IDL routine POLY. The same polynomial is used with all spectra. For example, to have a parabola centered on the dispersion axis of size L that reaches the maximum value y0 at x = 1 and x = L described as: y = (x – L/2)² / (L/2)² * y0 The coefficients array is set as: TRPOSPAR = [1.0, – 4.0 / L, 4.0 / L ^ 2] * y0 When instead using a filename, the format of the file must be the following: The data must be put in the zeroth extension in the form of a two-dimensional image. The image must have P columns and nspec rows where P is the polynomial order – 1 and nspec is the number of spectra. For each pixel on the dispersion axis, the respective cross-dispersion kernel is shifted according to the offset specified with this variable. It is important that the kernel contains enough elements to accommodate the shift without losing information (as the profile is shifted off either side). Therefore, either provide wide cross-dispersion kernels with enough zero-valued elements on the sides or zero-pad the cross-dispersion kernel using the keyword CZPAD (when using CKERNEL or the default) or ZPAD (when using KERNEL).
	ckernel		A scalar string with the name of a fits file, which shall contain a (one-,) two- or three-dimensional array with sets of one-dimensional kernels that are used across the cross-dispersion axis of the CCD for multiple wavelengths and one or all spectra. Each kernel is specified as a one-dimensional array in the first dimension. The same kernel is used at all wavelengths if CKERNEL is specified as a one-dimensional array or if the second dimension contains one element. The same approach applies to the third dimension, which could be left out altogether to use the same kernel with all spectra. Otherwise, the third dimension must contain as many elements as there are spectra. Multiple kernels can be specified for a single spectrum by stacking them in the second dimension. The center wavelength of each kernel is specified with an array of values with as many elements as there are wavelength elements in CKERNEL (second dimension); this array must also be available in the same file (see below). Kernels at intermediate wavelengths are linearly interpolated from the provided kernels. Kernels at wavelengths lower (higher) than the minimum (maximum) value in the wavelength array use the respective kernel without any interpolation. The kernels are assumed to be specified at the same wavelength for all spectra. The kernel are provided in the DATA extension. All kernels are assumed to be subsampled according to the header keyword CSUBSAMP (which must be a positive odd integer) and the number of elements in a kernel when it is not subsampled must be odd (to have one center element). The center-wavelength array is provided in the extension named ARRAY, as a one-dimensional floating-point type array with a unique list of monotonically increasing wavelengths that specify the center wavelength of each kernel provided in CKERNEL. Note: CKERNEL is only used if KERNEL is unset.
	csubsample		A scalar integer that specifies the subsampling factor used on the cross-dispersion axis; the value must be odd and greater than or equal to 1. Note: CSUBSAMPLE is only used with the default profiles when both KERNEL and CKERNEL are unset. The default value is: 1
	czpad		A scalar integer that pads kernels provided with the kernel keyword (CKERNEL) as well as default kernels with zeroes. The value must be set to 0 <= CZPAD <= 10 and is multiplied with the subsampling factor CSUBSAMP(LE) to get the actual padding width. The default value is: 0
	bias_level		A scalar decimal-value specifying the bias level that is added to the images before they are written to the output files. The default value is: 600.0
	num_obj		A scalar integer that specifies how many copies are written of the object image. The added noise is different in each copy. The default value is: 3
	num_cont		A scalar integer that specifies how many copies are written of the continuum image. The added noise is different in each copy. The default value is: 3
	num_arc		A scalar integer that specifies how many copies are written of the arc image. The added noise is different in each copy. The default value is: 1
	num_bias		A scalar integer that specifies how many copies are written of the bias image. The added noise is different in each copy. The default value is: 5
	userparfile		A scalar string with the name of a file with user- defined parameters. The following parameters are read here from this file: 'methodimcombine' see p3d_misc_imcombine.pro. 'detsec' see p3d_misc_detsec.pro.
	opath		A scalar string that specifies the path where the output images are written. The default value is: '.'
	simopfx		A scalar string that specifies a prefix for the (automatically generated) output filename. The default value is: 'p3d_sim_'
	detector		A scalar integer that specifies the currently selected detector; DETECTOR is a zero-based value. The default value is: 0
	gratingselect		A scalar string that defines the wavelength range to use out of the available ranges defined in the gratings file (see the parameter gratingsfile_dm in the instrument-parameter file). The default is to use the first wavelength range in that list. The default value is: first entry
	compress		Set this keyword to compress all output files using gzip; the file suffix '.gz' is then appended to all output filenames. The default value is: 0
	nthreads		A scalar integer that specifies how many threads to use in the convolutions. The default is to use the maximum number available, but no more than 16: MAX = !cpu.hw_ncpu < 16L. Note! This keyword is not *yet* used. The default value is: max
	logfile		This keyword specifies the name of an optional log file when p3d_csim is launched as a separate program.
	loglevel		If this routine is launched as a separate program, and LOGFILE is used, this keyword specifies the log level; [1], 2, 3 (the more output, the higher the value). The default value is: 1
	fnotify		Set this keyword to the name of a file that is written when the tool has finished; this can be a useful feature when using the p3d queue handler. In that case, set this keyword and check if the file exists before continuing. And do not forget to remove the file.
	cmdline	X	This keyword is used when the tool is launched by the p3d server.
	topwid	X	Error messages are displayed using DIALOG_MESSAGE instead of MESSAGE if this keyword is set, using this widget id as DIALOG_PARENT.
	logunit	X	Messages are saved to the file pointed to by this logical file unit if it is defined.
	verbose		Set this parameter to a positive scalar integer to make p3d write some information on STDOUT about what is going on. The following four values are acknowledged: 0 Writes no information at all; this is the default. 1 Writes the more important information; regarding subroutine configurations, mostly. 2 Writes most information; includes a more verbose output than 1. 3 Writes all information, including information on the execution state of GUI subroutines; this may be a useful mode when debugging the code.
	quiet		Set this keyword to avoid printing the disclaimer on the screen when this tool is started from the command line or from the shell.
	error	X	This scalar integer returns an error code if set. Any value different from zero means that an error has occurred.
	debug	X	An error handler is not setup when this keyword is set. Otherwise, the error handler is setup using the routine CATCH, which makes each subroutine exit quietly when a bug is encountered. Use this keyword when debugging p3d.
	cdebug		A keyword as DEBUG used with the called C routines; valid values are 0-6 where a higher number results in more output to STDERR.
	help	X	Set this keyword to show this routine documentation and then exit.

Examples:

This routine can be called in three different ways (for use on WINDOWS platforms, replace the path character '/' with '\'):

i) As a stand-alone tool from the IDL command line

All arguments and keywords but three have default options. It is necessary to specify the instrument parameter file to use, the object spectrum (see the Links section above), and the arc spectrum (see the example data section above). All other keywords are optional:

parfile = 'mosms.prm'
psf_spectrum = 'bd_28d4211_stis_001.fits' ;; just an example
arclines = 'mosms_arc_spec.dat'           ;; just an example
opath = 'Output' + path_sep()
logfile = opath + 'p3d_csim.log'
detector = 0 ;; Select the 1st wavelength range entry [gratings file]
p3d_csim, parfile, psf_spectrum=psf_spectrum, detector=detector, $
    arclines=arclines, opath=opath, logfile=logfile, /verbose

ii) As a stand-alone tool from a terminal, using the IDL VM

All arguments and keywords but three have default options. It is necessary to specify the instrument parameter file to use, the object spectrum (see the Links section above), and the arc spectrum (see the example data section above). All other keywords are optional. The recommended approach is to use the provided shell script p3d_csim_vm.sh or the Python tool p3d_dispatch.py:

# Using the shell script, without using the p3d queue system:
parfile=mosms.prm
psf_spectrum=bd_28d4211_stis_001.fits  # just an example
arclines=mosms_arc_spec.dat            # just an example
opath=Output
logfile=$opath/p3d_csim.log
detector=0    # Select the 1st wavelength range entry [gratings file]
p3d_csim_vm.sh -s $parfile psf_spectrum=${psf_spectrum} \
    arclines=${arclines} detector=$detector opath=$opath \
    logfile=$logfile /verbose

# Using the Python tool, using the p3d queue system:
parfile=mosms.prm
psf_spectrum=bd_28d4211_stis_001.fits
arclines=mosms_arc_spec.dat
opath=Output
logfile=$opath/p3d_csim.log
detector=0    # Select the 1st wavelength range entry [gratings file]
p3d_dispatch.py csim $parfile psf_spectrum=${psf_spectrum} \
    arclines=${arclines} detector=$detector opath=$opath \
    logfile=$logfile /verbose

Note: In this approach, it is necessary to specify all options using the full names of the keywords. Partially correct keyword names are not permitted, as they are inside IDL. Hence, as an example, you must type "p3d_csim_vm.sh userparfile=$userparfile" and -NOT- "p3d_csim_vm.sh user=$userparfile".

Note: The tool launches in the current directory, which is why it is unnecessary to specify full paths if files reside in the current directory.

Note: When several files are specified in the first argument, they must be separated with either a comma or a semicolon.

Note: Parameters indicated with an 'X' in the list above are unavailable in this approach.

iii) As a stand-alone tool from Python, using the IDL VM

This approach is analogous to ii), but is executed from the Python command line. Here is an example:

# Launch p3d_csim in a single IDL session, without using the
# p3d queue system:
from p3d_dispatch import *
parfile = 'mosms.prm'
psf_spectrum = 'bd_28d4211_stis_001.fits'
arclines = 'mosms_arc_spec.dat'
opath = 'Output'
detector= '0' # Select the 1st wavelength range entry [gratings file]
logfile = opath + os.sep + 'csim.log'
commandline = parfile + ' psf_spectrum=' + psf_spectrum +
  ' arclines=' + arclines + ' opath=' + opath +
  ' detector=' + detector + ' logfile=' + logfile +
  ' loglevel=2 /verbose'
p3d_dispatch('csim', commandline, noqueue=True)

Creates a trace-mask image from a set of continuum images.

Before any spectra can be extracted from a raw-data image file, it is necessary to know their locations in the image. The tracing procedure of p3d locates all the spectra, for a pre-defined instrument, and saves a trace-mask image. In this image, the position on the cross- dispersion axis, starting at the (lower) edge of the image, is stored for each spectrum. Additionally, if requested, p3d also calculates cross-dispersion line profiles for each spectrum and wavelength bin (pixel). These profiles are used to extract more accurate spectrum positions (i.e. a more accurate trace mask), and to perform optimal extraction.

It is highly recommended to use an optimal spectrum extraction method that accounts for overlapping spectra (so-called fiber-to-fiber cross talk) for most instruments. The line profiles used in such an optimal extraction are calculated by this tool, after the calculation of a first-guess trace-mask image. Because the process of calculating line profiles is computationally very expensive, for most instruments, they are only calculated if this option is activated (see the master user-parameter file). However, if p3d is able to (automatically) compile the C version of the profile-calculation routine (which it in most cases can), the associated calculations will be much faster; some factor ten seems typical on a machine with eight threads.

The output data files, consisting of the trace-mask image and the line-profiles data cube are by default written to the same file, in consecutive extensions. They are instead written to separate files if the keyword ALLINONE, or the user-parameter 'allinone', is unset. There is no real advantage to either approach; although, if separate files are used and the line-profiles data cube file is renamed, it will not be found by other tools.

The raw images should be continuum images, it is otherwise difficult - or even impossible – to trace the spectra in regions where there is no signal, or where there are very densely packed emission lines. Either a lamp continuum or a twilight flat (if available, and appropriate) could be used as an input image.

Please, see the p3d A&A paper as well as the master user-parameter file and the p3d documentation WIKI, for help on how to setup the parameters so that spectra in your data are located and traced properly.

This routine calls the routine p3d_cr, if requested to do so, which cleans single input images of cosmic-ray hits; see the CRCLEAN keyword. The outcome of the cleaning procedure depends both on properties of the input image as well as the algorithm parameters. Different parameters might produce a different outcome. There is no guarantee whatsoever that the cleaning removes only cosmic-ray hits. Check the output images to build your own opinion regarding the quality of the outcome. Also, see p3d_cr for more details.

General information on how to learn more about this routine

The nearly, and mostly, full automatic tracing procedure is described in detail in the p3d A&A paper: Sandin et al. 2010, A&A, 515, 35.

Instrument parameter files (*.prm), which are required to run this routine, are found in the directory "!p3d_path/data/instruments/*/" (outside of IDL, check "${p3d_path}/data/instruments/*/").

A default (master) user-parameter file is available in "!p3d_path/data/master_userparfile.dat". Copy this file to your data-reduction output directory, rename the file to, for example, "user_p3d.dat" and edit it to suit your needs.

Examples on how to use this tool are provided in the EXAMPLES section below. Also see the tutorials in the directory "!p3d_path/doc/".

See the file README in the root of the p3d package for installation instructions, if p3d is not already installed on the computer.

Additional information, documentation, new versions, etc., can be found at the p3d project web site "https://p3d.sourceforge.io".

Requirements and format of the input and the output data

p3d only works with FITS-formatted images, i.e. fits files. Such files have a header that contains additional information about the observing conditions, the telescope, the instrument setup, and other useful information. Both input and output fits files can be compressed using gzip, to save storage space (also on Windows platforms); p3d handles such compressed files transparently.

This routine takes one, or several, continuum-data raw images as input. The images are combined (by default), but p3d can also calculate a trace-mask image for each file if combineimages_tr==yes (in the user-parameter file). Prescan and overscan regions must not be removed in the input data, they are removed by p3d during the reduction. Input files of instruments using several readout ports must also not be combined in advance, p3d combines the images automatically.

The output – the trace-mask image – consists of an image with as many pixels on the dispersion axis as there are pixels on the dispersion axis in the input image(s) – prescan and overscan regions are (mostly) kept. On the cross-dispersion axis, there are n rows, one for each spectrum. The number of rows in the output file is always the same, pre-defined, number. For each spectrum and pixel on the dispersion axis, the trace-mask image contains the distance to the edge of the CCD. In addition to the standard fits-file header keywords, the extracted spectrum images contain additional p3d-specific keywords:

The line-profiles data cube is written either to the PROF extension of the trace-mask image or to a separate file. Its header contains the following set of additional p3d-specific keywords:

If several input files were specified, which were thereafter combined by p3d, before calculating a trace-mask image, the combined image contains the following p3d-specific header keywords:

Saved, and optionally already combined, bias-subtracted raw-data images contain only one additional header keyword:

Input parameters:

	filename		A scalar string that specifies the name of the raw data file to be used when tracing spectra.
	parfile		A scalar string that specifies the name of a file with instrument-specific setup parameters. These files follow a strict format. Already properly formatted files are available for all configured instruments in the directories p3d_path/data/instruments/*/. It is sufficient to specify the file basename, without the full path of the p3d subdirectory; for example: "nvimos_hr.prm".

Keyword parameters:

	masterbias		A scalar string with the name of a master-bias image file. This file is required if optimal extraction is used and none of the following four keywords is used: BIASPX, BIASPY, BIASOX, BIASOY. Note! This keyword must not be set if the prescan and the overscan regions should be used to estimate the bias level.
	biaspx		Set this keyword to use the x-axis prescan region when subtracting a bias. Specifically, a median value is calculated for each pixel on the y axis in the x-axis prescan region. The same array is used with all y-axis columns of bias subtracted data.
	biaspy		Set this keyword to use the y-axis prescan region when subtracting a bias. Specifically, a median value is calculated for each pixel on the x axis in the y-axis prescan region. The same array is used with all x-axis rows of bias subtracted data.
	biasox		Set this keyword to use the x-axis overscan region when subtracting a bias. Specifically, a median value is calculated for each pixel on the y axis in the x-axis overscan region. The same array is used with all y-axis columns of bias subtracted data.
	biasoy		Set this keyword to use the y-axis overscan region when subtracting a bias. Specifically, a median value is calculated for each pixel on the x axis in the y-axis overscan region. The same array is used with all x-axis rows of bias subtracted data.
	biasconstant		This keyword can be used in two ways: Set this keyword to use a constant value as a bias value; otherwise an array is used. All prescan and overscan regions specified using the keywords BIASPX, BIASPY, BIASOX, and BIASOY can be used when this keyword is set; otherwise only one of them. If this keyword is set to a double type array or to a comma-separated string with values > 1 (when called from the shell), these values are used to create a bias; the number of elements must be the same as the number of blocks.
	savebiassub		Set this keyword to write the bias-subtracted raw-data image before spectrum extraction. The name of the output file is the same as the (combined) input file, with the added intermediate suffix '_subbias'. Also see the user parameter 'savebiassub'.
	userparfile		A scalar string with the name of a file with user- defined parameters. The following parameters are read here from this file: 'keywordsfile' The name of the file with instrument-specific header keywords. 'combineimages_tr' To specify if input images are to be combined or not. Corresponds to IMCOMBINE. 'compress' Corresponds to COMPRESS. 'exmonitor' Corresponds to EXMONITOR. 'a2' To use A3 instead of A4. 'a3' To use A3 instead of A4. 'a5' To use A3 instead of A4. 'letter' To use US Letter instead of A4 with PostScript and PDF output. 'legal' To use US Letter instead of A4 with PostScript and PDF output. 'tabloid' To use US Tabloid instead of A4 with PostScript and PDF output. 'psusletter' To use US Letter instead of A4 with PostScript and PDF output. 'psnumwave' To select the total number of of wavelengths where a plot of the line profiles is made. 'detector_sepgrp' Corresponds to DETSEPGRP. 'detector_sepnum' Corresponds to DETSEPNUM. 'opfx' Corresponds to OPFX. 'sfx' Corresponds to SFX. 'icsfx' To specify the data output file intermediate suffix of combined files. 'trsfx' To specify the data output file intermediate suffix of the created trace-mask image. 'methimcombine_tr' To specify the method that is used to combine the input images. 'allinone' Corresponds to ALLINONE. 'nthreads_cr' Corresponds to CRNTHREADS. 'nthreads' Corresponds to NTHREADS. 'crclean' Corresponds to CRCLEAN. 'biaspx' Corresponds to BIASPX. 'biaspy' Corresponds to BIASPY. 'biasox' Corresponds to BIASOX. 'biasoy' Corresponds to BIASOY. 'biasconstant' Corresponds to BIASCONSTANT. 'savebiassub' Corresponds to SAVEBIASSUB. 'gapfile' To read the sprectrum gaps file. Additionally, the following user parameters are used by p3d_cr: 'laimagemethod' Corresponds to IMAGEMETHOD. 'laimageclean' Corresponds to IMAGECLEAN. 'ladispmedian' Corresponds to DISPMEDIAN. 'crwriteall' Corresponds to WRITEALL. 'nocrc' Corresponds to NOCRC. 'sigclip' Corresponds to SIGCLIP. 'objlim' Corresponds to OBJLIM. 'ratlim' Corresponds to RATLIM. 'crfwhm' Corresponds to FWHM. 'crgausskernelsize' Corresponds to GAUSSKERNELSIZE. 'fwhm_tr' Used to set the width of the Gaussian profile when crfwhm is unavailable and IMAGEMETHOD is unset. 'sigfrac' Corresponds to SIGFRAC. 'growradius' Corresponds to GROWRADIUS. 'maxiter' Corresponds to MAXITER. 'cleansfx' Corresponds to CLEANSFX. 'masksfx' Corresponds to MASKSFX. 'laplsfx' Corresponds to LAPLSFX. 'poflsfx' Corresponds to POFLSFX. 'siprsfx' Corresponds to SIPRSFX. 'laprsfx' Corresponds to LAPRSFX. 'finesfx' Corresponds to FINESFX. 'wallsfx' Corresponds to WALLSFX. Note! If any of the corresponding keywords is specified, that value takes precedence.
	ofilename	X	This keyword returns the full name of the created trace-mask file.
	opath		A scalar string that specifies the path, where the output data are saved. The default value is: '.'
	opfx		A scalar string that specifies a prefix for the (automatically generated) output filename. The default value is: ''
	sfx		A scalar string that specifies the raw data filename suffix to use. The default value is: .fits
	detector		A scalar integer that specifies the currently selected detector; DETECTOR is a zero-based value. The default value is: 0
	exmonitor		If this keyword is set, a trace viewer is shown after all the tracing and the line-profile fitting is done when optimal extraction is used. The default value is: 1
	imcombine		Set this keyword to combine all input images before locating the spectra. The default value is: 1
	compress		Set this keyword to compress all output files using gzip; the file suffix '.gz' is then appended to all output filenames. The default value is: 0
	crnthreads		A scalar integer that specifies how many threads used when cleaning single input images of mic-ray hits. If this keyword is not set, READS is used instead. If nthreads was not set her, the default is to use the maximum number of ilable threads minus one, but the maximum number 16: (!cpu.hw_ncpu – 1) < 16. The default value is: max-1
	nthreads		A scalar integer that specifies how many threads to use in the parallelized line profile calculations. The calculation speed scales nearly linearly with the number of threads. The default is to use the maximum number available, but no more than 16: max = !cpu.hw_ncpu – 16. The default value is: max
	logfile		This keyword specifies the name of an optional log file when p3d_ctrace is launched as a separate program.
	loglevel		If this routine is launched as a separate program, and LOGFILE is used, then this keyword specifies the log level; [1],2,3 (the higher the value, the more output). The default value is: 1
	detsepgrp		(Profile fitting) This keyword can be set in instrument setups where there are more than one detector on the spatial axis, and the background levels differ (making a line-profile fit tricky). The keyword should in this case be set to a positive non-zero integer that indicates beyond which spectrum the new detector begins. This value can also be set with the (instrument- or) user-parameter 'detector_sepgrp'. If this keyword, or the user-parameter is set, it is also mandatory to set DETSEPNUM.
	detsepnum		(Profile fitting) Together with DETSEPGRP, this value specifies the number of spectra in the group that falls within the first detector; the value must be a positive non-zero integer. If DETSEPGRP is set, it is also mandatory to set this value. This value can also be set with the (instrument- or) user-parameter 'detector_sepnum'.
	gui	X	If this keyword is set, the routine is assumed to be called from the p3d GUI – meaning that there is no initialization of the p3d system variables.
	subroutine	X	If set, parts of the routine opening message will not be written to the log file.
	cinv		Set this keyword to plot white lines on a black background instead of the default, which is to plot black lines on a white background.
	allinone		Unset this keyword to write the output data to separate files, instead of writing the trace-mask image to the DATA extension and the line-profiles image to the PROF extension. The default value is: 1
	crclean		Set this keyword to clean all uncombined input images of cosmic-ray hits before the tracing procedure is started; either a master-bias image or prescan and overscan regions must be used to provide the bias level to use this functionality. The following keywords are used to configure the cosmic-ray cleaning procedure; they are only used if CRCLEAN is set; see p3d_cr and p3d_tool_crays_lapycosmic for more details:
	sigclip		A scalar decimal value that specifies the clipping sigma.
	objlim		A scalar decimal value that specifies the object limit value; this value is only used if IMAGEMETHOD is set.
	ratlim		A scalar decimal value that specifies the ratio limit value; this value is only used if IMAGEMETHOD is unset.
	crfwhm		A decimal scalar value, or a two-element array, that specifies the full width at half maximum of the gaussian kernel used to convolve the raw data with. This keyword corresponds to FWHM in p3d_cr. This value is only used if IMAGEMETHOD is unset.
	gausskernelsize		A scalar integer that specifies the size of the Gaussian kernel used when convolving the input image with a Gaussian kernel using the PyCosmic algorithm. This value is by default set to 2 * q + 1, where q is the maximum value of the x and the y pixel where a one-dimensional Gaussian reaches 15% of its maximum value (1.0); this condition makes use of the full-width-at-half-maximum value of FWHM. The value must be odd, and the maximum value is in any case 19. Be aware that convolution times increase strongly with the size of the Gaussian kernel.
	sigfrac		A scalar decimal value that specifies a scale factor for pixels surrounding a cosmic-ray masked pixel. This parameter is only used if GROWRADIUS is non-zero. The default value is: 0.5
	growradius		Set this keyword to grow all cosmic-ray hit pixels by a frame this many pixels wide; this is done to check if the surrounding pixels are affected by cosmic-ray hits as well. The default value is 1, the values 0 and 2 are also accepted. Surrounding pixels are checked if they are cosmic-ray affected using the first criterion S' > SIGFRAC * SIGCLIP. The default value is: 1
	maxiter		A scalar integer value that specifies the maximum number of iterations used when identifying cosmic-ray hits in each image.
	imagemethod		Set this keyword to use the original algorithm of P. van Dokkum (L.A. Cosmic), which was developed for cosmic-ray detection in images. The default is otherwise to use the IFS-modified algorithm with IMAGEMETHOD unset.
	imageclean		Set this keyword to use the original two-dimensional median filter when replacing the flux in masked pixels with that of the surrounding pixels. Otherwise an [11 pixel] one-dimensional median filter is used. The default value is: 1
	dispmedian		Set this keyword to a positive integer > 1 to subtract sampling flux along the dispersion axis. The value defines the width of the median window.
	writeall		Set this keyword to write all intermediate images in addition to the cleaned image and the cosmic-ray mask.
	showcrgui		Set this keyword to show a properties GUI of the cosmic-ray cleaning parameters before the algorithm is executed.
	nocrc		Set this keyword to force the use of the slower IDL-based cosmic-ray cleaning routine instead of the faster C-based routine.
	fnotify		Set this keyword to the name of a file that is written when the tool has finished; this can be a useful feature when using the p3d queue handler. In that case, set this keyword and check if the file exists before continuing. And do not forget to remove the file.
	font		Set this scalar string to the name of the font to be used with all widget components of this tool. It is advised to use a fixed font, the outcome is otherwise difficult to predict. Set the keyword to 'big' to use the following pre-defined font ed-medium-r-normal--18-120-100-100-c-90-iso8859-1'; this could be useful on a system using a 4k screen (with 3160x2560 pixels). Another option to specify a font is to set the environment variable p3d_font or the p3d IDL system variable !p3d_font. The default font used with the widgets is, for your information, printed to the screen when FONT is unset and the highest verbosity level is used (verbose = 3). On Linux machines, use a shell command such as 'xlsfonts' to list all available fonts.
	cmdline	X	This keyword is used then the tool is launched by the p3d server.
	stawid	X	If set, various messages are written to the p3d GUI status line (this must be the widget id of that label widget).
	topwid	X	If set, error messages are displayed using DIALOG_MESSAGE, using this widget id as DIALOG_PARENT, instead of MESSAGE.
	logunit	X	Messages are saved to the file pointed to by this logical file unit, if it is defined. This keyword must not be used if this routine is called as a script instead of using the GUI.
	verbose		Set this parameter to a positive scalar integer to make p3d write some information on STDOUT about what is going on. The following four values are acknowledged: 0 Writes no information at all. This is the default. 1 Writes the more important information; regarding subroutine configurations, mostly. 2 Writes most information; includes a more verbose output than 1. 3 Writes all information, including information on the execution state of GUI subroutines. This may be a useful mode when debugging the code.
	quiet		Set this keyword to avoid printing the disclaimer on the screen when this tool is started from the command line or from the shell.
	error	X	This scalar integer returns an error code if set. Any value different from zero means that an error has occurred.
	debug	X	An error handler is not setup if this keyword is set. Otherwise, the error handler is setup using the routine CATCH, which makes each subroutine exit quietly when a bug is encountered. Use this keyword when debugging p3d.
	cdebug		A keyword as DEBUG used with the called C routines; valid values are 0-6 where a higher number results in more output to STDERR.
	help	X	Set this keyword to show this routine documentation and then exit.

Output parameters:

out

X

Upon a successful execution, this variable contains the trace mask (data) when the routine exits.

Examples:

This routine can be called in four different ways (for use on WINDOWS platforms, replace the path character '/' with '\'):

i) Using the p3d GUI

There is no need to setup the input parameters, p3d handles this transparently.

ii) As a stand-alone tool from the IDL command line

In addition to your input continuum images, it is necessary to pass the name of the (provided) instrument-specific parameter file, the master-bias image file or (at least) one of the bias prescan and overscan parameters (if you use optimal extraction), as well as any additional arguments to p3d.

files = ['tracefile_1.fits.gz', 'tracefile_2.fits.gz']
parfile = 'larr2k.prm'
opath = 'Output'
masterbias = opath + '/masterbias.fits.gz'
userparfile = opath + '/user_p3d.dat' ;; If you use this file
detector = 0 ;; Required with multi-detector configurations.
p3d_ctrace, files, parfile, masterbias=masterbias, $
    userparfile=userparfile, opath=opath, detector=detector, $
    logfile=opath+'dred.log', /verbose

iii) As a stand-alone tool from a terminal, using the IDL VM

In addition to your input continuum images, it is necessary to pass the name of the (provided) instrument-specific parameter file, the master-bias image file or (at least) one of the bias prescan and overscan parameters (if you use optimal extraction), as well as any additional arguments to p3d. The recommended approach is to use the provided shell script p3d_ctrace_vm.sh or the Python tool p3d_dispatch.py.

# Using the shell script, making use of the p3d queue system:
parfile=larr2k.prm
opath=Output
masterbias=${opath}/masterbias.fits.gz
userparfile=${opath}/user_p3d.dat # If you use this file
detector=0 # Required with multi-detector configurations.
p3d_ctrace_vm.sh tracefile_1.fits.gz,tracefilefile_2.fits.gz \
    $parfile masterbias=$masterbias userparfile=$userparfile \
    opath=$opath detector=$detector logfile=$opath/dred.log /verbose

# Using the Python tool, without using the p3d queue system:
parfile=larr2k.prm
opath=Output
masterbias=${opath}/masterbias.fits.gz
userparfile=${opath}/user_p3d.dat # If you use this file
detector=0 # Required with multi-detector configurations.
p3d_dispatch.py --noqueue ctrace \
    tracefile_1.fits.gz,tracefilefile_2.fits.gz \
    $parfile masterbias=$masterbias userparfile=$userparfile \
    opath=$opath detector=$detector logfile=$opath/dred.log /verbose

Note: In this approach, it is necessary to specify all options using the full names of the keywords. Partially correct keyword names are not permitted, as they are inside IDL. Hence, as an example, you must type "p3d_ctrace_vm.sh masterbias=$masterbias" and -NOT- "p3d_ctrace_vm.sh ma=$masterbias".

Note: The tool launches in the current directory, which is why it is unnecessary to specify full paths if files reside in the current directory.

Note: When several files are specified in the first argument, they must be separated with either a comma or a semicolon.

Note: Parameters indicated with an 'X' in the list above are unavailable in this approach. This includes the two output variable parameters OFILENAME, and OUT.

iv) As a stand-alone tool from Python, using the IDL VM

This approach is analogous to iii), but is executed from the Python command line. Here is an example:

# Launch p3d_ctrace in a single IDL session, without using the
# p3d queue system:
from p3d_dispatch import *
files = 'tracefile_1.fits.gz,tracefilefile_2.fits.gz'
parfile = 'larr2k.prm'
opath = 'Output'
userparfile = opath + '/user_p3d.dat' # If you use this file
detector = 0 # Required with multi-detector configurations.
masterbias = opath + '/masterbias.fits.gz'
commandline = files + ' ' + parfile + ' masterbias=' + masterbias +
  ' userparfile=' + userparfile + ' opath=' + opath +
  ' detector=' + detector +
  ' logfile=' + opath + '/dred.log /verbose'
p3d_dispatch('ctrace', commandline, noqueue=True)

Combines four extracted VIMOS spectrum images of the four separate detectors to create one RSS file for the entire field.

The purpose of this tool is to combine extracted science-object images. The four input images should all have been reduced separately, they should belong to the same observation block (OB), and they should preferrably all have been flux calibrated. Before the four input images are combined, the files – including the corresponding error files, output bad-pixels mask files, and saturated-mask files – are checked in the following ways:

The flux level varies across the four detectors, which is why data are "re-normalized" by default to get the same flux in a selected telluric line; the re-normalization is switched off when the NORENORMALIZE keyword is set. See the next subsection for details of this re-normalization procedure.

Spectra of the combined image are by default re-arranged to go from south to north, and from west to east – the resulting images should use the "vimos_positions_rer.dat" fiber position-table file. Set the KEEPARRANGEMENT keyword if the wish is instead to just concatenate the data, without re-arranging them – the resulting images should in this case use the "vimos_positions_coc.dat" fiber position-table file.

The output-data files, consisting of the combined spectra, the corresponding combined error image, the combined bad-pixels mask image, the combined saturated-pixels mask image, the combined data-quality mask image, and the re-normalization correction factor array (if it is used), are by default written to the same file, in consecutive extensions. They are instead written to separate files if the keyword ALLINONE, or the user parameter 'allinone', is unset. There is no real advantage to either approach; although, if separate files are used and one or more of the error image file, the output bad-pixels mask image file, the saturated-pixels mask image file, or the re-normalization correction factor file are renamed, they will not be found by other tools.

Re-normalizing data

Spectra are normalized anew to account for imprecise flux calibration between the four detectors, as well as other issues that are difficult to handle. Without such an additional normalization, differences detector-to-detector are seen clearly in spatial maps of the combined image between the four separate quadrants.

The re-normalization procedure integrates the flux in one selected telluric line across all working fibers. Thereafter, all spectra are multiplied with the correction factor: median_intensity / intensity of each individual spatial element.

The following four subsections discuss all options and issues considered in the re-normalization procedure.

Spectra of dead and unused fibers and low-transmission fibers

Only fully working fibers (spatial elements) are considered in the re-normalization procedure. Information about dead and unused fibers, as well as low-transmission fibers, is retrieved using the 'deadfibersfile' parameters in the instrument-parameter file.

Selecting the telluric line

By default, p3d selects the telluric line from the default line-list file (p3d_path/data/tables/linelists/telluric_lines_lores.dat), where the strongest line is used (due to a simplified measurement, it is possible that the second strongest line is selected instead). Any other line-list file can be selected using the TELLURICLINE keyword or the user parameter 'telluricline'. However, to ensure that data are normalized with a telluric line that does not contain any emission from the object, it is instead recommended to specify the wavelength of a pre-selected emission line, using the same keyword TELLURICLINE (using the unit Angstrom [Å]). Nlended lines are not handled.

Setting the normalization intensity

The normalization of the intensity in the telluric line is by default calculated as the maximum value of the median intensity of the individual detectors; this is the median_intensity value above. Use a different value with the keyword MAXINTENSITY or the user parameter 'maxintensity' if you are able to determine this value more accurately by some other means.

Normalizing spectrum to spectrum or per quadrant

Also by default, all spectra are normalized individually. If you, for some reason, instead want to normalize all spectra of one quadrant with the median intensity in that quadrant, set the keyword ONLYNORMQUADRANT or the user parameter 'onlynormquadrant'.

Non flux-calibrated data

Data that are not flux calibrated are by default treated in the same way as flux-calibrated data. There is the option, however, to normalize spectra per quadrant using a total value of the mean flat-field image spectrum. The mean value must be available in the FFLMN keyword of the object spectra; this value is present in p3d-reduced data. Set the NOFCDGOODNORM keyword or the user parameter 'nofcdgoodnorm' to use this functionality.

General information on how to learn more about this routine

The image combination procedure is described in more detail on the p3d documentation WIKI (see below).

Instrument parameter files (*.prm), which are required to run this routine, are found in the directory "!p3d_path/data/instruments/*/" (outside of IDL, check "${p3d_path}/data/instruments/*/").

A default (master) user-parameter file is available in "!p3d_path/data/master_userparfile.dat". Copy this file to your data-reduction output directory, rename the file to, for example, "user_p3d.dat" and edit it to suit your needs.

Examples on how to use this routine are provided in the EXAMPLES section below.

See the file README in the root of the p3d package for installation instructions, if p3d is not already installed on the computer.

Additional information, documentation, new versions, etc., can be found at the p3d project web site "https://p3d.sourceforge.io".

Requirements and format of the input and the output data

p3d only works with FITS-formatted images, i.e. fits files. Such files have a header that contains additional information about the observing conditions, the telescope, the instrument setup, and other useful information. Both input and output fits files can be compressed using gzip, to save storage space (also on Windows platforms); p3d handles such compressed files transparently.

The output file always contains the sum of the pre-defined number of extracted spectra in each input image (with as many pixels on the cross-dispersion axis), which are wavelength calibrated and normalized with a flat-field image since earlier – see p3d_cobjex. Data are stored in the so-called row-stacked spectrum (RSS) format. To create a spatial map of the intensities at any wavelength, using this RSS file, it is necessary to make use of a fiber position-table file; depending on your setup, you should use either "vimos_positions_rer.dat" ([KEEPARRANGEMENT = 0]) or "vimos_positions_coc.dat" (KEEPARRANGEMENT = 1). Extracted data can also can also be saved using the Euro3D format (use the keyword SAVEE3D or the user parameter 'savee3d'), and data of instruments using square-shaped spatial elements are easily converted to a cube format using the tool p3d_rss2cube. In addition to the standard fits-file header keywords, the extracted spectrum images contain additional p3d-specific keywords:

The re-normalization correction-factor binary-table extension, or file, contains two additional p3d-specific keywords:

The name of the extracted flat-field image file. Note! This keyword is only written if ALLINONE is unset.

Input parameters:

	filenames		A four-element string array that specifies the names of the extracted data files to combine. The array must contain one file per quadrant. The files can be in any order, the routine figures out by itself which quadrant each file belongs to.
	parfile		A scalar string that specifies the name of a file with instrument-specific setup parameters. These files follow a strict format. Already properly formatted files are available for all configured instruments in the directories p3d_path/data/instruments/*/. It is sufficient to specify the file basename, without the full path of the p3d subdirectory; for example: "nvimos_hr.prm".

Keyword parameters:

	dfilenames		A four-element string array that specifies the names of the extracted error data files to combine. The array must contain one file per quadrant. The files must be specified in the same order as in FILENAMES. If DFILENAMES is not provided, the extracted error-data filenames are taken from the keyword IMERR in the data header of FILENAMES.
	keeparrangement		If this keyword is set, spectrum data are simply combined (concatenated). The default behavior is otherwise to re-order them S->N, and W->E.
	norenormalize		No data re-normalization is performed if this keyword is set.
	nofcdgoodnorm		Set this keyword to re-normalize non flux-calibrated data using a renormalized gain value. The default is otherwise to use a telluric line in the same fashion as with flux-calibrated data.
	onlynormquadrant		With flux calibrated data, the re-normalization is y default performed on individual spectra (of the et of fully working fibers). Set this parameter to nstead perform the re-normalization with the mean ntensity value of a telluric line in each uadrant.
	telluricline		The re-normalization procedure of flux-calibrated data uses a telluric line, which is by default selected automatically in the high-resolution telluric-line list file ("telluric_lines_hires.dat"). Alternatively, set this parameter either to the name of any existing telluric line-list file, or to the wavelength of the particular line to use (the unit is Angstrom [Å]); example: TELLURICLINE = 5577. Note! The value must be within the wavelength range of the wavelength-calibrated and flux-calibrated spectrum image. Note! The exact wavelength is unimportant, but the value should be accurate enough that the line can be properly located by the fitting routine, i.e. the line center should be within a few pixels distance [CDELT] of the chosen value. Note! Ensure a telluric line is used that shows no signs of object emission. Otherwise, data quality is compromised. The p3d spectrum viewer p3d_sv could be used to check which line to use. Note! If this keyword is used with a filename, i.e. a string, the following search order of the path applies: i. The full path of an existing file must be specified if the string contains one or several path separators "/" or "\". ii. A file without any path separator is searched for in the user-parameter file directory, if such a file is used. iii. A file without any path separator is searched for in the p3d default line-list directory: data/tables/linelists/.
	maxintensity		A scalar floating-point value that specifies the intensity used to normalize all spectra. The default behavior is, when this value is unset, to instead first calculate a median intensity of the selected telluric line for each detector individually. Thereafter, MAXINTENSITY is set to the maximum value of the four median values.
	corrffile		A scalar string with the name of a file that contains the re-normalization correction factor array. The correction-factor array of this file is used if this parameter is set, otherwise a new array is calculated here. The format of this binary-table fits file is that it contains an array with four columns and 400 rows. Note! The following file search order of the path applies: i. The full path of an existing file must be specified if the string contains one or several path separators "/" or "\". ii. A file without any path separator is searched for in the same directory as (FILENAMES). iii. If the file was not found in the same directory as FILENAME, the parent directory is used instead.
	compress		Set this keyword to compress all output files using gzip; the file suffix '.gz' is appended to all output filenames. The default value is: 0
	savee3d		The output data are saved in the Euro3D (e3d) format in addition to the usual RSS format when this keyword is set.
	userparfile		A scalar string with the name of a file with user-defined parameters. The following parameters are read here from this file: 'keywordsfile' allows the use of an alternative header keywords file. 'compress' see COMPRESS. 'a2' To use A3 instead of A4. 'a3' To use A3 instead of A4. 'a5' To use A3 instead of A4. 'letter' To use US Letter instead of A4 with PostScript and PDF output. 'legal' To use US Letter instead of A4 with PostScript and PDF output. 'tabloid' To use US Tabloid instead of A4 with PostScript and PDF output. 'psusletter' To use US Letter instead of A4 with PostScript and PDF output. 'opfx' see OPFX. 'vcsfx' the file combination suffix ['_vmcmb']. 'vccorrf' the correction file suffix ['_corrf']. 'mosfx' the output bad-pixels mask file suffix ['_ocmsk']. 'mmsfx' the saturated-pixels mask file suffix ['_smask']. 'keeparrangement' see KEEPARRANGEMENT. 'norenormalize' see NORENORMALIZE. 'nofcdgoodnorm' see NOFCDGOODNORM. 'onlynormquadrant' see ONLYNORMQUADRANT. 'telluricline' see TELLURICLINE. 'maxintensity' see MAXINTENSITY. 'deadfibersfile' allows the use of an alternative dead-fibers file.
	ofilename	X	This keyword returns the full name of the created combined spectrum-image file.
	opath		A scalar string that specifies the path where the output data are saved. The default value is: '.'
	opfx		A scalar string that specifies a prefix for the (automatically generated) output filename. The default value is: ''
	logfile		This keyword specifies the name of an optional log file when p3d_cvimos_combine is launched as a separate program.
	loglevel		If this routine is launched as a separate program, and LOGFILE is used, this keyword specifies the log level; [1], 2, 3 (the higher the value, the more output). The default value is: 1
	allinone		Unset this keyword to write the output data to separate files, instead of writing the extracted spectra to the first (DATA) extension, its error to the ERROR extension, the bad-pixels mask image to the MASK extension, the saturated-pixels mask image to the SATMASK extension, the data-quality mask image to the QUALITY extension, and the re-normalization correction-factor array binary table to the CORRFMHR and CORRFACT extensions. The default value is: 1
	fnotify		Set this keyword to the name of a file that is written when the tool has finished; this can be a useful feature when using the p3d queue handler. In that case, set this keyword and check if the file exists before continuing. And do not forget to remove the file.
	cmdline	X	This keyword is used then the tool is launched by the p3d server.
	gui	X	If this keyword is set, the routine is assumed to be called from the p3d GUI – meaning that there is no initialization of the p3d system variables.
	subroutine	X	If set, parts of the routine messages are not written to the log file.
	stawid	X	If set, various messages are written to the p3d GUI status line (this must be the widget id of that label widget).
	topwid	X	If set, error messages are displayed using DIALOG_MESSAGE, using this widget id as DIALOG_PARENT, instead of MESSAGE.
	logunit	X	Messages are saved to the file pointed to by this logical file unit, if it is defined.
	verbose		Set this parameter to a positive scalar integer to make p3d write some information on STDOUT about what is going on. The following four values are acknowledged: 0 Writes no information at all. This is the default. 1 Writes the more important information; regarding subroutine configurations, mostly. 2 Writes most information; includes a more verbose output than 1. 3 Writes all information, including information on the execution state of GUI subroutines. This may be a useful mode when debugging the code.
	quiet		Set this keyword to not print the disclaimer on the screen when this tool is started from the command line or from the shell.
	error	X	This scalar integer returns an error code if set. Any value different from zero means that an error has occurred.
	debug	X	An error handler is not setup if this keyword is set. Otherwise, the error handler is setup using the routine CATCH, which makes each subroutine exit quietly when a bug is encountered. Use this keyword when debugging p3d.
	help	X	Set this keyword to show this routine documentation, and then exit.

Output parameters:

out

X

Upon a successful execution this variable contains the data image (odata) when the routine exits.

Examples:

This routine can be called in four different ways (for use on WINDOWS platforms, replace the path character '/' with '\'):

i) Using the p3d GUI

There is no need to setup the input parameters, p3d handles this transparently.

ii) As a stand-alone tool from the IDL command line

In addition to the extracted spectrum images, it is necessary to pass the name of the (provided) instrument-specific parameter file, as well as any additional arguments to p3d:

parfile = 'nvimos_hr.prm'
 files = ['VIMOS_oextr1.fits.gz', 'VIMOS_oextr2.fits.gz', $
          'VIMOS_oextr3.fits.gz', 'VIMOS_oextr4.fits.gz']
dfiles = ['VIMOS_oextr1_err.fits.gz', 'VIMOS_oextr2_err.fits.gz', $
          'VIMOS_oextr3_err.fits.gz', 'VIMOS_oextr4_err.fits.gz']
userparfile = '/user_p3d.dat' ;; If you use this file
p3d_cvimos_combine, files, parfile, dfiles=dfiles, $
    userparfile=userparfile, logfile=opath + 'dred.log', /verbose

iii) As a stand-alone tool from a terminal, using the IDL VM

In addition to the extracted spectrum images, it is necessary to pass the name of the (provided) instrument-specific parameter file, as well as any additional arguments to p3d. The recommended approach is to use the provided shell script p3d_cvimos_combine_vm.sh or the Python tool p3d_dispatch.py:

# Using the shell script, making use of the p3d queue system:
parfile=nvimos_hr.prm
 files=VIMOS_oextr1.fits.gz,VIMOS_oextr2.fits.gz,\
       VIMOS_oextr3.fits.gz,VIMOS_oextr4.fits.gz'
dfiles=VIMOS_oextr1_err.fits.gz,VIMOS_oextr2_err.fits.gz,\
       VIMOS_oextr3_err.fits.gz,VIMOS_oextr4_err.fits.gz
userparfile=user_p3d.dat # If you use this file
p3d_cvimos_combine_vm.sh $files $parfile dfiles=dfiles, \
    userparfile=$userparfile logfile=dred.log /verbose

# Using the Python tool, without using the p3d queue system:
parfile=nvimos_hr.prm
 files=VIMOS_oextr1.fits.gz,VIMOS_oextr2.fits.gz,\
       VIMOS_oextr3.fits.gz,VIMOS_oextr4.fits.gz'
dfiles=VIMOS_oextr1_err.fits.gz,VIMOS_oextr2_err.fits.gz,\
       VIMOS_oextr3_err.fits.gz,VIMOS_oextr4_err.fits.gz
userparfile=user_p3d.dat # If you use this file
p3d_dispatch.py --noqueue cvimos_combine \
    $files $parfile dfiles=dfiles, \
    userparfile=$userparfile logfile=dred.log /verbose

Note: In this approach, it is necessary to specify all options using the full names of the keywords. Partially correct keyword names are not permitted, as they are inside IDL. Hence, as an example, you must type "p3d_cvimos_combine_vm.sh dfilenames=$dfilenames" and -NOT- "p3d_cvimos_combine_vm.sh df=$dfilenames".

Note: The tool launches in the current directory, which is why it is unnecessary to specify full paths if files reside in the current directory.

Note: When several files are specified in the first argument, they must be separated with either a comma or a semicolon.

Note: Parameters indicated with an 'X' in the list above are unavailable in this approach. This includes the output variable parameter OFILENAME.

iv) As a stand-alone tool from Python, using the IDL VM

This approach is analogous to iii), but is executed from the Python command line. Here is an example:

# Launch p3d_cvimos_combine in a single IDL session, without using
# the p3d queue system:
from p3d_dispatch import *
parfile = 'nvimos_hr.prm'
opath = 'Output'
 files = 'VIMOS_oextr1.fits.gz,VIMOS_oextr2.fits.gz,' +
         'VIMOS_oextr3.fits.gz,VIMOS_oextr4.fits.gz'
dfiles = 'VIMOS_oextr1_err.fits.gz,VIMOS_oextr2_err.fits.gz,' +
         'VIMOS_oextr3_err.fits.gz,VIMOS_oextr4_err.fits.gz'
userparfile = opath + '/user_p3d.dat' # If you use this file
detector = 0 # Required with multi-detector configurations.
commandline = files + ' ' + parfile + ' dfiles=' + dfiles +
  ' userparfile=' + userparfile +
  ' logfile=' + opath + '/dred.log /verbose'
p3d_dispatch('cvimos_combine', commandline, noqueue=True)