README

p3d README

Version information: This is the subversion trunk version of p3d, which contains all the latest bleeding edge features; although, beware that some of the functionality may have been broken during the addition of other new features. If you want a version where few modifications are made, use one of the release packages instead (see: https://sourceforge.net/projects/p3d/files/).

$Id: README 5654 2022-07-15 12:49:25Z christersandin $

Table of Contents

1) Introduction

Section 1.1 contains a very brief introduction to p3d while the installation procedure is discussed in Section 1.2. Sections 1.3 and 1.4 describe how p3d is used from outside of IDL with the IDL Virtual Machine or from the IDL command line using a license, respectively. Section 1.5 discusses details of the required auxiliary software for users with a specific interest.

1.1) About p3d

p3d is a data-reduction tool for fiber-fed integral-field spectrographs. Please see the project web site for more details: https://p3d.sourceforge.io.

1.2) Installing and using p3d

The number of required steps to install p3d is very small for basic use. The only requirement besides p3d itself is a working version of Python (version >= 2.7) and IDL (version >= 6.2). In case that IDL is unavailable on the system, there is the option to ask us for a p3d-specific IDL distribution that works on all platform supported by IDL; IDL is assumed unavailable when the environment variable IDL_DIR is unset. To get a download link to the p3d-specific IDL distribution, please send an e-mail to the address given in the contact section.

The minimal installation consists of retrieving p3d, IDL, as well as the additional (auxiliary) software that p3d uses. p3d can download most of the additional software by itself when it is launched from outside IDL (with two exceptions, see Section 1.5); otherwise, it is only necessary to launch the software download tool p3d_acquire_xtools.py in the p3d vm directory. There are two approaches to retrieving a copy of IDL, see sections i and ii. Section iii contains information on how to acquire p3d itself (including this file). Finally, section iv describes how to make use of the Python astroquery module.

Attention: While p3d can be run using IDL versions >= 6.2, there seems to be a bug with IDL versions < 8.4 that appears when subtracting background regions in the spectrum viewer. Therefore, consider using IDL version 8.4 or higher.

i) Use the p3d-specific IDL distribution

Due to licensing issues of IDL, we cannot put the p3d IDL distribution on the internet for free download, but just send us an e-mail and receive a download link in return. The p3d IDL distribution is about 415MB large (207MB when compressed with zip) and works on all platforms supported by IDL version 8.8.1; this distribution is only meant to be used with the pre-compiled binary save files of p3d (regardless of if an IDL license is available or not) and not from the IDL command prompt. The reason is that it does not contain all of IDL, but only the parts that p3d use.

The distribution is a zip archive file that must be placed so as to have the idlrteX.X.X directory in the p3d contrib directory (see section iii below). The environment variable IDL_DIR, which configuration is described below, is set automatically to the correct path when it is unset and the p3d-specific distribution is identified.

ii) Install a full copy of IDL

A complete version of IDL can otherwise be downloaded from the following web page (it is necessary to register to be able to download the software): https://www.harrisgeospatial.com/My-Account/Downloads

The download procedure is specified on the following page (the following link should all go on one line: https://www.harrisgeospatial.com/Support/Self-Help-Tools/Help-Articles/Help-Articles-Detail/ArtMID/10220/ArticleID/17309/The-IDL-Virtual-Machine#howtodl

See the relevant IDL documents on how to install IDL (i.e. extract or copy the files to a proper location on your computer) and on how to acquire and setup an IDL license (if one is required). After IDL is installed, it is necessary to setup the system environment to tell it where IDL is found; this is unnecessary if the p3d-specific distribution is used (see i) or if your system administrator(s) has already done this. The environment variable IDL_DIR must be set to the name of the directory where IDL is installed. Check if it is already set by entering the following command (on *NIX-type platforms):

printenv IDL_DIR

If the output of the command points to the path of the IDL version to use, there is no need to set the path anew.

If IDL_DIR is unset and the p3d-specific distribution is not used, or if there is a wish to use a personal downloaded copy of IDL, the following line can, depending on the used system shell, be added to the bash terminal setup file in the home directory (~/.bashrc):

export IDL_DIR=/path/to/IDL/

or if the csh is used (~/.cshrc):

setenv IDL_DIR /path/to/IDL/

where the tilde character (~) denotes the user home directory.

It is possible to use the p3d-specific IDL distribution instead of an installed version of IDL (if the files are placed in the required directory) by simply unsetting the IDL_DIR environment variable as follows (on *NIX-type platforms):

unset IDL_DIR

Attention: If p3d is to be used on a Windows-based platform, the IDL_DIR environment variable is defined using Windows GUI tools. The exact procedure to set this variable varies depending on the used version of Windows, see below. The variable should also not in this case be set when using the p3d-specific IDL distribution.

Check that the IDL executable is available by entering the following command (this step is not required on Windows-based platforms):

which idl

If there is no output, a symbolic link could be created that points at the IDL binary; this can be done using the following example (assuming there is a bin/ directory in the home directory and that ~/bin exists in the PATH environment variable:

cd ~/bin
ln -s ${IDL_DIR}/bin/idl

When "idl" is typed at the prompt IDL should start. If there is no IDL license available, there will be a lot of information about this fact – but there is no need for concern about these particular messages if the intention is to run p3d using the IDL Virtual Machine.

iii) Get the p3d software

Either use the p3d development version through the subversion tool, or download one of the prepared releases from the program files area (https://sourceforge.net/projects/p3d/files/). The former approach is preferred when new functionality is required, which has not yet been included in a release version; although, there is a slight chance that the development code contains new bugs (that are corrected as soon as they are found and time permits, or when we are told of their existence).

To download p3d from the subversion repository at sourceforge.net, open up a terminal and change the directory to where the software is to be placed. Thereafter, enter (or paste) the following command:

svn checkout svn://svn.code.sf.net/p/p3d/code/trunk p3d

whereby all p3d-related files are placed in the directory named p3d (any directory name can be used). It is also possible to use the http protocol instead of svn by replacing "svn:" with "http:".

To update p3d when using the development version, change to the p3d directory in a terminal (or whatever name is used for this directory) and type the command 'svn update'. For any additional functionality of subversion, including a complete and very extensive manual, please consult its web page: http://subversion.apache.org/.

The required auxiliary programs of p3d are downloaded automatically to the p3d contrib and src directories if they are not detected when a p3d tool is launched from the shell, with one exception being the Voronoi binning tool of Cappellari & Copin (2003). The files of the following sources are required for full functionality (the last two are optional):

• Astro-lib (http://idlastro.gsfc.nasa.gov/homepage.html)

  • All files (because it is easier to download all files).

• Mpfit (http://purl.com/net/mpfit)

  • mpfit.pro

  • cmset_op.pro

  • cmpfit-1.4.tar.gz

• Voronoi bin tool (http://www-astro.physics.ox.ac.uk/~mxc/software/)

  • voronoi_2d_binning.pro

• SIMBAD astropy / astroquery (https://astroquery.readthedocs.io)

Also, if you intend to make use of the p3d queue system using the IDL virtual machine, it is necessary to have the the system tool "xwininfo" installed (the p3d Python queue tools use it to determine when the IDL popup window appears on the screen).

When the intention is to only use p3d from the IDL command line (see below), it is necessary to download the required auxiliary programs separately using the tool p3d_acquire_xtools.py; the same tool is called by the p3d Python tool and the p3d bash scripts when the files are non-existent:

p3d_acquire_xtools.py

The files are placed in contrib/astro-lib/ (for the files of Astro-lib) and in contrib/ (mpfit.pro and cmset_op.pro). The archive file cmpfit-1.4.tar.gz is extracted in src/, where it creates the sub-directory cmpfit-1.4.

Owing to licensing issues, the Voronoi bin tool voronoi_2d_binning is not downloaded automatically, but must be downloaded separately, putting the required file (voronoi_2d_binning.pro) in the p3d contrib directory (all tools will still work without this routine, but the related parts are deactivated).

Additionally, the Python astroquery package is not installed by p3d. To use the target-selection functionality of the p3d spectrum viewer, you need to install astroquery yourself, either on your computer, in a virtual environment that you launch p3d from, or using Anaconda. So, if you cannot install new Python packages on your computer, have a look at Anaconda or the tool 'virtualenv' – see section iv.

Note: It is also possible to download all parts and releases of p3d to the same directory by leaving out the "/trunk" part in the command above. Few users will want to do this. (The directory is in this case about 1400MB at around revision 3800.)

Note: Due to updated software on the sourceforge.net side, before 2012-12-07, the repository address was different. The old and the new repositories are incompatible, which is why it is necessary to replace older repositories with one using the address above, to allow any software updates.

Attention: When p3d starts, it places the directories it uses at the front of the IDL path, in the correct order. If other IDL tools depend on p3d, it is necessary to set the environment variable IDL_PATH or the IDL !path variable in the IDL startup file, adding the p3d directory at the required position in the used configuration. Doing this, p3d adds the additional directories at the location of that string. (There will be confusion if p3d versions are mixed up in IDL_PATH or !path.) The way p3d works, it uses the parent directory of the called p3d tool (using the IDL Virtual Machine or from the IDL command line) as the path to p3d. Likewise, p3d sets up the preferences (environment) variable IDL_DLM_PATH to include the src directory in the p3d path.

Attention: If the intention is to use p3d on a Windows-based platform, these environment variables are defined using Windows GUI tools. The exact procedure varies depending on which version of Windows is used.

Windows XP: Control Panel –-> System –-> Advanced system settings –-> Environment Variables

Windows Vista: Start –-> Computer –-> Right Click –-> Properties –-> Advanced System Settings –> Advanced –> Environment Variables

Windows 7: Start –-> right click Computer –-> Properties –-> Advanced System Settings –-> Advanced tab –-> Environment Variables

Windows 8: Start –-> All Apps –-> Control panel –-> System –-> Advanced System Settings –-> Advanced –-> Environment variables

Windows 10: Window Key + X –-> System –-> Advanced System Settings –-> Environment Variables...

Attention: Set the environment variable ${p3d_font} to the name of an existing font (see the above example) if you want to use some other font than the IDL default font; this might not always be a good idea, but the option is available nevertheless. It is important to select a font that exists on the system, check which ones are installed using, e.g., the 'xlsfonts' command. It is, additionally important to check that the used font is not too large; because, if it is, some widgets might not fit inside the respective tool (if this point seems vague, try a very large font, launch p3d or p3d_sv and look at the resulting windows). Due to restrictions in how IDL parses command-line arguments, the font string must not contain any white space!

iv) Get access to the Python tool astroquery

There are three ways to get the Python-to-IDL bridge to work: use a version of Python installed on your system that is compatible with the used IDL version, install a virtual environment using the tool virtualenv, or use Anaconda. The different approaches are described below in separate sections.

The Python-to-IDL bridge is available starting with IDL version 8.5. Current versions of IDL support the following Python versions:

IDL version Supported versions of Python 1.0-8.4.1 N/A 8.5.0-8.5.1 2.7, 3.4, 3.5 8.6.0-8.6.1 2.7, 3.4, 3.5 8.7.0 2.7, 3.4, 3.5, 3.6 8.7.1-8.7.3 2.7, 3.5, 3.6 8.8.0 3.6, 3.7, 3.8 8.8.1 3.6, 3.7, 3.8, 3.9 8.8.2 3.7, 3.8, 3.9, 3.10

Older versions of IDL become obsolete in terms of Python support when the supported Python versions become unavailable.

iv-a) Use Python on your system

This is the easiest approach if your platform supports it. Just ensure that you use a version of Python that is supported by the IDL version you use (see above). Thereafter, install the Python "astroquery" package.

iv-b) Use virtualenv

The second easiest approach is to use the Python package virtualenv. Create a virtual environment based on the required version of Python and install astroquery. These are the steps to do just that, assuming that the Python package "virtualenv" is installed on your machine:

# Change directory to where the virtual environment should reside
cd <virtual environment directory>

# Create the environment venv_py36, here using Python 3.6: virtualenv –-python=python3.6 venv_py36

# Activate the environment; this needs to be done every time you # launch p3d from a new terminal: cd venv_py36 . ./bin/activate

# Install astroquery: pip install astroquery

# Run p3d, or whatever is needed: p3d...

# You could deactivate the virtual environment when you're ready: deactivate -– -– -– -– -– -– -– -– -– -– -– -– -– -– -– -– -– -– -– -– -– -– -–

iv-c) Use Anaconda

The third approach is to use Anaconda; this approach is the most versatile and also the most cumbersome in having it work with IDL. There are two approaches here based on the default version of Anaconda: change the Python version Anaconda uses to what you need, or – if needed – create a virtual environment inside Anaconda that uses the required version of Python.

Begin with downloading and installing Anaconda using this link: https://www.anaconda.com/products/individual Having run the installation script, the package has been installed in the directory <anaconda>, which could be "/usr/local/anaconda3".

If the downloaded version of Anaconda uses the Python version you need to use, just install astroquery (and astropy) using the following command (the comment on the left-hand side that says "(base)" indicates that the default setup of Anaconda is used):

(base) $ conda install -c astropy astroquery

If the downloaded version of Anaconda uses, say Python 3.9, while you need to use Python 3.8, it is necessary to first update the installed version to use version 3.8:

(base) $ conda install python=3.8
(base) $ conda install -c astropy astroquery

Alternatively, install a virtual environment inside Anaconda that uses the Python version you need to use.

(base) $ conda create -n py38 python=3.8
(base) $ conda activate py38
(py38) $ conda install -c astropy astroquery
(py38) $ conda deactivate  # When ready...if needed.

Before you start p3d, it is in this approach necessary to adjust some environment variables, *after* the new environment is activated (the examples here apply to Linux platforms): -– -– -– -– -– -– -– -– -– -– -– -– -– -– -– -– -– -– -– -– -– -– -– (base) $ conda activate py38 (py38) $ export PYTHONHOME=<anaconda>/envs/py38 (py38) $ export PYTHONPATH=$PYTHONPATH:\ ${IDL_DIR}/lib/bridges:${IDL_DIR}/bin/bin.linux.x86_64 (py38) $ export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:\ $PYTHONHOME/lib:/usr/lib64:${IDL_DIR}/bin/bin.linux.x86_64 (py38) $ conda deactivate # When ready...if needed. (base) $ -– -– -– -– -– -– -– -– -– -– -– -– -– -– -– -– -– -– -– -– -– -– -–

The virtual environment approach is a bit cumbersome as it is necessary to adjust the variables PYTHONHOME, PYTHONPATH, and LD_LIBRARY_PATH after the environment is made active; the export lines could be placed in a separate script that is run whenever the environment is activated. It would, naturally, be practical to restore the variables when the environment is deactivated.

a) Use the Virtual Machine

The intention is that p3d is used with the IDL Virtual Machine. p3d configures itself when its main tools are launched; the use is more comfortable when the p3d vm directory is added to the system PATH. Additionally, it might be helpful to set the environment variable p3d_data_path to the name of the (base) directory where the raw data are kept. Depending on the system shell, the following lines could be added to a suitable place in the ~/.bashrc file (if using bash; or another "Bourne Shell"-type environment):

export PATH=$PATH:/path/to/p3d/vm        # Useful
export p3d_data_path=/path/to/IFU/data/  # Optional, but useful
export p3d_font=lucidasans-14            # Optional, experts only

or ~/.cshrc (csh):

setenv PATH ${PATH}:/path/to/p3d/vm      # Useful
setenv p3d_data_path /path/to/IFU/data/  # Optional, but useful
setenv p3d_font lucidasans-14            # Optional, experts only

b) Use p3d from the IDL command line

It is convenient to add the p3d directory to the IDL path to avoid having to enter the full path to the tool each time it is used. There are two ways to do this. Either set the environment variable IDL_PATH or the environment variable IDL_STARTUP. Avoid using both approaches, because there will be confusion and possible problems if the used p3d path changes and the change is not made in both places.

The first approach requires IDL_PATH to be set to include the p3d path. Put the p3d path at the front, unless an IDL program is used that itself uses p3d. Add the following line to ~/.bashrc (ensure that the string ":<IDL_DEFAULT>" is added at the end) using a "Bourne Shell"-type environment:

export IDL_PATH="/path/to/p3d:<IDL_DEFAULT> # Modify for your system

Instead add the following line to ~/.cshrc in a "C Shell"-type environment:

setenv IDL_PATH "/path/to/p3d:<IDL_DEFAULT> # Modify for your system

The second approach is to set the environment variable IDL_STARTUP to the name of an IDL program that defines the path. The following lines can be used to do this (the filename used here "IDL_STARTUP.PRO" can of course be chosen freely). For a "Bourne Shell"-type environment, add the following line to ~/.bashrc:

export IDL_STARTUP=/path/to/IDL_STARTUP.PRO # Modify for your system

For a "C Shell"-type environment, add this line to the file ~/.cshrc

setenv IDL_STARTUP /path/to/IDL_STARTUP.PRO # Modify for your system

As in the first approach, the p3d path is added at the beginning of the string unless an IDL program is used that itself uses p3d; in the latter case, the p3d path should come after the other IDL program. Add the following line to the file IDL_STARTUP.PRO:

!path = '/path/to/p3d:' + !path

Once the p3d tools are used, the IDL path is modified to fit p3d. If it is necessary to call one of the p3d modules in the p3d routines directory, it is recommended to first launch p3d with the SETUPONLY keyword, which only configures the IDL !path and then exits:

p3d, /setuponly

1.3) Use p3d with the IDL Virtual Machine, without a license

A Python tool as well as multiple bash scripts are provided to run p3d from the shell, outside of IDL, using the IDL Virtual Machine on all *NIX-type platforms as well as in Windows. To start the p3d main GUI using the p3d queue system, enter the following in a terminal (if p3d/vm was added to the system path):

p3d_dispatch.py [-s] <tool> <options>

or, if p3d/vm was not added to the system path (this is just an example, the full path could be used instead of setting p3d_path):

${p3d_path}/vm/p3d_dispatch.py [-s] <tool> <options>

Add the optional flag –-noqueue (or its equivalent -s) before the tool argument if you want to avoid using the p3d queue system. The purpose of the queue system is to click the IDL 'click-me' popup window only one time when no IDL license is available; this is facilitated by the use of a p3d server session that runs in the background where the server finishes one job in the queue at a time. The queue is managed by a p3d Python queue handler (see below).

The tool can be selected as one of the following 20 strings: p3d, p3d_sv, p3d_cmbias, p3d_ctrace, p3d_cdmask, p3d_cflatf, p3d_cobjex, p3d_cvimos_combine, p3d_cr, p3d_darc, p3d_rss2cube, p3d_cexposure, p3d_fluxsens, p3d_fluxcal, p3d_ifsfit, p3d_ifsfit_mosaic, p3d_csim, p3d_autoreduce, p3d_fastview, and p3d_d11 (more will be added). It is also possible to use any of the older bash-based scripts (which all call p3d_dispatch.py).

p3d_vm.sh [-s] [-rt] <options>
p3d_cmbias_vm.sh [-s] [-rt] <options>
p3d_ctrace_vm.sh [-s] [-rt] <options>
p3d_cdmask_vm.sh [-s] [-rt] <options>
p3d_cflatf_vm.sh [-s] [-rt] <options>
p3d_cobjex_vm.sh [-s] [-rt] <options>
p3d_cvimos_combine_vm.sh [-s] [-rt] <options>
p3d_sv_vm.sh [-s] [-rt] <options>
p3d_cr_vm.sh [-s] [-rt] <options>
p3d_darc_vm.sh [-s] [-rt] <options>
p3d_rss2cube_vm.sh [-s] [-rt] <options>
p3d_cexposure_vm.sh [-s] [-rt] <options>
p3d_fluxsens_vm.sh [-s] [-rt] <options>
p3d_fluxcal_vm.sh [-s] [-rt] <options>
p3d_ifsfit_vm.sh [-s] [-rt] <options>
p3d_ifsfit_mosaic_vm.sh [-s] [-rt] <options>
p3d_csim_vm.sh [-s] [-rt] <options>
p3d_autoreduce_vm.sh [-s] [-rt] <options>
p3d_fastview_vm.sh [-s] [-rt] <options>
p3d_d11_vm.sh [-s] [-rt] <options>

See the respective tool for detailed information regarding available options:

Note: The p3d Python tool as well as all shell scripts can be called with the -rt option. An IDL license is required in this case. This behavior could be useful if one wants to avoid having to click in the splash window that always appears each time the IDL VM is launched and the queue server is not used. Naturally, with the p3d queue server system, the need to click on those splash screens is delimited to when the p3d queue system is first launched.

Tutorials on how to use p3d in the data reduction and visualization are available in the doc directory. Here are two examples on how to use the tools from the system shell to get started:

p3d_dispatch.py p3d /cwd /ppak4k logfile=dred.log
opath=output_directory
p3d_dispatch.py [--noqueue] p3d_cobjex \
   file1.tar.gz,file2.tar.gz \
   opath=$opath masterbias=${opath}/masterbias.fits.gz \
   tracemask=${opath}/tracemask.fits.gz \
   flatfield=${opath}/flatfield.fits.gz \
   userparfile=${opath}/user_p3d.par logfile=${opath}/dred.log \
   loglevel=2 /verbose /quiet

The p3d IDL server allows any number of jobs to be submitted to the p3d queue. The p3d queue handler feeds the jobs to the p3d server, which is launched automatically and continues to run until the queue is instructed to shut down. Different tools of p3d that are started using the p3d server can use a different number of threads (cores) on the computer; the server is unaware of how many threads there are. To make efficient use of system resources, it is therefore necessary to specify how many threads should be used when a job is submitted. When all jobs are done, the queue is shut down using the following command:

p3d_queue_stop.py

If the queue should stop working for some strange reason, it is possible to kill all IDL sessions launched by the queue handler with the following command:

p3d_queue_kill.py

Note: The p3d_dispatch.py program checks if the required IDL binary meta files (e.g. p3d_*.sav) are present in the vm directory. IDL launches using these files.

Note: The Python tool p3d_dispatch.py as well as all bash scripts p3d*_vm.sh require that Python is installed on your computer, any version >= 2.7 will do. Also note that, if the command /usr/bin/env does not exist, it is necessary to launch this (as well as the other) python program(s) preceeded by the name of the Python command (with the full path):

python p3d_dispatch.py
python p3d_acquire_xtools.py
python p3d_tool_makevm.py

If there is a need to use the bash scripts instead of the Python program, it is necessary to have bash installed as /bin/bash on the computer (this does not apply to Windows computers). Please, edit the first line in the scripts accordingly if this is not the case (i.e. the first line stating "#!/bin/bash"), or execute them as in the following examples:

bash p3d_vm.sh
bash p3d_cmbias_vm.sh

Note: With an IDL license, it is possible to recompile the p3d binary files used with the IDL Virtual Machine. For this purpose, it is strongly advised to use the provided Python tool p3d_tool_makevm.py (see section 5.0).

Note: The debug option (/DEBUG) is unavailable when using p3d from the system shell (also when the -rt option is used). However, it is still possible to select not-yet-fully supported instruments from the command line, which can otherwise only be selected with the instrument-selection GUI when /DEBUG is set.

1.4) Use p3d from the IDL command line, with an IDL license

After performing the installation as decribed in Section 1.2, just enter the name of the tool to use on the command line, with the necessary arguments and keywords. For example, to launch the p3d data-reduction GUI, enter:

p3d

Available options are found in the source file, which can be browsed using any available text editor or, e.g., the commands more or less), but they can also be shown using the HELP keyword:

p3d, /help

The help option is available with all tools and routines of p3d.

Note: The exact order of entries in the IDL path is important.

Note: Do not include the p3d vm directory in the IDL !path variable! If this is done, there will be a confusion in the path if any of the program routines is changed. So, even though a p3d routine was changed, IDL might choose to use the pre-compiled version of the same routine in the binary meta file p3d_tool_server.sav. Consequently, if the vm directory is kept in the IDL !path, it is necessary to recompile all routiness after any change is made. Avoid this hassle by not having the vm directory in !path.

1.5) Additional details about the auxiliary software

Some routines or parts of routines are, in addition to their IDL implementation, also provided in C. The reason is that compiled C code in computationally intensive cases (involving nested loops) run faster than the corresponding IDL code; often drastically faster. Additionally, in C routines it is possible to make use of more of the computing power on a multi-processor computer; through OpenMP (http://www.openmp.org).

Here are some of the tasks available in both C and IDL code:

• The calculation of cross-dispersion line profiles.

• The calculation of a median-filtered science image, in regions where there are no spectra, to subtract a scattered-light, or a dark-current, component.

• The spectrum extraction, using regular tophat extraction and optimal extraction with the multi-profile decvonvolution method and the band-diagonal option.

• The conversion from pixel space to spatial-element space when selecting a region of interest in the spectrum viewer.

• The calculation of pixel-based cross-dispersion profile images used in the optimal extraction.

• The combination of multiple images using the min/max filter, and the "slowimcombine" mode.

• Cosmic-ray cleaning of single images.

• The resampling during wavelength calibration from pixel-based coordinates to wavelength.

The C routines are compiled automatically by p3d whenever they are required. With the exception of the spectrum viewer p3d_sv, all available threads on a multi-processor computer are used by default, use the NTHREADS and CRNTHREADS keywords to change this number. The spectrum viewer uses one thread by default. To use the compiled C routines, ensure a working C compiler is available. If the compilation fails, for any reason, p3d falls back to use the corresponding IDL version. Use the keyword CDEBUG to make the compilation process more verbose. It is also possible to completely deactivate the C modules by setting the keyword NOC or the user parameter 'dontusecroutines' to 'yes'. The C-based cosmic-ray cleaning routine can be deactivated separately by setting the NOCRC keyword (NOC with p3d_cr) or the 'nocrc' user parameter.

Note: If one or several of the C routines in are updated after the shared library is built, it is necessary to either set the rebuild library parameter ('rebuildlib yes') or to remove the library file in src/p3d_clib[.so,.dll]. A new library file is then compiled from the source files.

Note: The C routines are parallelized using OpenMP. Please try a more recent compiler in case messages are shown indicating that OpenMP-related routines are not found.

Note: The native compiler on MacOSX platforms clang does not include support for OpenMP (it seems). Therefore, p3d uses the gcc compiler, which needs to be installed on the system; one way to install gcc is to use the tool 'Homebrew'. Have a look at the following web page on how to install this tool:

https://brew.sh

Should you choose to install, for example, gcc version 7 using the formula gcc@7, the resulting compiler binary is named gcc-7. The next step is to set the environment variable P3D_GCC_FULL_PATH to the full path of the used gcc compiler, for example:

export P3D_GCC_FULL_PATH=/usr/local/bin/gcc-8

Note: The C routines currently compile on Linux and MacOSX platforms. Please contact us if you need support for Windows. In principle, it should only be a matter of launching idl and p3d from the Microsoft Visual Studio command prompt (because, only that way is the compiler environment setup properly). It will also be a matter of checking that the compiler options set in "p3d_misc_get_callext_exlib" are the correct ones. Any input on this issue is well received.

Warning: If p3d is used on a platform while switching between the 32 bit and the 64 bit IDL environments, please remember to delete the shared-library files between the switches, otherwise the shared library will be of an incompatible architecture, which means big problems. p3d can use its scripts with the 32 bit environment on 64 bit machines if the IDL_OPTS environment variable is set to "-32". However, it is not recommended to do this switching between 32 and 64 bits, as the outcome is system-dependent and therefore generally unpredictable.

2) Contact

All collaborators, unless noted otherwise, have the following affiliation:

Leibniz Institute for Astrophysics Potsdam (AIP) Instrumentation An der Sternwarte 16 D-14482 Potsdam GERMANY Tel: +49-(0)331-74990 http://www.aip.de

The p3d-project web site is:

https://p3d.sourceforge.io

where the software can be downloaded, the documentation viewed, and much more. Please send an e-mail to the following address, should you want to use the p3d-specific IDL distribution:

olestreicher or p2w@users.sourceforge.net

Documentation is available in the individual routines – the same text is also available on the Documentation page at the project web site; this documentation is updated whenever there is a release. Detailed descriptions of the algorithms are provided in the (freely available) Astronomy & Astrophysics paper about p3d:

Sandin et al. 2010, A&A, 515, 35 http://www.aanda.org/articles/aa/pdf/2010/07/aa14022-10.pdf

And also see the Messenger and SPIE papers (download links are available at the p3d web site):

Sandin et al. 2011, The Messenger, 144, 13-16 "p3d ~ A Data Reduction Tool for the Integral-field Modes of VIMOS and FLAMES"

Sandin et al. 2012, in: "Software and Cyberinfrastructure for Astronomy II", Radziwill N. M., Chiozzi G., eds, Proc. SPIE, 8451, 84510F-1~16 "Automated and generalized integral-field spectroscopy data reduction using p3d"

2.1) Funding acknowledgments

This project has been made possible through funds from the following sources: BMBF: 03Z2A51, 05A08BA1

2.2) Collaborators

All programming contributors of p3d

Additional acknowledgments

Please see the p3d project web site for further acknowledgments.

2.3) Reporting Bugs

To report any kind of bugs we urge you to use the bug tracker available at the p3d-project web site (Develop/Tracker/Bugs); this way the bugs will be seen by all users and programmers, and won't be lost in an e-mail program.

At the bug-tracker page you can search previously reported bugs, to see if your bug has already been reported. Or you can open your own bug description (by clicking on "Add new"). Doing this, please ensure that you use the latest version on the subversion trunk of p3d (or specify the revision or version of the release you use). Thereafter, specify a category for the bug, and provide a short summary of the problem, and most importantly, describe the problem with enough detail that somebody who does not use your setup and your data files is able to follow your arguments. Bug reports without a useful description will be assigned a low priority, or even be deleted. If you think that it would help us, attach all relevant data files to your report (please compress fits-files using tar/gzip/bzip2/xz, zip, or use the compress-option of p3d to save space). Additionally, please specify a reasonable priority to your bug. While our intention is to correct serious bugs as soon as possible ~ these are the ones that prevent p3d from producing spectra ~ GUI-related issues are assigned a lower priority.

Note: When reporting a bug, turn on the debugging mode (/DEBUG) of p3d and paste the resulting output from the IDL terminal/window (once the bug has been reached) into your bug description. The reason we ask you to do this is that the information returned by the error handler, when the DEBUG keyword is unset, only provides the name of the file where the problem occurred, and an IDL-specific number. Such sparse information is nearly always insufficient to interpret.

2.4) Feature requests

If p3d is missing some functionality that you would very much like to have included, you can make a feature request using the tracker on the p3d project web site (Develop/Tracker/Feature request). Following the description on how to add a new bug description we ask you, also in this case, to be as detailed as possible. Please understand that we are a team with limited resources. If your request concerns a huge modification, which requires many days, or even months, to code, it might not happen at all. On the other hand, we are highly interested in taking on project members who are interested in contributing to p3d with their own time and coding. Please drop us a note if you think this could apply to you.

Alternatively, send an e-mail with your feature request to the e-mail address provided in Section 2.

3) Software Platform

p3d is nearly completely based on the language of the (proprietary) Interactive Data Language (IDL; cf. http://www.exelisvis.com), and cannot be used if IDL is unavailable ~ the easiest way to get IDL is to ask us for the p3d-specific IDL distribution, see Section 1.2. Although, it is necessary to have an IDL license to modify the code, p3d can be run without an IDL-license using the supplied binary meta-files (vm/p3d*.sav). In this case, using the, so-called, IDL Virtual Machine; p3d has been implemented in such a way that it can be used with full functionality without a license.

Since p3d is based on IDL it can be used on any platform supported by IDL. However, the installation procedure differs slightly for an installation on a Windows-based platform compared to on the other supported platforms; the required environment variables need to be setup differently (see above) and the compilation of the C routines was not yet configured [please inform us if you are willing to help us finish this task].

3.1) Coding Philosophy

p3d is a large and growing piece of software. To keep it manageable, we have applied rules of coding practice; use of this practice has so far made debugging easy as bugs are located quickly.

These are the rules, which we would ask any project programming member to follow:

• All files must have filenames which conform to: p3d_task_name.pro

• Code layout: + All files must have a fully explained header. + All routines use the options: compile_opt hidden, IDL2 + The maximum number of character columns per line is 79. + All blocks are indented by 2 characters. + Comment lines begin with ';;' + Integers are with few exceptions specified as long with L, e.g. '0L'. Exceptions: VERBOSE and ERROR variables use integers. + All commands and statements are written with small characters.

• Common blocks are only allowed for variables which obviously will never ever change, i.e. pretty much never.

• An error handler is setup for every routine, unless the user keyword DEBUG is set.

• Logging comment statements must always use 'p3d_misc_logger', adopting the different levels of verbosity.

• Input parameters should be checked for consistency in terms of type, size, value and range, as far as possible.

• All command keywords are for clarity specified in full, i.e. no abbreviations (for example, /xstyle not /xst).

• Text widgets must include error checking to ensure that entered values are of the right type and range.

• All widgets in GUI tools with a status line must include the UNAME field that specifies what the widget in questions does.

• No file on the trunk should, as far as possible, be committed before the modifications are confirmed to work as intended.

• All commits to the repository should include a description of the modification (the longer the better, usually). An example is:

  svn ci --message 'Correct/modify this and that.' <file>

• Larger modifications should use a branch, and not the trunk.

• The idea is to copy the trunk to branch/ in preparations for a release. Then we only make bug fixes on that branch before we copy the final release version to tags/.

4) Supported Instruments

We have developed p3d to be a general tool. General in the sense that it can be used with any integral field unit (IFU) of fiber-fed integral-field spectrographs. Additionally, the p3d spectrum viewer and the continuum subtraction routine (p3d_d11) can be used with data cubes of any origin, such as MUSE.

Before p3d can be used with any instrument, it must be properly setup. p3d has been setup, tested, and found to work well with the following instruments:

See the p3d-project web page for more detailed information. Specifically, have a look at the documentation WIKI for instrument-specific recommendations for your data-reduction work: https://sourceforge.net/p/p3d/wiki/Home/

Note: Cosmic-ray cleaning of single images and both approaches of optimal extraction are supported on all instruments.

5) Recompiling the binary meta-files for use with the IDL VM

The simplest way to (re-)compile the binary meta-files, which are used with the shell scripts (and the IDL Virtual Machine), is to use the provided script in p3d_path/vm/. An IDL license is required to recompile p3d! The following steps could be used from a terminal:

export p3d_path=/path/to/p3d/   # Optional
export IDL_DIR=/path/to/IDL/    # IDL version >=6.2 is required!

setenv p3d_path /path/to/p3d/   # Optional
setenv IDL_DIR /path/to/IDL/    # IDL version >=6.2 is required!

p3d_acquire_xtools.py

p3d_tool_makevm.py

6) IDL Terms of use

The following terms of use apply when using IDL; the text is pasted from the web page of exelisvis:

IMPORTANT: READ CAREFULLY

Licensee acknowledges that it is not a citizen, national, or resident of, and is not under the control of the government of Cuba, Iran, North Korea, Sudan or Syria, and that Licensee will not download or otherwise export or re-export the Software Product and any related technical data directly or indirectly to the above mentioned countries nor to citizens, nationals, or residents of those countries. Licensee will comply with all applicable export, re-export and foreign policy controls and restrictions imposed by the United States and will take the necessary actions and precautions to ensure that it complies with all such laws or regulations.

I am not located in and that I am not a national of Cuba, Iran, North Korea, Syria, or Sudan, and I agree not to transfer the Software Product to any such locations or individuals;

I am not listed or affiliated with anyone on any of the following lists:

Denied Persons List – U.S. Department of Commerce Unverified List – U.S. Department of Commerce Entity List – Supplement No. 4 to part 744 of the Export Administration Regulations Specially Designated Nationals – U.S. Treasury

I also agree to comply with all U.S. and international laws (including the obtaining of any required import license or authorization), and I understand and agree that Exelis Visual Information Solutions is not responsible for my failure to comply with any such laws.

You hereby certify that you are authorized to act on behalf of your company. I Agree Cancel

View our Export Compliance document for information regarding the export of the Software Product. We recommend you seek legal counsel if you have any questions regarding the legality of your use of the Software Product.