Download
The software can be downloaded in two ways – either download a stable release version or a development version from the project Subversion tree. Either version contains all source files, required data files, and pre-compiled binary files (without these binary files p3d cannot be used without an IDL license).
Stable release- Download the latest version (as well as older versions) at the project web page. A tarball is available for Linux and and MacOSX platforms, while a zip-file is available for Windows platforms.
- The development version is updated more often than the release version, but may also be unstable, at times. Just like the release version, this version always contains the necessary IDL-compiled binary file; this binary file is – or should have been – updated whenever there is a more significant modification to the code. Fetch the latest source files using Subversion; use the following command line to place a copy of p3d under p3d/ in the current directory (it is also possible to use "svn:" instead of "http:"):
$ svn checkout http://svn.code.sf.net/p/p3d/code/trunk p3d
- Updating p3d is as easy as typing svn update from within the project directory. Change to the p3d directory and, if using a Unix-type terminal, type:
$ svn update - In case a file of the p3d distribution was modified, at a place where the same file has been changed in an updated version, there will be a conflict when the update is performed; this would also happen if the binary files (*.sav) are compiled locally, before updating p3d. If this should happen, first edit the source file(s) to keep the version to keep, and then resolve the conflict using svn. If the problem was with the binary files, first delete the .sav files before updating the code anew, or select either the (tf) option (or the (mf) option) to replace (keep) the .sav files. Here is an example on how to resolve a conflict during an update:
svn update
$ svn update
Conflict discovered in 'p3d.pro'.
Select: (p) postpone, (df) diff-full, (e) edit,
(h) help for more options: p
C p3d.pro
Updated to revision 667.
The file can now be edited by hand (it was also possible to select the e option) and thereafter it should be indicated that the conflict has been resolved:
$ emacs p3d.pro
$ svn resolve p3d.pro
If you are certain that you want to use your own version, you could instead type:
$ svn resolve --accept mine-full p3d.pro
Resolved conflicted state of 'p3d.pro' - For further information about Subversion, see its project page and online documentation.
- You can also browse the Subversion tree of p3d online.
Information on how p3d is installed on the system is described on the readme page.
Note on the installation procedure on Windows platforms:
Although p3d can be used under Windows, we have not yet been able to write detailed installation instructions for this platform; we do not use Windows machines in our own research. We would be most grateful if you could help us out with such instructions. The installation procedure is in principle the same as for the other two platforms; i.e. Linux and Mac OSX.
- 2.10 – 2023-01-01 – "Explorer"
- 2.9 – 2022-07-15 – "Avalon 1"
- 2.8 – 2021-12-31 – "God Phoenix"
- 2.7 – 2020-08-27 – "Excelsior"
- 2.7.1 – 2021-10-13
- 2.7.0.1 – 2020-10-13
- 2.6 – 2018-05-07 – "Discovery"
- 2.6.5 – 2020-07-16
- 2.6.4 – 2019-12-08
- 2.6.3 – 2019-09-05
- 2.6.2 – 2018-12-28
- 2.6.1 – 2018-08-01
- 2.5 – 2017-06-28 – "Heart of Gold"
- 2.5.2 – 2017-12-04
- 2.5.1 – 2017-11-06
- 2.4 – 2016-09-10 – "Millenium Falcon"
- 2.4.3 – 2017-06-06
- 2.4.2 – 2017-04-13
- 2.4.1 – 2017-03-10
- 2.3 – 2015-12-16 – "Bounty"
- 2.3.1 – 2016-02-08
- 2.2 – 2012-12-13 – "Serenity"
- 2.2.6 – 2014-09-22
- 2.2.5.1 – 2013-12-19
- 2.2.5 – 2013-11-06
- 2.2.4.1 – 2013-06-20
- 2.2.4 – 2013-06-17
- 2.2.3 – 2013-06-10
- 2.2.2 – 2013-05-29
- 2.2.1 – 2013-01-21
- 2.1 – 2012-01-16 – "The Warbird"
- 2.1.4 – 2012-09-07
- 2.1.3 – 2012-06-28
- 2.1.2 – 2012-04-23
- 2.1.1 – 2012-02-06
- 2.0 – 2011-06-17 – "The Swashbuckler"
- 2.0.3 – 2011-10-13
- 2.0.2 – 2011-09-28
- 2.0.1 – 2011-08-18
- 2.0b – 2011-05-03
- 2.0a – 2011-01-27
- 1.1 – 2010-04-16 – Upperkuchen
- 1.1.1 – 2010-04-21
- 1.0 – 2010-02-02 – "Destiny"
Change log
Here is a list of changes between the releases (the text is a copy of the NEWS file).
Here is a list of changes between the releases (the text is a copy of the NEWS file).
$Id: NEWS 5701 2023-01-01 16:49:10Z christersandin $
2.10 (2023-01-01; Revision 5700) - Enterprise
o All tools (in ${p3d_path}):
New instrument setup.
- Add support for the new instrument setup [PlasMark/]Andor. This is a new
instrument that makes use of a 2kx2k Teledyne e2v CCD42-40 Ceramic AIMO
Back Illuminated Compact Package High Performance CCD sensor. Two
instrument setups are implemented, using 6 bundles of 12 spectra (total
72 spectra) and 15 bundles of 24 spectra (total 360 spectra). However,
these setups are only examples, and the existing setup can be adjusted
or extended to handle any configuration of spectra up to 500 spectra.
(It would be tricky to put more spectra on a 2kx2k CCD chip.)
o The spectrum viewer (p3d_sv):
Correct bugs.
- Remove the normalization of the error array when errors are unavailable.
- Correct the handling of the REGIONS argument when starting p3d_sv using
p3d_dispatch.py.
- Correct a bug, where the wrong structure element was used:
(*state).s.f.ofilename_spat[(*state).s.f.spms_index]
==>
(*state).s.f.ofilename_spms[(*state).s.f.spms_index]
o Line fitting (p3d_ifsfit):
Add some functionality, correct bug.
- Make further adjustments to have the tool work with single spectrum
files.
- Add the option to use a specific output directory (OPATH).
- Add the option to subtract the background (on the fly) when a keyword
with the name of the background data extension is set (BGSUBKWD).
- Single spectrum plots will now always use the full plotting area
(NXPLOT = NYPLOT = 1).
- Set the spaxel area to 1.0 when using the single-spectrum mode;
otherwise, all intensities will be negative.
2.9 (2022-07-15; Revision 5660) - Avalon 1
o Auto-reduction tool (p3d_autoreduce, p3d_fastview):
New tools.
- Add a tool (p3d_autoreduce), to reduce data automatically that are
made available in a specified directory.
The tool p3d_autoreduce makes use of a directory with reduced files
(PATH_REDUCED) that are used to reduce object data files that are placed
in another directory (PATH_DATA). All reduced files that are placed in
the reduced-files directory must be prepared with version 2.9 or later
of p3d, as they contain the required FITS header keyword P3DFTYPE. That
keyword is used by p3d_autoreduce to figure out what kind of data
product each reduced file is.
The reduced files are checked regularly for new files (POLLTIME_REDUCED)
as is the object data directory (POLLTIME_DATA). Whenever one or a set
of files are placed in the data directory, they are reduced, the latest
file first. Whenever a new file is reduced, the previous raw file as
well as the reduced file are deleted.
The first time, the object data file is reduced using tophat reduction.
If the used instrument is setup to handle optimal extraction and there
is no new file in the data directory, the object file is reduced anew
using such optimal extraction.
- Add a tool (p3d_fastview), to view raw data, extracted data, and a
selected spectrum on the fly. The tool interacts with p3d_autoreduce.
The tool p3d_fastview provides a graphical user interface that allows
some control over setup parameters used with p3d_autoreduce, can launch
and shut down p3d_autoreduce, as well as pause or resume its operation,
or just make use of a running session, and views images and plots of raw
data as well as reduced data.
The shown data include: the raw CCD image, the reduced row-stacked
spectrum image, and a plot of a selected spectrum in the reduced
spectrum image.
The p3d_autoreduce pause and resume functionality can be used to save
raw data as well as reduced data. Because, raw data files as well as
reduced data files are otherwise removed whenever a new data file
appears in the data directory.
2.8 (2021-12-31; Revision 5589) - God Phoenix
o All tools (in ${p3d_path}):
New instrument setup.
- Add support for simulations and data reduction of the instrument setup
PlasMark. Simulated data include calibration data such as bias
images, continuum images, and arc images. And object data, which
currently consist of object data. The instrument has, as with
3D-Cancerspec, been configured with two default setups using 200 or
500 spectra on a 2048x2048 CCD.
However, as a new feature it is now possible to select the number of
spectra to use with a keyword; the number of spectra can be chosen in
the range 100-1000. Simultaneously, one can configure how many spectra
there are in each group of spectra, and how they are separated. It is
also possible to apply a random offset to the fiber separation (with the
user parameter 'dist_random_tr') - the same random offset value is used
with all continuum, arc, and object images. Also in this case is
spherical aberration simulated by convolving simulated images with a
Gaussial kernel; the kernel width is set with the user parameter
'aberration_fwhm'.
Synthetic fiber position tables are generated automatically for use
with p3d_sv, as a square or near square configuration using
square-shaped spatial elements. Such automatic fiber position tables
are needed whenever the number of fibers is neither 200 nor 500.
A new tutorial describes how p3d is setup to simulate the data and then
reduce the same data - tutorial_plasmark_simulations.pdf. The tutorial
comes with simulation and reduction scripts written in three languages:
IDL, Python, and shell script (bash).
- Add the header keyword P3DFTYPE to all main output products. The purpose
of this keyword is to allow an easy identification of the type of data
product that is created. For example, for created trace masks, the
keyword is set to "MASTER TRACE MASK", and for extracted object spectra
is is "EXTRACTED OBJECT.
- Specify the parameter REQDTYPE_RAW and REQDTYPE (with values) in the
instrument-specific keywords file, along with the parameters below (in
either the instrument parameter file or in the user parameter file) to
have p3d check that both raw data files as well as used reduction
products are of the correct type:
The following parameters specify a value that must equal the
corresponding value set for the RAWREQDTYPE in the instrument-specific
keywords file:
Used by p3d_ctrace to identify raw continuum images:
reqd_type_bias value
Used by p3d_cdmask to identify raw arc images:
reqd_type_continuum value
Used by p3d_cflat to identify raw continuum or sky flat images:
reqd_type_arc value
Used by p3d_cobjex to identidy raw object images:
reqd_type_flat value
reqd_type_object value
The following parameters specify a value that must equal the
corresponding value set for the REQDTYPE in the instrument-specific
keywords file:
Used by p3d_ctrace, p3d_cdmask, p3d_cflatf, and p3d_cobjex to identify
prepared master bias images:
reqd_type_masterbias MASTER#BIAS
Used by p3d_cdmask, p3d_cflatf, and p3d_cobjex to identify prepared
trace mask images:
reqd_type_tracemask MASTER#TRACE#MASK
Used by p3d_cflatf and p3d_cobjex to identify prepared dispersion mask
files:
reqd_type_dispmask MASTER#DISPERSION#MASK
Used by p3d_cobjex to identify prepared flat field images:
reqd_type_flatfield MASTER#FLAT#FIELD
2.7.1 (2021-10-13; Revision 5535) - Exelsior
o Filter tool (continuum subtraction) (p3d_d11):
New tool.
- Add a tool (p3d_d11 and p3d_tool_d11), to subtract the continuum of
extracted and calibrated data cubes. The tool cycles through the layers
(wavelength bins) and subtracts the continuum through a normalization
procedure using narrow bandpasses on the blue and red side of each
layer. The normalization is made using the spectrum of an aperture that
is defined using a more "calm" region in the data cube. Telluric lines
as well as (optionally redshifted) emission lines are masked, meaning
that the bandpasses are shifted towards lower and higher pixels to avoid
those.
This is the IDL-version of the tool, which is also available as a
standalone Python.based tool at GitHub, where it is called astro-d11:
https://github.com/ChristerSandin/astro-d11
and also
pip install astro-d11
o The spectrum viewer (p3d_sv):
Correct two bugs.
- Correct the arc line number tag written to dispersion masks; these are
now read correctly using the tag ARCLxxx instead of ARCLNxxx.
Checking for any useful values in the variable DDATA, it is necessary
to use the function MAX and not MIN.
2.7.0.1 (2020-10-13; Revision 5486) - Excelsior
o Dispersion-mask creation routine (p3d_cdmask),
The science-object extraction routine (p3d_cobjex):
Correct Bug that prevents routine from working.
- Correct a bug where the new variable SPNUM_FREE isn't set, which
prevents the tools from working. A workaround would be to set
the instrument parameter spnum_free in the instrument parameter
file, but that is not encouraged, hence this release.
o Dispersion calculation (p3d_wavecal_dispersion_correction):
Correct bugs that appear when the C routines aren't used
- Correct a bug, use NWAVES in a spot where it wasn't set.
Add missing variables (SATMASK, OBPMASK).
o Line fitting (p3d_ifsfit):
Correct bug.
- Set RACOMMENT and DECCOMMENT to empty strings when they aren't defined.
o Routine setup (p3d_misc_initialize):
Correct routine to work in Windows.
- Replace the PATH separator ':' with ';' when run in Windows.
2.7 (2020-08-27; Revision 5469) - Excelsior
o All tools (in ${p3d_path}):
Major and minor updates.
- It has become obvious that the IDL asynchronous queue system does
require an IDL license to be used; it is simply not [clearly] stated
in the IDL documentation that this is the case (it used to be, but
was removed...and will be put back in an upcoming version of IDL).
Because of this license requirement, the p3d queue server now runs
on one thread, where jobs are finished, one at a time.
While somewhat annoying, this should not affect individuals that
strongly, unless they would reduce a very large amount of data.
- Fitting done using the C-based version of MPFIT now uses version 1.4.
o Virtual Machine (all tools):
New version
- The supplied binary file now uses IDL 8.8.0. To use p3d without a
license, it is therefore now required to either install IDL version 7.1
or newer, or request a p3d-specific IDL 8.8.0 distribution from the p3d
support team.
o The p3d web page:
- Add a separate page that describes the easy way to use the p3d
spectrum viewer with publicly available MUSE data cubes.
The page can now be accessed using the simpler shortcut address:
https://p3d.sourceforge.io/MUSE
o The spectrum viewer (p3d_sv):
Improve description on how to use Python, correct bugs.
- It can be tedious to make use of the target-selection Python tool
astroquery; because, IDL must find the Python tool. Therefore, the
README file has been updated with three approaches towards using
Astroquery. The instructions describe in detail what one needs to do
to make it work. To use this functionality with the virtual machine, it
is necessary to use the p3d-specific IDL distribution version 8.8.0.
- Correct two issues related to the Voronoi binning map functionality used
when saving a created bin map.
- Correct the line fitting routine for the spectrum window so that no
error is reported when there is no range to fit, or when there are no
data to fit (as is the case before any spectrum has been selected).
o Line fitting (p3d_ifsfit):
Correct bugs.
- Add support for bin maps where the bin numbers do not have to be
increasing with a stride of 1; that is, the following bin number array
is now supported: 16, 19, 27, 102, ....
- Add the feature that line groups are skipped where the data only contain
NaNs, which could be the case if all elements are masked in the selected
range.
2.6.5 (2020-07-16; Revision 5428) - Discovery
o All tools (in ${p3d_path}):
New instrument setup, adjusted the tools for use with queue system.
- Add support for simulations and data reduction of the instrument setup
3D-Cancerspec. Simulated data include calibration data such as bias
images, continuum images, and arc images. And object data, which
currently consist of object data. The instrument has been configured
with two default setups using 200 or 500 spectra on a 2048x2048 CCD.
However, as a new feature it is now possible to select the number of
spectra to use with a keyword; the number of spectra can be chosen in
the range 200-1000. Simultaneously, one can configure how many spectra
there are in each group of spectra, and how they are separated. One out
of two additional new features is that one can apply a random offset to
the fiber separation (the user parameter is called 'dist_random_tr', and
the value can be set in the range 0.5-2.0 px); the same random offset
value is used with all continuum, arc, and object images. The second new
feature is that spherical aberration is simulated by convolving
simulated images with a Gaussial kernel. The kernel width is set with
the user parameter 'aberration_fwhm', which can be set in the range
0.5-5.0 px.
Synthetic fiber position tables for this instrument are generated
automatically as a square or near square configuration using
square-shaped spatial elements. The two default setups come, as usual,
with pre-configured fiber position tables.
The data-reduction procedures have been adjusted to allow reduction of
such simulated data where the number of spectra is not fixed; the values
are instead written to the FITS headers. The spectrum viewer has
likewise been adjusted to allow the use of a synthetic fiber position
table.
A new tutorial describes how p3d is setup to simulate the data and then
reduce the same data - tutorial_3D-Cancerspec_simulations.pdf.
The tutorial comes with simulation and reduction scripts written in
three languages: IDL, Python, and shell script (bash).
- Add the keyword FTNOTIFY to all tools; this keyword can be set to a
string with a file name. Each tool that is called with this keyword set
creates a file with the corresponding name once the tool has finished
executing. A typical use for this keyword is to control how jobs are
submitted to the p3d queue server; because, one tool might depend on
output written by another tool. Set this keyword and then have your
reduction script wait for the presence of the specified file before it
continues with the next task.
The new tutorial reduction scripts tutorial_ramcancer_reduce.* provide
illustrating examples on how the keyword can be used.
o Dispersion-mask creation routine (p3d_cdmask):
Fix a potentially annoying bug.
- Correct the use of two variables in p3d_wavecal_fit_maskwd, In some
cases, this fix will make it easier to create a dispersion mask.
o Virtual Machine (all tools):
New version
- The supplied binary file now uses IDL 8.7.3. To use p3d without a
license, it is therefore now required to either install IDL version 7.1
or newer, or request a p3d-specific IDL 8.7.3 distribution from the p3d
support team.
o Tutorials and associated scripts (in doc/):
Minor adjustment.
- Fix the MRS1 setup tutorial to use the correct line numbers associated
with revision 5316 of p3d.pro and p3d_misc_choose.pro. Also copy these
files to a separate directory (doc/aux) and make use of these copied
files in the tutorial so that the lines in the tutorial will not be
messed up the next time either routine is edited.
- Adjust the comment on how to use the respective IDL-based tutorial to
use the correct tutorial file name.
2.6.4 (2019-12-08; Revision 5322) - Discovery
o All tools (in ${p3d_path}):
New instrument setup, improved queue system.
- Add support for the Potsdam Arrayed Waveguide Spectrograph (PAWS). Data
of this instrument can now use the spectrum tracing, creation of a
dispersion mask, flat fielding, and spectrum extraction routines.
All instruments supported by p3d up to now are integral field units,
where each spectrum is linked to specific spatial coordinates on a
two-dimensional surface, such as the sky. This instrument is different.
Here, one spectrum is split into consecutive spectrum orders on the CCD.
The number of spectrum orders depends on the specific instrument setup.
Separate spectrum orders partly overlap in wavelength.
p3d reduces these data as those of any other [supported] instrument.
However, there are a couple of differences:
o The dispersion mask must be prepared for each spectrum order
separately; some new functionality in the dispersion-mask creation
tool make this process easier (viz. the option to quickly step to the
next unhandled spectrum with a mouse click).
o The extracted spectrum orders are combined into one spectrum that
contains the full wavelength range. This spectrum, as well as its
error and any other reduction products, are saved to the same files as
the usual data output, and specifically to extensions which names are
prefixed with a "C". For example, extracted spectrum orders are saved
to the "DATA" extension while the combined spectrum is saved to the
"CDATA" extension. Likewise, the spectrum order errors are saved to
the "ERROR" extension while the combined spectrum errors are saved to
"CERROR".
o There is no spatial position table associated with separate spectrum
orders, and the resulting data cannot consequenly be viewed with the
spectrum viewer tool p3d_sv.
- Revise the p3d queue system to use the new IDL routines that handle
asynchronous jobs.
The p3d job queue handler was originally written as a set of Python
programs (p3d_queue_class.py, p3d_queue_handler.py, p3d_queue_kill.py,
and p3d_queue_stop.py) that kept track of submitted jobs that were
forwarded to a number of IDL-based p3d servers (the idea of running
multiple p3d servers is that these could make better use of available
resources on a multi-core machine). While the Python jobs did as as good
job as they possibly could, the separate IDL servers would sometimes
pick the same job to work on, creating a mess.
Thanks to the added functionality of the IDL Asynchronous Jobs, which
were added with IDL version 8.7.0, the queue system has now been
(partly) rewritten to make use of this more stable approach.
Beginning with this version, the p3d queue handler now launches *one*
p3d server; this server should be launched using the IDL Virtual Machine
to avoid using an IDL license (and this is the default). The p3d server
will run until it is shut down (by executing p3d_queue_stop.py, or if it
is necessary with p3d_queue_kill.py)[, or until a bug is encountered,
and this should of course not happen].
The p3d server reads job files (written by the Bash and Python scripts
in the vm directory) and submits the jobs to an internal asynchronous
job queue where each job is launched as soon as resources become
available. (The actual launch is accomplished by spawning a child
process using the IDL to IDL bridge; there is no need to click on any
additioanl IDL VM popup splash windows in this approach.)
o Tutorials and associated scripts (in doc/):
Adjustment.
- Adjust the tutorials as well as the related reduction scripts to now use
the p3d queue server.
o Virtual Machine (all tools):
New version
- The supplied binary file now uses IDL 8.7.2. To use p3d without a
license, it is therefore now required to either install IDL version 7.1,
or request a p3d-specific IDL 8.7.2 distribution from the p3d
support team.
2.6.3 (2019-09-05; Revision 5216) - Discovery
o All tools (in ${p3d_path}):
New instrument setup.
- Add support for the fiber-Bragg grating instrument PRAXIS. Data of
this instrument can now use the spectrum tracing, creation of a
dispersion mask, flat fielding, and spectrum extraction routines.
o C-compilation on Mac / Darwin platforms - p3d_misc_get_callext_exlib:
Improve user experience.
- For users on Darwin platforms, add use of the environment variable
P3D_GCC_FULL_PATH, which is set by the user to the full path of the
gcc binary she or he wants to use with p3d; this allows selection
of any particular gcc version that is installed using Homebrew
(see https://brew.sh). Any p3d user on a Mac / Darwin platform is
recommended to install a version of gcc that includes support for
OpenMP.
o A couple of minor bug fixes:
Small but important fixes.
- p3d_ctrace, p3d_cdmask, p3d_cflatf, p3d_cobjex:
Allow BISCONSTANT to be used when there is no master-bias image.
- p3d_extract:
Correct the routine when BIAS is a double variable.
- p3d_extract_tophat, p3d_sv:
Make tool work again when there are no error data.
- p3d:
Correct use of the dispersion-mask spectrum-viewer button.
- p3d_sv:
Correct a bug that appears whenever a cube is viewed.
2.6.2 (2018-12-28; Revision 5160) - Discovery
o The spectrum viewer (p3d_sv):
New functionality and bug fixes
- The spectrum viewer now handles image (files) that contain a second
image in the extension BSW_DATA; this second image could contain
so-called laser offset, or beam-switched, spectra that are offset from
the regular spectra. The user can select between the regular spectra,
the [laser] offset spectra, the difference between the two, and a,
so-called, laser normalized difference between the two using the new
droplist widget on the Extra control panel or using the new menu entries
in the "Image View" menu. Two additional widgets allow the offset BSW
data to be multiplied with a factor before the offset spectra are
subtracted from the nominal spectra; the default multiplication factor
is 1.0.
- Add File menu entries to look at the FITS file headers of the BSW_DATA
and BSW_ERR extensions, when they are present.
- Add the feature that the viewed spectrum can be saved as a plain-text
ASCII file in addition to as a FITS file (simply use the file endings
".txt" or ".txt.gz").
- Additionally, several or all spectra in the [FITS file] image or cube
can be exported to plain-text ASCII files by selecting the corresponding
option in the File menu. Each plain-text file contains a header
describing the origin of the data and what each column and row contains.
Note! please be careful when using this functionality with huge [MUSE]
cubes; because it will take a while to write, say, one million text
files and there is [currently] no way to stop the writing once it has
begun.
- The spectrum viewer can now be used with reduced data cubes of the
MaNGA instrument. So far the masked elements are not treated, and
so they're set to 0, which results in some messages saying
"Floating divide by 0". However, just don't use those elements around
the border of the IFU and you're fine.
- The configuration save file version is now 1.1, which includes saving
and loading the image selection, the instrument-parameter file
(PARFILE), the detector (DETECTOR), the user-parameter file
(USERPARFILE), and the window size (WINXYSIZE); the tool has been
updated to that it uses the required version of the configuration file
(instead of *any* version). Also correct a few bugs that didn't allow
loading a state file.
- So, after having run the spectrum viewer for any image (and not only a
cube), just launch the tool again without using any options. Voilá, it
should load the last viewed image automatically and also set nearly all
options that were used in the last session. A very convenient feature!
- Correct the spatial map regarding the viewed image and the selected
spectrum when using data not sampled on the sky (MRS1 and MRS2). The
shown spatial map now really corresponds to the selected spectrum.
- Correct a bug that appears when the computer isn't equipped with the
gcc compiler; because, in that case a bug in the IDL routine
p3d_display_sv_event_plot has complained about a missing variable
(LOWER).
- Correct the region statistics median calculation that is calculated when
using an RSS image and the C routine p3d_csum. A bug has so far rendered
p3d and IDL crashing whenever this was attempted; this is fixed with
revision 5127.
o Spectrum extraction:
New functionality.
- Raw data that contain a second set of spectra offset from the nominal
set can now be reduced in two ways:
o The offset spectra are reduced the same way the nominal spectra are
reduced. The offset spectra are written to the BSW_DATA and BSW_ERR
extensions.
o Set the user parameter 'bsw_px_subtraction yes' to instead subtract
the offset spectra from the nominal spectra directly, before the bias
image is subtracted from the raw data. In the ideal situation, spectra
(extended on the cross-dispersion axis) would be subtracted right away
and no extra consideration would be required. However, due to the cost
of CCDs, it is rarely the case that spectra do not overlap, and
because they do, the direct subtraction isn't complete. Instead, the
approach implemented here is to immediately subtract the contribution
to the flux of the center pixels of the offset spectra. Thereafter, a
regular spectrum extraction takes place (preferably using optimal
extraction), whereby the fraction of the flux outside of the center
pixels is kept. The resulting flux at each wavelength and for each
spectrum is calculated as the sum of the directly subtracted value of
the center pixels and the difference between the nominal and offset
flux values of the regularly extracted flux wings. Another consequence
of using this mode is that the parameter 'pmultfac' is set to 1
instead of the default value 10 (any value other than 1 is
conceptually difficult to implement).
o All tools (in ${p3d_path}):
New functionality.
- Add support for the new experimental instrument R10E-Atik. This
instrument is much simpler than other instruments and has spectra that
appear as wide bands on the cross-dispersion axis instead of peaks. The
spectrum center positions are extracted using a boxcar smoothed array.
Moreover, the raw data contain hardly any keywords that describe the
used setup. Consequently, this setup can be improved upon.
- Written multi-extension FITS files now contain header keywords EXTENxxx
that provide the number and names of all written extensions in the file.
- Add the raw data file suffix keyword SFX to p3d and the main reduction
tools. While a similar functionality is achieved with the user-parameter
'sfx', this keyword allows the use of raw-data files in the p3d main GUI
that do not use the filename suffix '.fits'.
o Correction for differential atmospheric refraction (p3d_darc):
Minor new functionality
- To analyze second-order spectra - where features appear at the double
real wavelength - simply set the new keyword USE_SECONDORDER_SPECTRUM
or the user-parameter 'use_secondorder_spectrum'. The wavelength array
used to calculate the spatial offset due to differential atmospheric
refraction is thereby calculated using the true wavelength of the
second order features. For example, your data contain second order
features of [NeIII] 3568, which appear in your data at the double
wavelength, i.e. at 7132 Angstrom, assuming neglibible redshift.
o Virtual Machine (all tools):
New version
- The supplied binary file now uses IDL 8.7.1. To use p3d without a
license, it is therefore now required to either install this version of
IDL or request a p3d-specific IDL 8.7.1 distribution from the p3d
support team.
o The p3d main GUI (p3d_tool_gui):
Correct minor bug.
- A small bug has prevented the p3d main GUI to be launched using the IDL
VM when any other than the default instrument is used; this has been
fixed with revision 5047.
o The RSS to cube conversion routine (p3d_rss2cube):
Correct minor bug.
- Correct a bug whereby the tool was unusable (a variable was never set);
this has been fixed with revision 5046.
o A number of minor bug fixes.
2.6.1 (2018-08-01; Revision 5021) - Discovery
o All tools (in ${p3d_path}):
New IFU, beam-switched data
- Add support for the new IFU MRS2 using the MRS1 setup files. To use
MRS2, one simply specifies the MRS1 parameter file while setting the
DETECTOR keyword to 1 (instead of the default 0, which makes use of
MRS1).
- Add support for beam-switched data in the object extraction. So far,
this is implemented for the MRS2 setup (through the presence of the two
header keywords BS_NROWS and BS_DIR in the raw data headers).
In beam switched data, the IFU is pointed in a different direction
during half the time. For example, half the time the IFU points at the
target and the other half it points at the sky. The spectra of the
offset observations are offset from the first position by a number of
pixels on the cross-dispersion axis (BS_NROWS, which is set to the
offset when there is no binning of the CCD data on the cross-dispersion
axis).
Using this mode, the continuum and arc images that are used for tracing,
to create a dispersion mask, and flat field (normalize) the spectra,
should not be beam switched. Only the object image can show regular
spectra that are interleaved with the beam-switched spectra.
o Tutorials and associated scripts (in doc/):
Addition and improvements.
- Add a visualization tutorial for the instrument setup MUSE-WFM.
- Adjust the MRS1 data-reduction tutorial and scripts to include
information about the new MRS2 IFU.
- Adjust the Python-based scripts to work with Python version >= 3.
- Add checking of the environment variable "rootpath" to the IDL-based
tutrial scripts.
o Spectrum extraction:
New functionality, correct a bug.
- Add a parallelized C version of the regular tophat extraction routine
"p3d_extract_tophat", i.e. "p3d_ctophat.c". The new routine takes
extraction times down from about 1.25s to about 0.075s with PMAS-Larr
data (on an 8-core AMD Ryzen 1700X using all 8 cores).
- Correct a bug in the C routine p3d_cssl_median.c; the variable
kArgsExpected has now been set to the correct value, 9.
o Line fitting (p3d_ifsfit):
Correct bugs.
- Correct the handling of sky and their associated auxiliary lines to make
them work as intended, which is to allow fitting the sky with emission
lines.
- Fix the tool so that it again correctly works with scalar redshift
values.
o The p3d main GUI (p3d_tool_gui):
Correct bugs.
- Correct the tool so that the tool p3d_sv - once again - can be used from
the p3d main GUI panel.
- Correct the raw-image viewing tool (p3d_display_imv) so that it can view
images again when there is no corresponding raw image present.
o Spectrum Viewer (p3d_sv):
Correct bugs.
- Correct the setup of the spatial-map regions map filename array so that
the filename is set correctly when DS9 regions are used.
- Correct the ROI region selection button so that the tool continues to
work also when the popu window (XROI) is closed immediately without
selecting any region of interest.
o Bias-subtracted images (p3d_ctrace, p3d_cdmask, p3d_cflatf, p3d_cobjex):
Correct a bug in p3d_cflatf, adjust the other routines.
- Correct the saving of a bias-subtracted image using "p3d_cflatf", the
resulting file is now a working FITS file.
- Adjust the FITS header keyword IMTYPE in the written file for all four
tools.
2.6 (2018-05-07; Revision 4912) - Discovery
o Spectrum Viewer (p3d_sv):
New functionality.
- Add a target setup panel that makes it easier to log fits and save the
current spectrum, the current spatial map, and the current region (as
well as its background region) to a region file, all simultaneously with
*one* click. The target setup panel is accessed from the Extra control
panel through the button "Show Target Setup".
The target setup includes setting: a target name, a target type, a
target comment, a target output path (with buttons to choose the session
output path or the target name, or one can set the path freely), a
target filename prefix (with buttons to choose the name of the used data
file or the target name, or one can set the prefix freely), and buttons
that allow the control of which files are written when the button
"Save Files / Log Fit" is pressed. The four files are: the log file,
the currently viewed spectrum, the currently viewed spatial map, and the
currently selected region as well as its background, if any.
The open log file button has been moved to the target setup panel.
- More information is written to the line fit log file. Additions include
the name, target type, and target comment. The mean target coordinates
of all spatial elements in the region selected in the spatial map are
written in column 9 for each fitted line. The RA, DEC coordinates are
calculated using the same window coordinates and are output to the tenth
column. Please note that the RA, DEC coordinates will be inaccurate if
the IFU coordinates are inaccurate.
- The target setup panel adds yet another panel, the target selection
panel that allows a quick selection of a target name after the tool has
made a query to the SIMBAD database (using the Python astroquery
package). This functionality is only available with IDL version >=
8.6.0 (8.7.0 when using the p3d-specific IDL distribution) and the
astroquery package must be installed on the computer before launching
the spectrum viewer; either on the system or in a virtual environment.
- Load and save the configuration. Until now, it has been necessary to
make all setups each time the spectrum viewer is restarted. Beginning
with this version, it is possible to save and load the configuration
using the state save/load functionality; the state is also saved to a
file in the user home directory each time the tool is closed. Most
settings are saved, but not all. The panning of the spectrum image, the
spatial map, and the spectrum plot is not saved. Also not the setup in
the sensitivity-function panel.
- Add the functionality that the output path is saved when saving anything
to a file; to avoid having to select the output directory anew each
time. The default output path can be set using the keyword OPATH.
o Spectrum Viewer (p3d_sv):
Adjustments to the functionality.
- The spatial map x coordinate flip is now unset by default; this has the
consequence that coordinates in the spatial map panel increase
East => West, as in ds9.
- The default is now to show "spectrum - background" in the spectrum plot.
- The default spatial element selection mode is now "Sum" (instead of
"Single").
- Saving the current spectrum now saves both the main spectrum (its error)
and its background spectrum (its error) to the same file, if a
background was selected. The background is saved to the extension
DATA_BG and the error to the extension ERROR_BG, if they are available.
The background spectrum is rescaled to use as many spatial elements as
the main spectrum.
- Only one region is saved (along with its background region, if any) when
saving the current region using the target setup panel.
o All tools using widgets (p3d, p3d_sv, p3d_ctrace, ...):
Improved functionality.
- It is now possible to set the font keyword to a font of the user's
choice, and it should all look fine (for as long as the font uses a
fixed character width and isn't too large).
- The option FONT='big' selects a - hopefully available - font for use
with 4K (3840x2160) screens. The spectrum viewer (p3d_sv) also provides
the option FONT='default' to switch back to the default font if needed.
o Virtual Machine (all tools):
New version
- The supplied binary file now uses IDL 8.7.0. To use p3d without a
license, it is therefore now required to either install this version of
IDL or request a p3d-specific IDL 8.7.0 distribution from the p3d
support team.
o A number of minor bug fixes.
2.5.2 (2017-12-04; Revision 4742) - Heart of Gold
o Simulation tool (p3d_csim):
New feature. Multiple important bug fixes.
- Add the functionality to use two-dimensional kernels when creating the
simulated data (KERNEL). The kernel data are saved as a four-dimensional
array in a fits file, where the kernel image is stored in the first two
dimensions, the wavelength index is the third dimension (optional), and
the spectrum index the fourth dimension (optional).
- Correct the use of read noise and gain factors. Noise is now added to
all blocks in multi-block instrument setups in a correct way.
- Add a proper description on how to use the DKERNEL and CKERNEL
parameters, and correct their use in the code.
- Correct the use of the DSUBSAMPLE and CSUBSAMPLE parameters; they must
both now also be set to an odd positive integer. The header keyword
DSUBSAMP and CSUBSAMP must be set to the subsampling factor when using
the KERNEL, DKERNEL and CKERNEL keywords, respectively.
- The simulation works by shifting the kernels on the dispersion axis
and the cross-dispersion axis according to how the spectra are
shaped on the dispersion axis (DISPMASK) and cross-dispersion axis
(TRPOSPAR). For this to work correctly, it is necessary that the
provided spectra are padded with zeroes on the sides of the profiles.
Alternatively, the kernels can be zero padded using the keywords
ZPAD (used with KERNEL), DZPAD (used with DKERNEL and the default
dispersion kernel), and CZPAD (used with CKERNEL and the default
cross-dispersion kernel).
o Virtual Machine (all tools):
New version
- The supplied binary file now uses IDL 8.6.1. To use p3d without a
license, it is therefore now required to either install this version of
IDL or request a p3d-specific IDL 8.6.1 distribution from the p3d
support team.
o A number of minor bug fixes.
2.5.1 (2017-11-06; Revision 4700) - Heart of Gold
o Line fitting (p3d_ifsfit):
New functionality. Correction
- Sometimes, when the velocity varies across the observed object, it is
insufficient to merely increase the allowed deviation of fitted
wavelenghts using the keyword DWL. At such times it could work better to
use a two-dimensional redshift map instead of a constant redshift.
Add support for the use of two-dimensional redshift maps. Up to now,
the redshift has been specified as a scalar decimal value using either
the keyword Z or the keyword VEL_Z. Now, it is also possible to provide
a map of the redshift that is provided as a fits file using either one
of the same two keywords; the map is a one-dimensional array (RSS
images) or a two-dimensional image (data cubes).
- The C-based calculation of the binned redshift array has been adjusted
to reflect the IDL code; this improvement affects, in particular, faint
data (on the spatial boundaries, mostly) where spectra are only present
in some wavelength bins.
- Improve the description of the contents of the map images that are
written along with the binary-table fits files. The map files are
written when the spatial elements are square shaped. Each fitted
property is converted to a spatially extended image that is written to a
separate extension. When a two-dimensional redshift map is used, the
velocity version of the input redshift
(v0 = c * ((1 + z)² - 1) / ((1 + z)² + 1)) is written as the last
extension (to simplify the checking for potential problems).
o Adjust the code to work with IDL 8.6.1; specifically, this is achieved by
replacing the formerly used notion for octal constants with a format that
works in the latest version of IDL. For example, <"305b> is replaced with
<'305'ob>.
2.5 (2017-06-28; Revision 4664) - Heart of Gold
o Simulation tool (p3d_csim):
Multiple important bug fixes.
- Correct the written simulated raw files so that they can be reduced
with the p3d data reduction tools.
o Tutorials and associated scripts (in doc/):
Improvements.
- Add images to the description of the dispersion-mask creation part in
all three data-reduction tutorials (MRS1, PMAS-Larr, and PMAS-PPak).
- Add the feature to all scripts that the environment variable "rootpath"
can be set to change the path; this way it isn't necessary anymore to
edit any of the scripts before using them. Adjust the line numbers in
all tutorial scripts to apply to the adjusted scripts.
o All tools (in ${p3d_path}):
Important bug fixes and ajustments.
- Instruments that use the NGC CCD controller and provide combined raw
images that include prescan and overscan regions in the image (such as
ERA2) are now correctly sliced up again (i.e., the prescan and overscan
regions are correctly removed). The bug was introduced with revision
4330 and fixed with revision(s 4620 and) 4630.
- Set the default polynomial order used when fitting extracted continuum
spectra with a linear polynomial to 0 [MRS1 instrument]. The spectrum
is difficult to fit with lower order polynomials, and doing so anyways
introduces strange features in flat-fielded spectra.
- Correct all examples that are run from the command line using a shell -
remove all white space before and after equal signs (that I introduced
earlier to make the source code easier to read).
o A number of minor bug fixes and adjustments to the language of comments
throughout the code.
2.4.3 (2017-06-06; Revision 4608) - Millenium Falcon
o Simulation tool (p3d_csim):
New tool.
- Add a tool to add spectra to simulated "raw" CCD images. The tool
p3d_csim can be used with any setup that contains the required
header keywords preset file. The first added setup is to model
Mosaic-like detectors, it is called mosms. Here is a list of features
that are included in the simulations:
+ Select the instrument configuration, and thereby lock information
about how many spectra to model, their separation, their geometrical
placement on the IFU, and the wavelength range.
+ Select the spectra to work with.
o The object spectrum can currently be that of a point source, which
can be provided as a two-column plain-text file or a binary-table
fits file. The point source is spread out using a defined seeing and
a modeled two-dimensional Gaussian profile.
o A continuum spectrum is used to create traces for all spectra;
either provide a spectrum in a file or use a constant value
(default).
o An arc spectrum is created using a list of wavelengths and
intensities, provided as a two-column plain-text file or a arrays.
+ For each kind of image, except the bias images, the spectra are
centered on the spatial axis on the simulated CCD image. The spectra
are convolved separately on the dispersion and cross-dispersion
(spatial) axes:
o One kernel or a set of kernels spread out on the dispersion axis
describe how the dispersion varies on the dispersion axis. It is
possible to use the same kernel(s) for all spectra, but one can also
use different kernels. The resulting kernel is interpolated for
pixels intermediate to the provided wavelengths. The kernel can be
subsampled to any depth.
Variations of the dispersion are accounted for by offsetting the
dispersion kernel using an optional dispersion mask.
The spectrum is at first convolved with the kernel on the
dispersion axis.
o Another kernel describes how the spectrum is spread out on the
cross-dispersion axis. It is specified just as the dispersion-axis
kernel described above.
Spectra that are curved on the CCD image are account for by
offsetting the cross-dispersion kernel using a trace fit for each
spectrum, provided as a set of polynomial parameters.
The spectrum is then multiplied with the normalized cross-dispersion
axis profile for each pixel on the dispersion axis. The resulting
profile is added to the output image.
o After both convolutions, the output image is rebinned to remove the
subsampling.
+ Each image is split into blocks, according to the definition in the
instrument parameter files. Prescan and overscan regions are added to
the image and it is flipped according to how the corresponding
instrument CCD image would appear.
Poisson noise and readout noise as well as a bias level is added to
the image before it is converted to the unsigned integer type
(16 bits per pixel) and is written to a FITS file using multiple
extensions.
+ The idea is then that the resulting bias, continuum, arc, and object
images are fed back to the reduction tools of p3d to reduce the files
to allow an analysis of what results.
2.4.2 (2017-04-13; Revision 4551) - Millenium Falcon
o Tutorials (in doc/):
New data-reduction and visualization tutorials.
- Add data-reduction and visualization tutorials for three instrument
setups: MRS1, PMAS/Larr, and PMAS/PPak.
+ The data-reduction tutorials describe all steps necessary to convert
raw input images to extracted spectra; this includes the five steps of
creating a master bias, tracing the spectra, creating a dispersion
mask, extracting spectra for normalization of extracted spectra, and
extracting object spectra, using all previous products. The PMAS/Larr
tutorial is a bit more extensive than the other two and also includes
the creation of a sensitivity function, flux calibration using the
sensitivity function, correcting for effects of DAR, and converting
the resulting RSS image to a data cube. The tutorials come with both
reduction scripts - written in Python, bash, and IDL - and data files
that can be used to follow all steps of the reductions. The tutorial
data are available in a file that can be downloaded through a link on
the p3d web page. The reduction scripts are available both in the p3d
distribution as well as in the tutorial data files.
+ The visualization tutorials describe how one can make use of the
functionality in the p3d spectrum viewer. Examples of covered issues
include:
o Starting the spectrum viewer from the command line.
o Using the spectrum image: changing colors, selecting wavelength
bins, panning in wavelength and scrolling across spectra.
o Using the spatial map: panning and zooming, creating regions,
adding background regions, and blinking between spatial-map
images of different wavelength bins.
o Using the spectrum plot: panning and zooming, showing emission
lines, fitting emission lines and setting the redshift.
As with the data-reduction tutorials, the visualization tutorials come
with both launch scripts - written in Python, bash, and IDL - and data
files that can be used to follow the tutorial. The tutorial data are
available in a file that can be downloaded through a link on the p3d
web page. The launch scripts are available both in the p3d
distribution and in the tutorial data files.
o The science-object extraction routine (p3d_cobjex):
Add a new feature.
- The keyword cosmic-ray cleaning keyword CRCLEAN can now be specified
for each input image individually.
o The tracing procedure (p3d_ctrace):
Add new functionality.
- Add a new keyword, IMCOMBINE, which allows the tool to either combine
all input images before tracing spectra (as before) or trace spectra in
each image individually.
o Correction for differential atmospheric refraction (p3d_darc):
Minor adjustments.
- Correct a few minor bugs to make the tool work anew with instruments
such as the PMAS/Larr.
- In the properties popup GUI: add a label on the first line to indicate
which file the properties are shown for.
o All tools (in ${p3d_path}):
Bug fixes and minor adjustments.
- Correct the handling of pre-scan and over-scan regions so that they
can again be used instead of master-bias images, if necessary.
- Correct the creation of landscape PostScript images, they are anew
created as intended.
- Correct the handling of decimal-type keywords when the tools are called
using Python.
- Capitalize widget labels to increase the appeal.
o Numerous minor bug fixes and adjustments to comments throughout the code.
2.4.1 (2017-03-10; Revision 4452) - Millenium Falcon
o Tutorials (in doc/):
New tutorial (which is an all-new feature).
- Add an instrument configuration tutorial for MRS1. The tutorial
describes all steps necessary to setup p3d for the instrument MRS1 (and
this tutorial can be used to setup similar instruments as well). The
tutorial comes with a small data set that can be used to have a look at
the exact format of the raw files that p3d expects for this instrument.
The tutorial data are available through a link on the p3d web page.
o All tools (in ${p3d_path}):
Instrument setups.
- Adjust the setup of MRS1.
o The use of prescan and overscan regions (p3d_tool_pobias):
Important bug fix.
- Corrected a serious bug, where two keywords had ambiguous names;
resulting in the feature of using prescan and overscan regions
being unavailable.
- Correct the output to the PostScript file of the value of
!p3d_data_path.
o Line fitting (p3d_ifsfit, p3d_ifsfit_mosaic):
Minor (bug) fixes.
- Fix output maps to conform to FITS specifications.
- Exclude pixels with non-positive errors from both binning and single
spectra; this allows the C-based routine cmpfit to always converge.
- Fix a type to convert MUSE variances to sigmas.
o Some additional minor bug fixes and adjustments to comments.
2.4 (2016-09-10; Revision 4387) - Millenium Falcon
o Spectrum Viewer (p3d_sv):
New functionality.
- Increase the number of definable regions to 1024, which may all overlap.
A region may use any (one) other region as a background (except itself),
which co-added and scaled spectrum is subtracted from the co-added main
spectrum. The spectrum plot now shows either the unsubtracted spectrum,
the background-subtracted spectrum (this is the default), the
background-subtracted spectrum and the background spectrum, or all three
spectra.
- Regions and background regions are selected either by entering the
required index or by clicking in the spatial map - using the Ctrl and
Ctrl+Shift modifier keys simultaneously.
- Regions configurations can be saved as well as loaded to and from a
plain-text file. Such region files can be loaded already upon launchning
the spectrum viewer through use of the keyword REGIONS.
- It is also possible to load DS9 region files. The following seven
DS9-types of regions are supported: circle, annulus, ellipse (as well as
ellipse ... & !ellipse ...), box (as well as box ... & !box ...),
polygon, and text (which is actually a property and not a region). All
regions can have local properties associated with them; specifically,
the following properties are recognized: text, width, color, and
textangle. All colors in the file /usr/share/X11/rgb.txt are recognized.
- Multiple DS9 region files are loaded either by selecting the load option
multiple times or by selecting several files simultaneously.
- Additional checkbox buttons allow viewing all defined regions
simultaneously in the spatial-map panel, together with the DS9
properties if such are defined.
- An additional option in the Spatial-map menu allows all spatial elements
in all regions to be unselected with one click; a dialog GUI is shown to
confirm the action.
- When multiple groups are chosen for the same region in the regions tool
(XROI), they are now xor:ed when calculating the spatial elements mask.
For example, as a result, by creating two circles with the same origin
using different radii, the resulting region is an annulus.
- Add coordinate information for the spectrum image and the spectrum
plot. The information includes the wavelength pixel, the wavelength
or the wavenumber, the flux and its error (when available) at the
position of the mouse cursor. The background flux (and its error) is
shown for the spectrum plot when a background region is defined. The
spectrum number is shown when the mouse pointer hovers above the
spectrum image.
- The flux is indicated in the spectrum panel with a diamond symbol. When
the actual value is higher or smaller than what is seen in the spectrum
plot, a plus (minus) symbol is shown above (below) the diamond.
- Add the option to use physical coordinates of the spatial elements
that are flipped on the East-West axis. The flipped coordinates are the
same that one would see in the tool ds9.
- Add the option to fit a Gaussian emission line in the spectrum plot;
this feature is used by clicking on the respective outer background
points and then at the position of the emission line. The fitted
parameters are shown in the coordinate field until either a new fit is
made or the "Clean Fit" button is selected.
- Add emission line list files that are used to show the locations of
emission lines. The redshift used is either the default system redshift
(specified with the REDSHIFT_Z and REDSHIFT_VEL keywords) or the
redshift of the line fit. Additional widgets allow the system redshift
to be altered inside the tool.
- Data cubes that use the FITS header keyword BUNIT (such as data cubes of
the MUSE pipeline) now make use of this constant when it is in the form
'10**(xxx)*erg/s/cm**2/Angstrom' to scale the read data with the factor
indicated with xxx.
o Line fitting (p3d_ifsfit):
Major new functionality and bug fixes.
- Now also handles binned data in three different approaches:
o A bin map can be provided as a plain-text file or a binary-table fits
file. The format of the bin map includes x and y positions for each
spatial element as well as a bin index. The data can be contained in
the fits files with a freely chosen extension name (defaults are
BIN_TABLE and VORONOI_TABLE) and column names (defaults are BINIDX or
BINNUM for the bin index and X and Y for the //x// and //y//
positions, respectively).
o The second approach is to let p3d_ifsfit calculate a bin map using
the tool VORONOI_2D_BINNING (Cappellari & Copin 2003), where it is
possible to define the bin range to use and the target
signal-to-noise value.
o The final approach is to use a cube image that was already binned,
and where the bin map is provided either as a separate extension or
in a separate file. This setup requires quite a number of fulfilled
conditions to work - please see the documentation of p3d_ifsfit for
more details.
- Both binned and unbinned data use the C-based DLM p3d_cifsfit, which
means fast and parallelized fits.
- Now also writes spatial map image files in addition to the binary-table
fits files. The image files contain images of all fitted parameters and
their errors in separate extensions. While the binary-table fits files
are meant to be used with p3d_ifsfit_mosaic, the image files can be used
by any tool reading fits images.
2.3.1 (2016-02-08; Revision 3993) - Bounty
o All tools:
Bug fixes.
- Correct the size of written postscript and PDF documents.
- Correct all tools so that they can be used with the updated version of
the IDL astro-lib routine READCOL (the affected version is of Oct 2015).
o Line fitting (p3d_ifsfit):
Minor new functionality and bug fixes.
- Add the *real* redshift as a separate keyword, Z (which could be used
instead of the wavelength shift of the keyword REDSHIFT). Alternatively,
the redshift can be specified as a velocity using the new keyword VEL_Z.
- Correct the C-version of the line-fitting tool.
- Correct the use of the tool when LOGLEVEL is not used.
o Graphical presentation of line fits (p3d_ifsfit_mosaic):
Bug fixes.
- Correct the filename use when writing to a PDF file.
- Correct the color bar for non-log plots.
o Create a dispersion mask (p3d_dmask):
Bug fix.
- Correct the tool so that it can be used again.
o Line fitting (p3d_sv):
Minor bug fixes.
- Correct the standard-star tab calibration-file widgets.
- The fine shifts of the wavelength range in the spectrum-plot window
(holding the Ctrl or Ctrl+Shift keys when using the mouse wheel) now
works with a minimum shift to higher wavelengths of 1 pixel (could be 0
pixel until now).
2.3 (2015-12-16; Revision 3951) - Bounty
o All tools (in ${p3d_path}):
Improved the documentation, minor and important new functionality.
- Entries in log files are now prefixed with a time stamp.
- Add the option to avoid using the p3d queue system; this is accomplished
by setting the -s or --noqueue flag when calling the bash scripts or
p3d_dispatch.py. Use this functionality to avoid having to use the p3d
queue system that requires some effort to learn and understand.
- Made improvements to the p3d queue system to make it more stable. The
queue system now requires Python version >= 2.7 to run.
(Only one hurdle remains to have it work on Windows-based platforms;
the POSIX tool "xwininfo" is used to check for the existence of the IDL
popup window on the screen.)
- Add a p3d-specific IDL distribution that removes the necessity to
register with EXELISVIS to download IDL; this distribtion is used
whenever IDL is not installed on the system. IDL is assumed to be free
of IDL when the environment variable IDL_DIR is unset. Please e-mail us
to get a download link for this distribution file; use the e-mail
address in the README file.
- The spectrum viewer, the dispersion-mask creation GUI, and the new tool
p3d_ifsfit_mosaic now include the ColorBrewer color tables; also when
the IDL version < 8.3. These color tables are available as values
higher than the regular IDL tables. To specify the Spectral ColorBrewer
color table in an IDL-independent way, it is suggested to use use either
the command:
"COLORTABLE='CB25'" (on the IDL command line) or
"COLORTABLE=CB25" (in a shell or in Python, outside IDL)
or the equivalent command:
"COLORTABLE='25,brewer.tbl'" (on the IDL command line) or
"COLORTABLE=25,brewer.tbl" (in a shell or in Python, outside IDL)
- All PostScript and PDF printouts can now be made using one of the
following paper sizes: ISO A2, A3, [A4], A5, US Letter, Legal, or
Tabloid (A4 is the default when nothing else is specified). The paper
size is set either with the keywords A2, A3, A5, LETTER, LEGAL, or
TABLOID or with the user-parameters 'a2', 'a3', 'a5', 'letter', 'legal',
or 'tabloid'.
- Add support for the new instrument Wise/WIFU, according to the wishes of
A. Levi.
- E3D files are now written with the full header of the original data
intact.
- The p3d main GUI has been updated; mostly on the inside, but also on the
outside:
o Data can now be flux calibrated from the GUI.
o The IFS/IFU selection GUI is rewritten to use several tabs.
o The code is rewritten for increased flexibility (it would now be
possible to add language files, if the community would have such an
interest).
- Fiber position-table files are now read before they are used, and can be
individual to an IFU for the same instrument setup.
- Add rudimentary support for IMACS. Some work is still required with this
rather complex instrument.
o The spectrum viewer (p3d_sv):
Major new functionality
- The spectrum viewer has been renamed from p3d_rss to p3d_sv, where sv
stands for spectrum viewer (rss stands for row-stacked spectra).
- The spectrum viewer can now be used with reduced data cubes of the MUSE
instrument. The amount of available RAM on the computer sets the only
limit to what size can be loaded. The keyword NO_ERRORS can be set to
avoid loading the error or variance image and thereby save on loading
time and RAM.
- Spatial maps of square-shaped spatial elements are now drawn with
regular images, while circular and hexagon-shaped spatial elements keep
using polygons. Using regular images, the spectrum image and the
spatial map are recalculated every time they change; this is made so to
avoid using more memory than the exiting window sizes (enabling the
use of huge MUSE datacubes).
- Add coordinate labels that show the world coordinates (RA, DEC), the
spatial-element id and image row, spatial-element coordinates, and
window coordinates on the status line.
- All draw areas on the first two tabs (not the sensitivity-function tab)
can now be controlled using the scroll wheel - to pan and zoom. Hover
the mouse pointer above the question mark below the respective draw area
on the right-hand side to see help on how to use the mouse.
- Added two reference wavelength bins in the spectrum image that can be
used with the new blinking functionality to blink between the regular
wavelength bin and either one or both of the reference bins. The blink
time can also be configured.
- All draw widgets are now redrawn by p3d instead of being handled by IDL.
- The tool can now be resized; the relative height of the spatial-map area
can be set with a widget slider in the new 'Extra' control panel.
- The code has been recoded for speed and is split into several source
files for increased visibility.
o Line fitting (p3d_ifsfit, p3d_ifsfit_mosaic):
Major new functionality
- Add the tool p3d_ifsfit, which can be configured to fit any number of
emission and absorption lines in IFU data, using any (possible)
wavelength range and combination of separate and blended lines. For
example, both a Gaussian-profile emission line (HII regions) and a
Lorentzian-profile absorption line (stellar population) of Halpha can be
fitted simultaneously.
- Add the tool p3d_ifsfit_mosaic that creates output for presentation from
data of p3d_ifsfit; output to (encapsulated) postscript, screen, or
bitmap files. Options exist to control sizes, number of panels, colors,
axes, fonts, properties, color-bar appearance, etc.
o Numerous minor bug fixes and adjustments to comments throughout the code.
2.2.6 (2014-09-22; Revision 3223) - Serenity
o All tools (in ${p3d_path}):
Improved the documentation, minor and important new functionality.
- Add support for the new instrument ERASMUS-F. This instrument is
configured in a similar way to how VIRUS is done. The instrument uses
six bundles that are split on two detectors, left and right. The
reductions need to use the correct detector, and only one bundle with
19 fibers each is reduced at any one time.
- Add support for some of the IFUs of VIRUS (i.e. not VIRUS-P)...
additional support is added as the IFUs become available (and are used).
- Line-profile plots are now drawn on multiple pages, for instruments
that use many pixels.
- Add the color table "cubehelix" (all its options are available through
keywords in p3d, and p3d_rss):
http://www.mrao.cam.ac.uk/~dag/CUBEHELIX/
o The spectrum viewer (p3d_rss):
Minor new functionality
- The spatial map can now be saved as a bitmap image or to a postscript
file, the output includes a color bar. Formats include TIFF, JPEG, PNG,
PS, and EPS. The size of the image is specified in advance with
droplist widgets in the menu. The output is optionally annotated with
the spatial-element ID and row number.
- Initial support to draw the x-axis with the unit wavenumber or
nanometer, instead of the default unit Ångström.
o The tracing procedure (p3d_ctrace):
Add new functionality.
- Add the new profile function "TophatGaussian" to complement the already
existing types that are used when calculating cross-dispersion profiles
("Gaussian", "Lorentzian", "DoubleGaussian", "Lorentzian/Gaussian"; note
that the name strings differ slightly compared to before, but the older
strings are still acccepted, such as "Lorentz/Gauss"). The new profile
function is now used by default with the VIRUS-P, the VIRUS-W, and the
VIRUS setups.
Note! This is not an optimal profile for these instruments, but it
appears to be slightly better than a Gaussian profile.
o Compilation of the IDL routines for the p3d server (p3d_tool_makevm):
Convenience fix.
- The compilation is now correctly reported to fail if the required IDL
routines are not available in the path.
o Various minor bug fixes and adjustments to comments throughout the code.
2.2.5.1 (2013-12-19; Revision 3053) - Serenity
o Cosmic-ray cleaning (p3d_cr):
Fixed a bug.
- The C-based cosmic-ray cleaning routine is reactivated after the
problematic bug that is mentioned for release 2.2.5 was fixed. The fix
was applied with revision 3025.
o Correct a bug in the C routine p3d_cssl_median.c. Switching from using
IDL_LONG to size_t, in an iterator, I had not checked that the value was
compared with a negative value. The result was that the code crashed.
The bug could be circumvented by not using the scattered-light
subtraction (by using the following statement in the user-parameter
file: "subtractscatteredlight no"). This bug was introduced in revision
2989, and was fixed with revision 3045.
2.2.5 (2013-11-06; Revision 3018) - Serenity
o All tools (in ${p3d_path}):
New instrument setups.
- Added support for the AAT/AAOMega/SAMI instrument. The setup should work
more or less starting with revision 2851 (adjusted the spectrum
extraction with in revision 2912).
- Added support for the VIRUS instrument. This new setup is only intended
to be used as a testing tool for lab data. The setup has initially been
configured for IFUs 11, 18, and 8 (as well as partly 3, 4, and 5), but
support for all 65-80 IFUs is easily added.
o All tools that combine raw-data images:
New functionality.
- The C-based module that combines raw images with the min/max filter
method, p3d_cimcom.c, now works as expected also when there are more
than 20 images. This is how it works from now on.
When the user selects 10 images or fewer - this value is set with the
user parameter imcomb_minmaxgroupn - the images are stacked and the two
pixels with the lowest and the highest values are discarded. Another set
with the lowest and the highest values are discarded if the number of
images is higher than 10, but lower than or equal to 20. The total
number of discarded pixels are as follows (assuming that
imcomb_minmaxgroupn=10), as function of the number of combined images n:
n<11, 2; 10=2780.
o C routines:
Optimization.
- Optimized the C routines p3d_cpixex.c and p3d_cmpd.c.
2.2.2 (2013-05-29; Revision 2771) - Serenity
o All tools (in ${p3d_path}):
Major new functionality.
- All tools in p3d were so far accompanied by their individual binary file
for use with the IDL Virtual Machine. This setup has now been completely
changed. From now on there is only one binary file that contains all
tools - "p3d_tool_server.sav". The binary file is launched as a server
on any number of threads on the computer where p3d is used (default is
all threads minus one).
Thereafter, a new Python-based p3d queue handler manages all p3d IDL
servers that were launched by the user, by submitting jobs to free
servers. Jobs are dispatched by the new tool p3d_dispatch.py, which is
also called by the now obsolete bash scripts in ${p3d_path}/vm/ (those
scripts are still kept to allow already written scripts to be executed
without any changes).
An important feature is that jobs are submitted using either all threads
or just one thread (the queue manages different jobs transparently). It
is up to the user to select an appropriate number of threads. Any tool
that uses multiple threads internally, such as profile fitting or
spectrum extraction, should be launched using one thread. The older bash
scripts have been modified such that p3d_ctrace_vm.sh, p3d_cflatf_vm.sh,
and p3d_cobjex_vm.sh require all threads.
Simultaneously the p3d compiling script, as well as the additional tools
download script are replaced with Python versions of the same tools:
p3d_tool_makevm.sh is replaced with p3d_tool_makevm.py, and
p3d_acquire_xtools.sh is replaced with p3d_acquire_xtools.py.
- Activated the new parallelized C routine that cleans single images of
cosmic-ray hits. For a test with one VIMOS image and 8 threads the
C based cleaning took 16.6s, and with 1 thread 45.8s. The corresponding
time using the IDL-based routine was 41.6s. The routine - p3d_ccr.c -
was mainly written by F. Tabataba-Vakili.
- Activated the new parallelized C routine that combines images using the
min/max method. For a test with three VIMOS images the IDL took 14.13s,
the C routine with one thread 0.335s, and with four threads 0.115s, the
corresponding speedup is thereby 40 and 120! The routine
- p3d_cimcom.c - was written by F. Tabataba-Vakili.
- Activated the new parallelized C routine that calculates pixel-based
cross-dispersion profiles for the optimal extraction when using
p3d_extract_optimal_mox and for viewing the profiles in
p3d_extract_optimal (the entry in the remaining routine p3d_optimal_mpd
was converted when that routine was made a C routine). The routine
- p3d_cpixex.c - was written by F. Tabataba-Vakili.
Minor bug fix.
- All tools in p3d can be used with the new p3d queue server system. Up to
now there was a bug when parsing filenames of keywords. This has been
fixed with revision 2680, but it is still impossible to use filenames
that contain only one starting slash; such as "/file.fits".
- The p3d main GUI tool has been corrected to reload multi-extension
dispersion-mask images properly (when creating a new dispersion-mask
image). The tool now also behaves properly also when the external
program ds9 is not in the path. Thanks to Pieter Gruyters for pointing
at the issues that these two bugs led to. This was fixed with revision
2693.
- The latest VIMOS setup, which is used since 2012.03.15 at the telescope,
has been fixed (bvimos_hr.prm). The cropped detector-section that is
used with HR-blue was used with all grisms, which is in error. This has
been fixed so that HR-blue uses a cropped version, while the other
grisms use the uncropped section. This was fixed with revision 2711.
This issue was addressed by P. Gruyters.
o The RSS to cube conversion routine (p3d_rss2cube):
Fixed a minor bug.
- The spatial element scale could not be specified using the keyword
SPAXELSCALE, but instead had to be provided in the user-parameter file.
This was corrected with revision 2667.
o Dispersion-mask creation routine (p3d_cdmask):
Fixed a minor bug.
- In p3d_wavecal_dispmask_gui there is a statement that compares an
integer with a string. Apparently this works great in IDL 7.1 x86_64,
but not at all in IDL 7.1.1 x86_64. This has been corrected with
revision 2685. This bug was spotted by C. Herenz.
- p3d_wavecal_dispmask_add_spec has had a bug since revision 2402 (this
was when the scaled polynomial fit of the wavelength array was added)
that prevents calibration-fiber data from being added for instruments
that use such; in this case FLAMES/Argus/IFU1/IFU2. This has been
corrected with revision 2729. This bug was found by S. Kamann.
o Spectrum extraction (p3d_cdmask, p3d_cflatf, p3d_cobjex):
- Reducing VIMOS data and not having /NOOBCHECK set there was a bug that
prevented the OB id number to be parsed in dispersion-mask images that
use the new extension-based format. This was corrected with revision
2761. This bug was found by A. Castrillon.
o Flux calibration (p3d_fluxsens, p3d_fluxcal, p3d_rss):
Fixed a minor/major bug.
- A sensitivity-function file can be saved using either p3d_fluxsens or
the sensititivity-function tab in p3d_rss. In the latter case the
wavelength-array keywords CRVAL, CDELT, and CRPIX, were so far not
written for the first dimension (as they have to be for p3d_fluxcal to
work). (Note! This only concerns instruments where the second axis is
the dispersion axis.) This has been corrected with revision 2678.
This bug was spotted by P. Gruyters.
o Updated support for some instruments:
- Integral (/INTEGRAL, /OLDINTEGRAL):
On a request of M. de Bont and K. Exter, the instrument-keywords were
adjusted so that keyword parameters containg a dash are read properly.
This makes possible the creation of a sensitivity function for this
instrument (the spectrum must still be corrected for flux lost between
fibers, but it is a start).
The old-Integral instrument-keywords file was updated in revision 2719,
previous revisions (since a long time back) are unable to extract
airmass values properly.
- VIRUS-P (/VIRUS):
After a request from E. Freeland (Stockholm Univ.) a dead-fibers file
was added for fiber bundle 2. Note that the fiber status have evolved
from the initial state (where everything worked) to now. Hence, it may
be necessary to edit a copy of the new dead-fibers file for intermediate
data.
o The p3d main GUI (p3d_tool_gui):
Bug fix!
- Switching to the new extension-based format for dispersion-mask images,
I had forgotten to fix the checking of which quadrant is used with
VIMOS files. This was fixed with revision 2621.
o Various minor bug fixes and adjustments to comments throughout the code.
2.2.1 (2013-01-21; Revision 2611) - Serenity
o All tools (in ${p3d_path}):
New instrument setup supported.
- Added support for the latest VIMOS setup, which is used since 2012.03.15
at the telescope. The new setup is used with the BVIMOS keyword, and the
bvimos_hr.prm instrument-parameter file. Only half of the CCD is used
with the new HR-blue configuration, compared to the older configuration.
This support was added with revision 2599.
o The dispersion-mask creation (p3d_wavecal_dispmask_gui):
Corrected a major bug.
- In previous updates to use fits-file extensions also with the
dispersion-mask file a zero-extension header was forgotten in the
first-extension header. It is impossible to save a dispersion mask with
this bug. This has been corrected with revision 2605, and is the cause
for this release.
o Various minor bug fixes and adjustments to comments throughout the code.
2.2 (2012-12-13; Revision 2578) - Serenity
o Flux calibration (p3d_fluxsens, p3d_fluxcal, p3d_rss):
Now ready to be used.
- With this release all flux-calibration routines, as well as related
parts in the master user-parameter file, are thoroughly documented and
tested. The flux calibration procedure is intended to be the following:
at first, a sensitivity function is created using observed spectra of a
standard star with the tool p3d_fluxsens. Thereafter extracted science
spectra are flux calibrated using p3d_fluxcal.
p3d_fluxsens is meant to be an automatic tool that avoids any user
interaction, an interactive version is instead available in the spectrum
viewer; all options of p3d_fluxsens are available through various
widgets under the sensitivity-function tab. To mention some features,
the sensitivity function is fitted to an observed (and sky subtracted)
spectrum using one of eight kinds of interpolation: regular polynomial,
Chebyshev polynomial, Legendre polynomial, three varieties of (cubic)
splines, and two varieties of quadratic piecewise interpolation; the
polynomial order can be chosen in the range 0-20.
One feature is only available in the spectrum viewer, the masking of
unwanted calibration standard-star data band-passes/entries. These
entries are easily rejected by clicking on them one by one with the left
mouse button, or dragging out a box with the left mouse button in one of
the plot panels. Edited data can be saved to a plain-text file for
continued use with other data or with p3d_fluxsens. Masked entries are
unmasked in the reverse order they were masked by pressing the right
mouse button repeatedly.
The polynomial fits were implemented and checked to work with revision
2415.
o The spectrum viewer (p3d_rss):
Interesting new functionality
- The spectrum viewer is able to read both RSS images and data cubes that
were created using the CALIFA project data-reduction pipeline. To view
such data files it is necessary to start the spectrum viewer from the
command line. Here is an example:
p3d_rss_vm.sh CALIFA_cube.fits.gz parfile=ppak4k.prm
Data cubes are converted to an RSS image, by the p3d spectrum viewer,
before the graphical user interface is launched.
o The science-object extraction routine (p3d_cobjex):
Corrected a major bug.
- Corrected a serious bug that affects some reductions using re-alignment
of the dispersion solution with telluric lines. Specifically, the bug
affects anyone using a dispersion mask that is poorly determined for the
lowermost pixels, which might be the case with data where there are no
calibration arc lines in the lowermost part of the wavelength range. The
correction was made to the routine p3d_wavecal_dispersion_correction.
This bug fix involves two issues. Since very long it was forgotten to
update the arrays with relative pixel sizes after offsets were applied.
The second issue is the following. The resampling wavelength array was
always calculated starting with the first pixel in the output array. If
the dispersion mask was poorly determined at the lower wavelength range
it was often impossible to create an accurate wavelength calibration, in
particular when the dispersion solution was re-aligned using telluric
lines. An example of data where this might happen are VIMOS/HR-orange
data, where there are no arc lines shortwards of about 5500Å. Wavelength
calibrated data using a dispersion mask that is poorly determined for
lower wavelengths would result in the wavelength array being shifted one
pixel at all wavelengths, of affected spectra.
The resampling wavelength array is now calculated starting from the
middle output pixel, by default. The starting pixel can be chosen with
the keyword RESAMPLE_STARTPX or the user parameter 'resample_startpx'.
This issue was completely corrected with revision 2349.
o Accuracy of dispersion masks (p3d_cdmask):
Major modification - significantly improved the accuracy.
- Polynomial fits have been drastically improved, by rescaling abscissa
arrays to always use the range [-1,1]. This way it is possible to use
high-order regular polynomials with good results; when the order is
about <15. For dispersion masks this adjustment has required saving of
additional data, with the lower and the upper pixel values that were
used in the scaling, in the dispersion-mask file. All tools that read
such dispersion-mask files automatically recognize file formats using
both the new format and the older format (which is unable to perform the
rescaling due to the missing data). The polynomial fits were implemented
and checked to work with revision 2410. The spectrum-viewer buttons in
the main GAUI panel were updated to work with revision 2478.
o All tools (in ${p3d_path}):
Major modifications.
- Added support for the instruments VIRUS-W and Integral (new CCD, all
four IFUs: SB1-SB3 and equalized). These instruments use files where
data of several blocks are contained in the same file.
- Cosmic-ray cleaning: Added the keyword GAUSSKERNELSIZE and the user
parameter 'crgausskernelsize' to the tools p3d_ctrace, p3d_cdmask,
p3d_cflatf, p3d_cobjex, and p3d_cr.
Also added some instructions on how to make the cosmic-ray cleaning
mimic the results achieved in Husemann et al. (2012; PyCosmic).
Basically, one must set the keyword GROWRADIUS to zero.
- Repaired the support for the FLAMES mini IFUs (IFU1 and IFU2). Due to an
improvement in revision 2095 to handle dead fibers correctly the routine
was unable to trace spectra at all. The "improvement" has been commented
out in revision 2508, and the routine works as it should again.
- Up to now (nearly) all output data were written to separate files. Now
the keyword ALLINONE, as well as the user parameter "allinone", has been
added to all tools. The default value of this keyword and user parameter
is that allinone is set. This means that (nearly) all output data are
written to the same file, using consecutive extensions. To have the same
behavior as before it is necessary to unset this keyword.
All tools of p3d handle files using both approaches, in any mixture.
For example, if parts of the files where created with ALLINONE unset, it
is possible to continue with ALLINONE set.
The names of the different extensions are explained in the master
user-parameter file.
The purpose of this change was to allow the resulting data cube files to
be used with the CASA viewer (which requires additional header
keywords), but also have a smaller number of resulting output files. All
written fits files are now (meant to) conform to the FITS standard.
Multi-extension fits files are easily examined using the program ds9
using commands such as the following examples:
ds9 file.fits[DATA] ::
To view data in the main extension.
ds9 file.fits.gz[ERROR] ::
To view data in the error extension.
ds9 file.fits.gz[DATA] file.fits.gz[MASK] ::
To open the main data and the mask image in separate frames.
This new mode was confirmed to work fully with revision 2478.
o The use of prescan and overscan regions (p3d_c?????):
Corrected a serious bug (that should affect very few)
- Prescan regions of PMAS/4kx4k data were never normalized to use the mean
gain, as the rest of the data are, including overscan data. Corrected
this bug by adding a correction factor to the block combined image, all
reductions making use of the prescan regions must be redone. This was
fixed with revision 2465.
- Added the possibility to specify bias values explicitly using the
BIASCONSTANT keyword, or the 'biasconstant' user parameter. This support
was added and checked to work with revision 2559.
o The master-bias tool (p3d_cmbias):
Minor new functionality
- Added the keyword NODIALOG, and the new user parameter 'biasnodialog',
which avoids the widget dialog that asks if a single image should be
smoothed. By default NODIALOG is set, and no smoothing is performed.
This support was included with revision 2462.
o Bias-subtracted images (p3d_ctrace, p3d_cdmask, p3d_cflatf, p3d_cobjex):
Minor new functionality
- Added the keyword SAVEBIASSUB, and the new user parameter 'savebiassub',
which save the (optionally combined) bias-subtracted raw-data image to a
separate file. This could be a useful option if one wants to check the
quality of the bias subtraction, in particular when using a bias image
that was created using prescan and overscan data. This support was
included with revision 2482.
o The RSS to cube conversion routine (p3d_rss2cube):
Fixed a minor bug.
- Corrected a bug that affects output saturated-pixels data cubes. So far
such data cubes were never created, instead the bad-pixels mask image
was written again, if a bad-pixels mask image was present.
o The p3d development version repository.
Changed location
- Until 7.12.2012 the p3d repository address was:
http://p3d.svn.sourceforge.net/svnroot/p3d
After 7.12.2012 the repository address was changed due to software
upgrade at sourceforge.net. The new addresses are:
svn://svn.code.sf.net/p/p3d/code
or
http://svn.code.sf.net/p/p3d/code.
Directories pointing at the old directory can be used for as long as
there is no need or wish to update p3d. The new repository is
incompatible with the old repository, which is why the old one must be
replaced with a copy using one of the two new repository addresses
before p3d can be updated anew.
p3d developers have two choices to access the repository: either using
the svn+ssh protocol or the https protocol. The exact command lines are
provided at the p3d web site when one is logged in (with repository
write access).
o Various minor bug fixes and adjustments to comments throughout the code.
2.1.4 (2012-09-17; Revision 2250) - The Warbird
o The spectrum viewer (p3d_rss):
Major new functionality
- Spatial elements used to be selectable (deselectable) only by clicking
on them one by one with the left (right) mouse button, or selecting
either all or no elements. With version 2.1.2 it became possible to also
select spectra of dead, low-transmission, and sky fibers.
Now it is in addition possible to select spectra (spatial elements)
using a widget tool that defines one or several regions of interest
(ROIs). These ROIs are defined interactively using either rectangles,
ellipses, polygons, or freehand drawn curves. The spatial elements that
fall roughly within all setup regions are selected upon closing the ROI
tool. The same regions are used anew whenever the ROI tool is opened
again in the same session. It is also possible to save the ROIs inside
the ROI tool, and later load then from the spectrum viewer for reuse.
Additionally, it is possible to save and load created spatial-element
bin map files. These map files are plain-text files that contain as many
bins as required; each bin is assigned a positive integer, which is
greater than zero. The spectrum viewer allows viewing and creating one
such bin at a time. The bin number is changed using a droplist widget.
o Regarding the instrument support for PMAS/4kx4k CCD (both PPAK and Larr):
- When files are downloaded from the Calar-Alto archive they have
filenames with a date and time string in them. This time string does not
work properly with previous versions of p3d, since p3d cannot determine
the names of all files of all blocks automatically. This has now been
corrected with revision 2196, which uses the original filename that is
available in the header-keyword keyword ORIG_FILENAME FILENAME
(of PMAS/4kx4k CCD files).
o The dispersion-mask creation (p3d_wavecal_dispmask_gui):
- At some point the feature was deactivated, which displays a help for
each button in the tool. It has now been reactivated again, with
revision 2185.
o The use of prescan and overscan regions (p3d_cr):
- There has been a small bug that would prevent prescan and overscan
bias images from being used with images that are to be cleaned of
cosmic rays. This was fixed with revision 2230.
o Combining multiple files with extracted data (p3d_cexposure):
Corrected a serious bug.
- An error was included with the new feature using bad-pixels mask images
of revision 2126 (which is included in release 2.1.3). The error of the
first file was never squared, which results in final errors that should
be squared to be more correct. This has been corrected with revision
2152.
o Various minor bug fixes and adjustments to comments throughout the code.
2.1.3 (2012-06-28; Revision 2134) - The Warbird
o The spectrum extraction
(p3d_cdmask, p3d_cflatf, p3d_cobjex, and p3d_cexposure):
New functionality
- An output bad-pixels mask image is written if a bad-pixels mask image or
a cosmic-ray mask image are provided, or if the cosmic-ray mask image is
calculated. Each spectrum element in the output mask contains the total
number of masked pixels in the bad-pixels mask image. The output mask
images are used by p3d_cexposure when extracted images are combined.
- The resampling of extracted spectra from pixel to wavelength units is
now done more carefully to completely avoid Not-a-Numbers in the output
spectrum image.
o Bias subtraction:
Minor new functionality
- New-generation-controller instruments (ERA2) can now also use prescan or
overscan regions to calculate a bias image.
o Saving data in the e3d format:
New functionality
- All routines writing science-object row-stacked spectrum (RSS) files now
also write e3d-formatted files. This option is activated either using
the keyword SAVEE3D or the user parameter savee3d. The following four
routines can now do this, in addition to p3d_cobjex: p3d_fluxcal,
p3d_cvimos_combine, p3d_darc, and p3d_cexposure.
o Removing of cosmic-ray hits in single raw-data images (p3d_cr):
Minor new functionality
- Added the new keyword and user parameter GROWRADIUS, which allows a
definition of the width of the frame of pixels around each pixel that is
masked as a cosmic-ray hit. Additional cosmic-ray hits are identified in
these neighbor pixels. In this case the first criterion is used as
S'>sigfrac*sigclip. The growing radius has previously always been set to
1, but can now also be set to 0 and 2. The default value is 1.
o The spectrum viewer (p3d_rss):
Minor new functionality
- The (approximate) spectrum-viewer window size can now be specified using
the keyword WINXYSIZE. Allowed values are:
[1000,750]<=WINXYSIZE<=[max(xsize),max(ysize)].
The p3d main GUI provides a set of pre-defined values in the Options
menu. Only those options are available that are smaller than or equal to
the screen size.
- Since revision 1982 it is no longer necessary to specify the IFU angle
header keyword (KWANGLE) when the PARFILE keyword is set. The correct
parameter name is now read automatically from the instrument-specific
keywords file. Note that this new feature only affects sessions that are
not based on use of the p3d main GUI.
Note! For instruments that do not use this option it is necessary to
update to revision 2116, since previous revisions contain a bug that
does not allow the spectrum viewer to be opened.
o The fiber position tables for FLAMES/ARGUS, FLAMES/IFU1, and FLAMES/IFU2
now also contain positions for the sky fibers. These sky fibers can be
viewed in the spectrum viewer when the keyword ANDSKY is set, or the GUI
menu option "Spectrum viewer - view sky fibers" is set to "yes". The sky
fiber positions do not correspond to the actual position on the sky.
o Various minor bug fixes and adjustments to comments throughout the code.
2.1.2 (2012-04-23; Revision 1948) - The Warbird
o All tools (in ${p3d_path}):
Improved the documentation, minor and important new functionality.
- Added support for the new instrument ERA2. This instrument uses files
where data of several blocks are contained in the same file. It has
been necesary to adjust p3d quite significantly to support this format.
Hence, some problems might occur elsewhere.
- Added support to use prescan and overscan regions to create a bias
when no master-bias image is specified. The region to use is simply
specified using the following parameters:
biaspx :: uses the x-axis prescan region. Specifically, a median value
is calculated for each pixel on the y axis in the x-axis
prescan region. The same array is used with all y-axis columns
of the data that are bias subtracted.
biaspy :: uses the y-axis prescan region. Specifically, a median value
is calculated for each pixel on the x axis in the y-axis
prescan region. The same array is used with all x-axis rows
of the data that are bias subtracted.
biasox :: uses the x-axis overscan region. Specifically, a median value
is calculated for each pixel on the y axis in the x-axis
overscan region. The same array is used with all y-axis
columns of the data that are bias subtracted.
biasoy :: uses the y-axis overscan region. Specifically, a median value
is calculated for each pixel on the x axis in the y-axis
overscan region. The same array is used with all x-axis rows
of the data that are bias subtracted.
biasconstant :: Set this keyword to use a constant value as a bias
value; otherwise an array is used. All the regions that are
specified using the keywords - or user parameters - biaspx,
biaspy, biasox, and biasoy can be used when this keyword,
or user parameter, is set; otherwise only one of them.
Default values are specified in the instrument-parameter files of the
following instruments: PMAS/Larr, PMAS/PPAK, VIMOS, FLAMES, GMOS. For
as long as the raw data contains either region any region can be used.
The use is specified either using a keyword in p3d_cr, p3d_ctrace,
p3d_cdmask, p3d_cflatf, and p3d_cobjex, or using the user-parameter
file.
Note! These regions are only used when no master-bias image is
specified.
This functionality is so far preliminary, and not fully documented. Full
support will be included starting release 2.2.
- The routine documentation has been significantly revised for clarity.
p3d_cobjex: improved the documentation so that it more accurately
describes how the science-object spectrum extraction works, including
information on how to use the instrument-flexure correction, as well
as scattered-light subtraction features.
p3d_cflatf: improved the documentation so that it more accurately
describes how flat-field data are treated, including a section on how
to combine a lamp flat-field image with a twilight flat-field image.
p3d_rss: Cleaned up the documentation, removing a part belonging to
p3d.pro.
- The instrument-parameter file PARFILE can now be specified without the
full path as a prefix. I.e., instead of
PARFILE=!p3d_path+'data/instruments/vimos/nvimos_hr.prm'
it is now possible to write
PARFILE='nvimos_hr.prm'
the file is searched for automatically in the p3d data subdirectory.
o Flux calibration (p3d_fluxsens, p3d_fluxcal):
New functionality.
- Added first versions of routines that allow flux calibration. These
routines are only preliminary in this release. More complete
functionality, and documentation, will be available in the future
release 2.2.
o The science-object extraction routine (p3d_cobjex):
Added some new functionality.
- Added the option to save the final normalized flat-field image. This
image is now stored when then keyword SAVEFINALFLAT is set, or the
user parameter ff_save_final_image is set to 'yes'.
- The flat-field transmission file of an extracted twilight flat-field
image is now properly used to normalize data for the fiber-to-fiber
transmission. To use this feature: first set the TRANSMISSION keyword
before running the p3d_cflatf tool (also see the new keywords PCUTLOW
and PCUTHIGH), and then extract the data using p3d_cobjex, setting the
NF2F keyword.
- Added the option that the wavelength arrays of individual spectra can be
corrected individually, when using one or several sky-emission lines.
Previously all wavelength arrays of the dispersion mask were always
offset using one median value calculated for all used sky-emission lines
and spectra. This approach has been found to be particularly poorly
suited with the multi-detector instrument VIMOS.
Note that the default method now is to use an individual correction.
A linear fit is now also made of the pixel offset between two or more
sky-emission lines. It is up to the user to select appropriately spaced
lines.
Additionally, the wavelength offset is weighted with the pixel size,
since the offset due to flexure takes place in pixel space, and not in
wavelength space. This treatment allows a more correct offset. See the
keyword ONESKYOFFSET, as well as the user parameter 'oneskyoffset', for
more information.
- During the dispersion correction the errors are now ensured to be
non-zero - there are no more NaNs in the resulting output images.
o The flat-field preparation routine (p3d_cflatf):
minor new functionality.
- Improved the documentation so that it more accurately describes how
flat-field data are treated, including a section on how to combine a
lamp flat-field image with a twilight flat-field image.
- Updated the handling of the fiber-to-fiber transmission file, so that it
will work to correct data using, for example, both a lamp flat and a
twilight flat.
- Added the keywords PCUTLOW and PCUTHIGH which can be set to use only a
part of the CCD before calculating the fiber-to-fiber transmission
array. It might be useful to change one of these values if the CCD
sensitivity is lower on one side. For example, with the VIMOS HR-blue
grating PCUTLOW should be set to something like PCUTLOW=32.0.
o The dispersion-mask creation tool (p3d_cdmask):
New functionality.
- Added the keyword LINELISTLIMIT, as well as the user parameter
'linelistlimit_dm', which allows an automatic selection of only those
entries that are brighter than the selected value. This option is only
available if the arc-line list file contains at least two columns, where
a relative intensity is specified in the second column (as a decimal
value). All entries are used, which do not contain a second column.
o The spectrum viewer (p3d_rss):
New functionality.
- Added the option to save either the currently selected spectrum (as
before) or both the currently selected spectrum and the subtracted
spectrum to the same file. Both options are available in the File menu.
- Added the option to divide all spectra with the selected spectrum, to
the previous option of subtracting the selected image from all spectra.
- Hexagon-shaped elements are now shown and selected properly in the
spatial map. Such elements are used with the GMOS-S and the GMOS-N IFUs.
- If the new keyword PARFILE is specified it is no more necessary to
specify the argument POSFILE, or the dispersion axis DAXIS.
- In addition to selecting " all" and "none" of the spatial elements in
the spatial map view, it is now possible to also select "a- t" (all but
low-transmission elements), " t" (low-transmission elements), "a-dt"
(all but dead and low-transmission elements), "a-d " (all but dead
elements), " d " (dead elements), " dL" (dead and low-transmission
elements), "a-s " (all but sky elements), and " sky" (all sky elements).
Of course, these changes only apply to the instruments which have such
low-transmission, dead, or sky fibers. It is, additionally required to
specify the PARFILE argument when starting p3d_rss.
o Various minor bug fixes and adjustments to comments throughout the code.
2.1.1 (2012-02-06; Revision 1615) - The Warbird
o Combining multiple files with extracted data (p3d_cexposure):
New functionality.
- Added a new tool that combines mutliple extracted exposures that are
taken illuminating the exact same field on the sky. The supplied data
are at first checked for consistency in image size and equal values on
the wavelength calibration parameters CRVAL and CDELT. Thereafter the
files are either averaged, weighting with the exposure time of the
individual exposures. Alternatively, setting the /SATURATED keyword it
is expected that all files have a different exposure time. In this case
the longest exposure is used as reference. Portions of individual
spectra, where a mask indicates that a pixel was saturated, are then
replaced with scaled data of the same spectrum and portion of a file
using a shorter exposure time. The routine accepts any number of files.
The routine produces one output file with merged data, and a file with
corresponding errors if all the error files of the input images are
available.
o Handling of saturated pixels in the raw data:
New functionality.
- From now on a new mask file is written along with extracted data
whenever the keyword SATMASK is set in the routines p3d_cdmask,
p3d_cflatf, and p3d_cobjex. This mask file indicates, for each element
in the spectra, if any of the used raw-data pixels were saturated. Only
the center part of the cross-dispersion profile is considered for
optimally extracted spectra. This functionality will be used to combine
extracted spectra that use different exposure times.
o The science-object extraction procedure (p3d_cobjex):
Minor bugs affecting the functionality.
- A cosmic-ray mask file that was provided as an input keyword (CRMASK)
was never identified.
- If a run would contain several groups, where some would require the
combination of images, and some others not, while /crclean was set, then
the combined images would not be properly reduced.
o The dispersion-mask creation procedure (p3d_cdmask):
Major bug affecting PMAS 4kx4k reductions.
- A new method to handle arc-line entries in vignetted regions of the
PMAS 4kx4k CCD, both Larr and PPAK, was introduced in version 2.1.
However, the initial version is completely filled with bugs. This has
been corrected with revision 1557, also the log-file output has been
improved. Additionally, the vignetted arc-line entries (wavelengths) are
indicated in the "Check the final fits" tab, for the affected spectra.
Note! To continue to use version 2.0, when creating a dispersion-mask,
it is necessary to comment the "vignettedfile_dm" entries in the
instrument-parameter files larr4k.prm and ppak4k.prm.
Minor issue affecting any data with calibration spectra.
- Corrected the display of the affected spectrum (number) in the
"Check the final fit" tab. The number is now showing the correct
spectrum also for data that contain calibration spectra.
o Compatibility with PINGSoft:
Added functionality
- Added PINGSoft-formatted fiber-position table files (to
${p3d_path}/data/tables/pingsoft_postables/), which can be used with the
reduced outcome of p3d to analyze the data using the IFS analysis tool
PINGSoft: http://www.ast.cam.ac.uk/ioa/research/pings/html/index.html.
These new tables were introduced with revision 1602.
o Reduction work with GMOS:
Minor issue affecting cosmic-ray cleaning with GMOS data
- Fixed the cosmic-ray cleaning routine so that it works with the
multi-extension data of GMOS. It did not until now. This was fixed with
revision 1577.
o Reduction work with MPFS:
Major issue affecting any accurate spectrum extraction
- Updated the instrument parameters so that it is possible to calculate
accurate cross-dispersion line profiles for optimal extraction by
default. This was made with revision 1584.
o Reduction work with VIRUS-P:
Major bug affecting VIRUS-P reductions.
- Fixed a bug that concerned how the parsing of the CCD binning parameter
CCDSUM. The bug caused all VIRUS-P reductions to not work, since quite a
while. This was fixed with revision 1572.
o The (automatic) feature to compile the C-language source files...
Minor bug affecting those with a src/ directory unavailable for writing
- Corrected the minor bug (a forgotten path separator) so that it works
again.
o Various minor bug fixes and adjustments to comments throughout the code.
2.1 (2012-01-16; Revision 1529) - The Warbird
o Correction for differential atmospheric refraction (p3d_darc):
- Added a new tool that corrects extracted science images for differential
atmospheric refraction. The new tool works out-of-the-box with all the
IFUs that use square-shaped spatial elements (VIMOS, FLAMES, PMAS/Larr,
MPFS, and SPIRAL). In some cases the required physical properties are
unavailable in the data header. In that case they must be entered by
hand.
- The routine allows the user to choose between three implemented
approaches to calculate the offsets. The results of all three approaches
are collected in a plot, which allows the user to choose the most
accurate method as he or she believes is more correct.
- The offsets are calculated using the wavelength-dependent
refractive-index equation of Ciddor (1996; as well as Edlén 1966).
- A second approach uses a height-integrating approach to calculate the
refractive-index that comes from the Starlink project. The routine
integrates the refractive index through the trophosphere and the
stratosphere.
- A third method uses an empirical fit of a two-dimensional Gaussian
profile to the data itself to calculate the DAR-caused offsets on both
the X and the Y axis.
- Most of the features of the DAR-correction routine of J.Walsh (ESO) have
been included with his permission.
- Please note that the calculated errors are off as correlated errors are
completely ignored (at the moment).
o Removing of cosmic-ray hits in single raw-data images (p3d_cr):
- Added a new tool that cleans single raw-data images of cosmic-ray hits.
The new tool works with all raw data that are used with p3d.
- The tool uses an IFS-adapted version of the L.A. Cosmic IDL routine of
J.Bloom (Berkeley); this adaptation has been made with the permission of
both P.van Dokkum (Yale) and J.Bloom. The modifications have been
implemented partly according to the suggestions of B.Husemann (AIP).
- By default the routine only produces a cleaned image and a cosmic-ray
hit mask. However, if the keyword WRITEALL is set then intermediate
products are also written to the disk at each iteration; these products
allow a close inspection of the quality of the cleaning procedure, and
may also require huge amounts of disk space.
- The cosmic-ray mask is used as a bad-pixel mask during (optimal)
spectrum extraction. Thereby the, so-called, weight of masked pixels are
set to zero - i.e. they are excluded from the calculations.
- The routine can be called separately, but is also available from
p3d_ctrace, p3d_cdmask, p3d_cflatf, and p3d_cobjex, when the new keyword
/CRCLEAN is set.
o The spectrum extraction:
- The resampling of the extracted spectra from pixel to wavelength units
can now be done using either a one-dimensional drizzling algorithm, or
the original linear-interpolation approach. The error calculation of the
original linear-interpolation method has been improved. The older less
accurate errors are still available for comparison; through the use of
the ORIGINALERRORS keyword.
- The multi-profile deconvolution method that uses a band-diagonal matrix
has been fully ported to C for fast and parallelized execution. The new
routine, which has been written by F. Tabataba-Vakili at AIP, compiles
and executes automatically; together with the other C routines.
o The RSS to cube converter (p3d_rss2cube):
- Added a new tool that converts the extracted images that p3d writes into
cubes, which may be easier to work with. This routine is only available
as a stand-alone tool, and also only works with the IFUs that use
square-shaped elements, i.e. PMAS/Larr, SPIRAL, VIMOS, FLAMES, and MPFS.
o The scattered-light subtraction:
- The scattered-light subtraction can now be switched on and off using a
keyword to the call of p3d_cflatf and p3d_cobjex.
o Regarding the instrument support for VIMOS and FLAMES:
- Due to an extremely serious bug in the routine that reads the gain and
the readout noise values from the data header, all errors calculated for
these two instruments so far have been wrong - they have been too small.
o Regarding the instrument support for VIMOS:
- Corrected a bug that would prevent a proper extraction of data that were
sampled using the medium resolution setup (the DETSEC parameter was
improperly read from the instrument-parameter file).
o Regarding the instrument support for PMAS/4kx4k CCD (both PPAK and Larr):
- The new PMAS 4kx4k CCD has a region in all four corners of the CCD that
are vignetted. When creating a dispersion-mask image, any arc line
inside of these vignetted regions are now automatically excluded from
the chosen selection of line-list entries before the arc-line positions
are fitted with a polynomial. This way it is thought that the wavelength
calibration can be made more accurate in affected spectra. Note that a
exact definition of the vignetted region is difficult to make. If you
think that you have a better solution on how to define the extent of
the vignetted regions, then please, have a look at the two files
ppak4k_vignet.dat and larr4k_vignet.dat.
o The spectrum viewer (p3d_rss):
- The spatial maps are now saved as two- or three-dimensional image arrays
instead as of an unordered one-dimensional array (which is how has been
until now). In the new file each pixel corresponds to a spatial element.
This feature is now only available on instruments with square-shaped
spatial elements, i.e. PMAS/Larr, SPIRAL, VIMOS, FLAMES, and MPFS.
o The instrument- and user-parameter file parser (p3d_misc_read_params):
- The routine did not return the correct flag if a parameter was only
found in the user-parameter file and not in the instrument-parameter
file. For example, if crval is only set in the user-parameter file then
it would always be reported as not found. This has been corrected with
revision 1414.
o Added the (automatic) feature to compile the C-language source files in a
temporary directory on the local computer. Up to now the compilation was
always done in ${p3d_path}/src/, which would not work on a central
installation.
o Various minor bug fixes and adjustments to comments throughout the code.
2.0.3 (2011-10-13; Revision 1181) - The Swashbuckler
o Corrected a problem that occasionally would prevent the spectra
from being properly traced. The story is:
The first part of the tracing procedure locates peaks in the
data. The routine (p3d_tracing_findspec0) tries to find a
resonable value on what is a minimum value in non-peak areas.
It could fail if the bias level varies across the image, as
it does with the PMAS 4kx4k CCD. Consequently the master-bias
image is now subtracted from the continuum image, before the
latter one is used to trace the spectra. This solution seems
to work.
This is a problem that is more likely to occur with a
multi-file instrument such as PMAS using the 4kx4k CCD;
because, the bias level differs across the files.
The problem was reported on Oct 13 2011, by Manu Nuñez.
o The VIMOS extracted-images combination routine; p3d_cvimos_combine:
Corrected a serious bug that would prevent the correct use of an input
renormalization correction-array file (the keyword CORRFFILE). Also
replaced the calculation of a mean telluric-line flux with a median.
Thereafter the fluxes are normalized again using the maximum
median-value quadrant.
2.0.2 (2011-09-28; Revision 1150) - The Swashbuckler
o Regarding the instrument support for VIMOS:
- Added a new feature to the combination routine (p3d_cvimos_combine).
Flux-calibrated data are now re-normalized using a telluric line. This
final normalization corrects for smaller inaccuracies in the data that
are left after the flux calibration (which is performed using, e.g.,
IRAF). This new re-normalization can be switched off, and the telluric
line that is used can be chosen freely (or automatically). The
re-normalization can be done both per spectrum and per quadrant.
- The telluric-line fits are now illustrated in a plot that includes
the raw data and the fit. One plot is created for each spectrum and
each detector. The file is compressed automatically to save space.
o The science-object extraction (p3d_cobjex):
- The telluric-line fits of the sky alignment procedure, which is applied
with the application of the dispersion solution, are now illustrated in
a plot that includes the raw data and the respective fit. One plot is
created for each spectrum and each detector. The file is compressed
automatically to save space.
o The spectrum viewer (p3d_rss):
- Added a colorbar for the spectrum image and the spatial image.
- Corrected a format mix-up when saving a mean or total spectrum. Also
corrected a serious bug affecting the error calculations of the
"sky"-subtracted spectrum.
- Corrected the annotation of the "viewed" wavelength in the spectrum
plot.
- The wavelength-range zoom value and the wavelength-range shift value
are now reset to the initial values when the 'Reset' button is pressed.
o Corrected a bug in the object-extraction routine p3d_cobjex, which could
appear if the GROUP keyword was used together with flat fielding. The
result could be seen as (kind of) randomly rearranged spatial maps
(fixed with revision 1054).
o Corrected the World Coordinate System header entries for the
cross-dispersion axis. Now the data are properly labeled in ds9. /PW
o Corrected the e3d output; due to an (old) -1 image-row offset the fiber
position table inthe e3d file was always wrong (corrected with
revision 1048).
o Cleaned up all routines of the trace/disp.mask/object shift reference
files. These were never used, and are not required anymore
(fixed with revision 1056).
o Various minor bug fixes and adjustments to comments throughout the code.
2.0.1 (2011-08-18; Revision 1028) - The Swashbuckler
o After creating a trace mask, fitting line profiles (also the trace-mask
creation routine), or extracting spectra (using any of the optimal
extraction methods), p3d now -always- creates a postscript printout.
This printout can be used to check if the (respective) fit was
satisfactory. The printout paper-size (US letter/a4) and the number of
wavelength bins that are shown can be set in the user parameter file.
The output file is compressed using bzip2, if this command exists in the
path; if not then the file is left as it is.
o Corrected a bug in the C routine p3d_cssl_median.c. The third dimension of
the input arrays arr and data was improperly accessed. This bug was
spotted by F. Tabataba-Vakili.
o Corrected an issue in p3d_flatfield.pro, which could, eventually, result in
an all-zeros flat-fielded object spectrum image.
2.0 (2011-06-17; Revision 1011) - The Swashbuckler
o Regarding the instrument support:
- VLT/UT2/FLAMES/(ARGUS, IFU1, IFU2) (/FLAMES)
Corrected the tracing procedure for the mini-IFUs. Francesco D'Eugenio
informed us of this inconsistency. Fibers 192, 229, and 278 (IFU1) and
192 and 303 (IFU2) are dead, as well as the last (off-chip) fibers. This
was corrected with revision 877 (tracings using all previous revisions
are consequently wrong).
o Recentering the calculated cross-dispersion line profiles on object data:
- Due to flexure data of most instruments are shifted on the CCD as the
time progresses (as a function on where on the sky the telescope is
pointing). Starting with Revision 897 p3d now attempts to correct for
such offsets in the cross-dispersion direction, whenever optimal
extraction is used and the parameter recenterprofiles is set to yes.
These offsets are normally very small, but even an offset of 0.10 pixel
can have a strong influence on the extracted intensities; the influence
becomes stronger the closer the spectra are arranged on the CCD. There
are three methods available to calculate these offsets: either the data
are smoothed (median-filtered) on the dispersion axis before the line
profiles are fitted anew (for one or a few pixels on the dispersion
axis), or the routine first searches for the brightest telluric line in
the data, before that line is fitted (this approach is only available if
a dispersion-mask image is provided as input to p3d_cobjex), or an
offset is provided explicitly. The recentering can be configured either
through the user parameter file (all settings), or through the RECENTER
keyword to p3d_cobjex (only the basic setup - which normally is enough).
o The fiber-flat field normalization (p3d_flatfield).
- Starting with revision 966 any dead, unused, or low-transmission fibers
are ignored when calculating a mean spectrum, which is used to
normalize the flat-field data using /NDISP.
o The dispersion-mask creation (p3d_cdmask and p3d_wavecal_dispmask_gui):
- Up to now it was always necessary to create a dispersion-mask image from
scratch interactively, which could be a quite time-consuming process if
there are many dispersion masks to create. Starting with revision 892 it
is now possible to specify a starting dispersion-mask image as an input.
Using such a starting image, which -must- use the same observing
conditions as the new dispersion-mask image, the creation process is
very fast!
- An arc-line list file can now be specified in the p3d main GUI, before
it launches p3d_cdmask.
- Starting with revision 1004 the used arc-line list is stored in the
header of the dispersion-mask image. Because of this it is not
necessary, anymore, to (first store and then remember to) provide
p3d_cdmask with the same arc-line list file that was used to create the
DISPMASKIN dispersion-mask image - the data are loaded automatically from
the header.
o The vimos extracted spectra combination program:
- The previously very preliminary version of the VIMOS extracted-objects
image combination program is now debugged, fully documented, it writes
log messages to both the status line and the log file. The routine,
p3d_cvimos_combine, can be called from both the IDL command line, the
shell prompt (using p3d_cvimos_combine_vm.sh), and from the GUI (using
the lowermost righthand-side tab in the VIMOS configurations).
o The spectrum viewer:
- Added error handling to all (mean) spectrum-writing commands. The mean
spectrum file now also contains the extended header keyword, to make the
file fully consistent with the FITS format.
o The log files:
- Starting with revision 909, added the file revision number to the
routine-specific identification string "rname". This addition makes it
possible to see which routine version was used to produced the outcome.
o The scattered-light subtraction:
- Starting with revision 963 the calculation of a median-filtered image,
using all parts of the image where there are no spectra, is now carried
out by a C routine. The C routine is orders of magnitude faster than the
IDL routine, and also makes use of multi-processor platforms (through
OpenMP). The routine is compiled automatically by p3d.
o Corrected numerous smaller bugs.
2.0b (2011-05-03; Revision 873) - The Swashbuckler
o Modified the implementation:
- VLT/UT3/VIMOS (/VIMOS, /NVIMOS, /MVIMOS, /MNVIMOS):
Data using the high (medium) resolution grism (HR [MR]) are now selected
using the /VIMOS and /NVIMOS (/MVIMOS and /MNVIMOS) keywords. Data from
before (after) the instrument reburbish in May-August 2010 must now use
the /VIMOS and /MVIMOS (/NVIMOS and /MNVIMOS) keywords.
o The IDL VM scripts
Made some minor corrections to how the code is executed using the IDL VM.
The routine scripts (p3d_c*_vm.sh) would have returned an error regarding
a missing environmental variable "path", which is never set, or used, in
these routines. Removed the statement "cd $path". The script "p3d_vm.sh"
has prevented a correct use of the data output directory. The script no
more changes directory to ${p3d_path} before launching the p3d GUI; this
allows a correct use of the /CWD keyword.
- These scripts contain a large number of additional improvements.
o The spectrum extraction
Added the option to subtract a scattered light component, which could just
as well be a dark current component. This mode is activated by setting the
darkscattsubtract parameter to yes in the user parameter file. The
subtraction works by fitting a two-dimensional image to the
bias-subtracted image where the spectra should be extracted from. The fit
is only made on the unmasked pixels; all pixels nearby to spectra are
masked. All properties of the fit are controlled with the options in the
(master) user parameter file.
o The cross-dispersion profile fitting routine
Has now been ported to C. Instead of the IDL version of MPFIT the new
routine uses the C version, which is named cmpfit. The necessary source
code is retrieved automatically with the script p3d_acquire_xtools.sh.
The C source code is compiled when it is used, completely transparently
to the user (using the IDL routine MAKE_DLL). If you see that all your
cores, on your multi-core computer, are not used, then check if your
compiler supports OpenMP. The routine has been found to compile
correctly on linux and macosx, but should also work as it is on solaris-
based platforms. We still need to set up the proper compiler options for
windows-based platforms.
If the compilation fails, for whatever reason, then the original IDL
routine is used instead.
o The dispersion-mask creation GUI
- Dead or unused fibers are no more fitted in the final step, when the
individual arc-line pixel positions are fitted to the raw data. This way
the probability that any arc-line in a spectrum after, or before, the
current spectrum will be incorrectly fitted is minimized.
- Corrected the behavior when reading a arc-line file using the File menu
option. Using the same approach as when the routine first starts, the
read line list is now keeping arc-line entries outside the current
wavelength range.
o Various smaller improvements:
- The default plot colors are now black lines on a white background. It
used to be the opposite.
- Also the spectrum-gap files (gapfile) can be put in the directory of the
user parameter file now.
o Corrected various bugs:
- Exporting the extracted science spectra in Euro3D format is now working.
- Corrected a bug in the spectrum viewer, where the summed spectrum was
saved instead of the average spectrum, even though an average spectrum
was selected.
2.0a (2011-01-27; Revision 661) - The Swashbuckler
o Documentation
All main data-reduction routines are now documented; this includes the six
files: p3d.pro, p3d_cmbias.pro, p3d_ctrace.pro, p3d_cdmask.pro,
p3d_cflatf.pro, and p3d_cobjex.pro. Also the spectrum viewer, p3d_rss.pro,
has been documented.
The documentation includes the following information:
- A short description of what the routine does.
- The main GUI routine (p3d.pro) and the dispersion-mask routine
(p3d_cdmask) both contain a complete description of how they are used.
- A section with information on where additional (detailed) information
can be found on how the routine works, and where the parameter files can
be found, which are required to run the routine.
- A section about the format of the input and the output files.
- Additional sections with more detailed information on specific tasks.
- A description of all input and output parameters/keywords.
- An example section with information on how to use the routine; both by
itself from the IDL command prompt, or using shell scripts together with
the IDL Virtual Machine.
o Added support for six additional instruments:
- WHT/WYFFOS/INTEGRAL (/INTEGRAL, /OLDINTEGRAL):
p3d has been configured to work with data of all four IFUs of INTEGRAL,
i.e.: SB1, SB2, SB3, and the Equalized IFU. In the standard setup the
data must come from the newer 2-detector configuration. If you have data
of the older one-detector configuration you must use the old detector
configuration (/OLDINTEGRAL).
As is the case with the IFU of MPFS all spectra of INTEGRAL are placed
on the CCD without any gaps. Consequently the spectrum profile fitting
becomes very slow, as all spectra are fitted simultaneously.
- VLT/UT2/FLAMES/(ARGUS, IFU1, IFU2) (/FLAMES):
p3d has been configured to work with all available configurations of
FLAMES/ARGUS and the mini IFUs FLAMES/IFU1 and FLAMES/IFU2.
- VLT/UT3/VIMOS (/VIMOS):
Specifically, p3d is now capable of working with data using the medium-
and high-resolution grisms (i.e. HR Blue, HR Orange, HR Red, and MR).
No support for the low-resolution grisms is planned, since these modes
produce data that is very difficult to reduce accurately.
Thanks to an improved tracing procedure it should not be necessary to
modify any parameters when tracing the spectra. But as always, please
check the outcome to see if p3d could indeed identify all spectra
correctly. VIMOS is a complex IFU to trace since there are so many
unused and dead fibers; actually, several spectra simply fall off the
CCDs. Creating a dispersion solution p3d starts off with a linear
dispersion solution, which has been calibrated by hand for the HR Blue
and HR Orange grisms. The remaining two grisms, i.e. HR Red and MR, have
not yet been calibrated, with respect to the wavelength ranges. So it
will be more tricky to create a dispersion solution for these grims
if you are uncertain on which arc lines you can see.
VIMOS data are provided for four separate detectors, referred to as
quadrants. The data must be reduced for each quadrant separately,
before they are combined into one data set. p3d checks all raw data
files for the header keyword that indicates which detector was used, it
is not possible to combine images originating in different detectors.
Starting October 2010 VIMOS has been upgraded with aligned CCDs. The
current configuration will unlikely work directly for data of this
upgraded IFU. However, the required changes should be limited to
providing new dead fiber files.
- Gemini/GMOS (GMOS-S and GMOS-N) (/GMOSS,/GMOSN):
Currently p3d supports the two one-slit modes; i.e. the red and the blue
slits. The two-slit mode is more complex to reduce, and is not (yet)
supported. Since we do not have data for GMOS-N the data files are not
completely configured for this IFU yet (in principle it should only be
necessary to modify the dead fibers file). Additionally, note that the
wavelength axis of data of these IFUs cover more than 6000 pixels. The
initial guess of the linear dispersion solution works poorly across all
the detectors, which is why it is necessary to make a manual dispersion
solution, shifting lines at the opposite end by as much as several
hundred pixels.
- MPFS (/MPFS3K):
p3d supports the newer 3kx2k CCD. The spectrum separation is nearly the
same as the FWHM; hence, the spectrum-to-spectrum cross talk is most
significant. The strong overlap currently prevents good fits of the
profiles. Because the spectra are so tightly packed it seems necessary
to limit the permitted FWHM range during the fitting process. The same
profile width does not seem to work for both the blue and the red ends.
A more complete solution will likely have to account for a varying
FWHM across the dispersion axis. Note that there is no gap between the
fibers of this IFU (O.K. a gap due to a dead fiber is used as a gap to
have two spectrum groups). The spectrum profile fitting procedure
becomes very slow since all spectra are fitted simultaneously.
- VIRUS-P (/VIRUS):
Both the setup of fiber bundles 1 and 2 are now available using the
/VIRUS keyword; the keyword /OLDVIRUS for fiber bundle 1 has been
removed from the routine p3d.pro.
- and PMAS:
Through a minor change in the gratings file the new V500 grating is now
fully supported (the only modification was to replace U300 with V500).
o Cross-dispersion line profile fitting procedure:
Before the signal of different spectra can be deconvolved during the
spectrum extraction it is necessary to calculate a line profile for each
spectrum. p3d now allows the background linear function to be controlled:
- By default the constant value and the slope are fitted together with
all the other profile parameters of each group of spectra.
- The slope can now be set to zero explicitly by setting the parameter
"lprofnobgslope" to "yes".
- The constant value can be set to any positive value (including 0.0),
by setting the parameter "lprofbgzlevel" to that value. If the parameter
is instead set to -1 the default fitting is achieved, while a parameter
value of -2 results in using the lowest value outside the fiber regions.
This last option is useful whenever scattered light seems to be present.
o The tracing procedure:
- When fitting profiles it is now also possible to fit the width of each
spectrum individually; by setting the 'lprofwidth' parameter to yes.
Please note, however, that this procedure currently seems to work well
only with Gaussian profiles. The other profile functions seem to need to
be more constrained when also the spectrum width is fitted. Since this
mode makes the profile fitting even slower the default is lprofwidth=no.
- It is now possible to use a list of spectrum separations for the second
tracing step. Such a list must have SPNUM-1 entries, where SPNUM is the
number of spectra. It is important for this tracing mode that the first
spectrum is actually found in the data. If it is not then it is
absolutely compulsory that the first spectrum/spectra are indicated in
the dead fibers file as being dead, unused, or possibly unused.
Currently this mode is used with the FLAMES/ARGUS IFU, where the
calibration fiber positions are difficult to fit with a constant
separation.
- Dead, unused, and possibly unused fibers are not used anymore in the
third tracing step; this is where the spectrum pattern is matched with
the spectra found in the second tracing step. This improvement was
necessary to trace spectra of IFUs with many unused or dead fibers,
such as VIMOS.
- Dead, unused, and possibly unused fibers are now indicated in the trace
inspector GUI with dark green symbols and lines.
o The flat fielding procedure:
Has been modified to include a better fiber-to-fiber transmission
normalization, as well as a division by the average spectrum. The new
modes are available through options in the user parameter file, as is the
old approach (where all spectra where divided with the average spectrum).
The fiber-to-fiber transmission function can now either be taken from the
current or a separate dataset. This approach is useful if you want to use
twilight flats to perform the fiber-to-fiber transmission correction, while
you at the same want to use a continuum flat to first smooth and replace
the dataset with a polynomial fit, before the full spectra are normalized.
o Using p3d with the IDL VM - i.e., without an IDL license:
This was previously a bit tricky as all configuration had to be done using
the GUI. This has now all become much easier.
- All commands that are required to start p3d (i.e. those of p3d.pro
and p3d_gui.pro - actually, nearly all commands) can now be entered at
the shell prompt when starting p3d using the VM script p3d_vm.sh.
For example, to start p3d for the VIMOS IFS instrument, with a log file
dred.log, and writing all output data to the current directory, you
would give the following command (this, currently, works on all the
UN*X platforms):
${p3d_path}/vm/p3d_vm.sh /vimos logfile=dred.log /cwd
- It is now also possible to call the individual routines using the IDL
VM. Five new shell scripts are provided in the vm/ directory, one for
each data-reduction task: p3d_cmbias_vm.sh, p3d_ctrace_vm.sh,
p3d_cdmask_vm.sh, p3d_cflatf_vm.sh, and p3d_cobjex_vm.sh. These scripts
are most easily called by writing shell scripts that define all the
required input parameters of the respective task. Each IDL routine
(i.e., p3d_cmbias.pro, p3d_ctrace.pro, p3d_cdmask.pro, p3d_cflatf.pro,
and p3d_cobjex.pro) has been modified to read command line parameters.
How the parameters are passed to the shell script is described in the
respective IDL routine. Here is an example on how to call p3d_ctrace.pro
from the terminal (outside IDL):
parfile=${p3d_path}/data/instruments/pmas/larr2k.prm
opath=Output
masterbias=${opath}/masterbias.fits.gz
userparfile=${opath}/user_p3d.prm
p3d_ctrace_vm.sh tracefile_1.fits.gz,tracefilefile_2.fits.gz \
$parfile masterbias=$masterbias userparfile=$userparfile \
opath=$opath logfile=$opath/dred.log /verbose
Note that it is necessary to click on the splash screen that pops up
each time the IDL VM is started. A trick on how to automatically click
on this splash screen (to actually start IDL) is given in README.
- The spectrum viewer can also be called using the IDL VM. A new shell
script is provided in the vm/ directory "p3d_rss_vm.sh". In comparison
to when this routine is called from inside IDL the first argument is
the filename of the spectrum fits-file. Inside IDL the first argument
would be the already loaded image variable.
o The fiber position tables for PMAS/PPAK/4kx4kCCD and PMAS/PPAK/2kx4k CCD
now also contain the proper positions of the sky fibers. These sky fibers
are consequently shown properly when viewing them in the spatial map of the
spectrum viewer.
o Corrected numerous bugs related to configurations where the dispersion axis
is in the second dimension (i.e. on the Y axis).
1.1.1 (2010-04-21; Revision 182)
o Corrected a problem that has affected all optimally extracted spectra of
all revisions and releases so far. The problem has been that it has been
necessary to apply a 0.5pixel offset to the profile spectrum positions
before the spectrum extraction. Previously this offset was applied in an
improper way which has not solved the problem. From now on all profile
positions are offset by 0.5 pixel before the data are saved to the *_cdlp*
file. In order to achieve a better accuracy in you extracted spectra it is
necessary to either recalculate the profiles or apply the 0.5 pixel offset
by hand, before extracting the spectra anew. If you want to know more about
how important it is with accurate profile center positions for the accuracy
of your extracted spectra please have a look in Sections 4.1 and 4.4 in
the p3d paper.
Note! Most instruments are affected by flexure, which has the potential to
cause an additional offset between the profiles calculated using a
continuum or sky flat exposure and the object exposure. p3d does not -yet-
correct for such offsets (although support might be added soon).
o The image viewer p3d_imv now also looks for raw data in the directory of
the main input file (FILENAME) and in the parent directory of FILENAME.
This makes it easier to inspect the data if the reductions are carried out
on one machine, and are thereafter transferred to another computer (where
the path is different).
1.1 (2010-04-16; Revision 162) - Upperkuchen
o Much improved support for use with the PMAS 4kx4k CCD.
o Implemented a much more strict checking of the available parameters in the
instrument keywords file(s).
o Included support for flipping the raw data on the dispersion axis. The
flipping is performed when the DFLIP parameter is set (either in the
instrument parameter file or in the user parameter ditto).
o File suffixes, and the string that is used with combined images, can now be
set in the user parameter file.
o All parameters in the master user parameter file
(data/master_userparfile.dat) are now fully described.
o Added transparent support for gzip-compressed input data files.
Z-compressed files (which were compressed using the UN*X-command compress)
and bzip2-compressed files are also supported on UN*X platforms - see the
READFITS routine of astro-lib for more information. Output data files are
compressed using gzip (all platforms!) if the user parameter compress is
set to yes (default==no).
o All references to PASP have been changed to A&A, which is where the paper
on p3d is published (yes, we changed our mind).
o Updated PPAK tracing parameters to work more generally.
o Document headers have been adjusted to work better with the IDL->html
translation tools of O. Streicher.
o Fixed numerous smaller bugs.
o GUI
- Added several different filters in the file dialogs, supporting
compressed files as well as non-compressed ones.
- Only the first-block files of multi-block instruments are now shown by
default in the file dialogs.
- The default width of file dialogs is now set to 500px when using the VM
(see the Xdefaults file in vm/).
o Image combination
- All image-combination-related operations have been moved to a separate
routine (routines/p3d_misc_imcombine_wrapper.pro) to comply with the
"Don't Repeat Yourself" coding principle.
- Added a slow combination method since the fast MIN() function of IDL
gives strange results (with PMAS at least). This slow method is now also
the default.
o Spectrum extraction
- The optimal extraction routines MOX and MPD have been moved to their own
routines.
- The default extraction method for MPD is now 'banddiagonal' and not 'svd'
as previously ('svd' is painfully slow).
- Information about what is going on is now also printed on the console
when verbose==1.
o Master Bias
- It is now possible to pick only one file and then let p3d smooth it
before it is used as a master bias. This now also works with the
PMAS 4kx4k CCD.
o Wavelength calibration / Dispersion Mask creation
- Now works properly with two input files for all supported instruments.
Two files (ideally of different exposure time) can be used if arc lines
on one side of the detector are much fainter/brighter than on the other
side.
- After entries in the line list were matched with the data a more precise
Gaussian fitting (or weighting) was previously performed in the
"Create dispersion mask" step. In order to allow a checking of the more
precise fitting this fitting has now been given its own function in the
GUI ("Fit centroids"). This function (button) must now be used after the
"Match centroids" step, and before the "Create dispersion mask" step.
- The default behavior for p3d is now to not rebin the data on the
dispersion axis before starting the GUI. This allows big data sets to be
used on very small screens (DBIN=1). If the user still wants to have the
data rebinned then she/he can select another mode in the Options menu,
or simply set the DBIN parameter in the user parameter file.
o Object extraction
- Now accepts a cosmic ray mask (although this method does not work well at
all).
- p3d can now also save output spectra in the E3D-format (finally; as we
have promised in the paper on p3d).
1.0 (2010-02-02; Revision 30) - Destiny
o Initial version.