文件列表:

ASCII_FILES_TO_PLOT (0, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL (0, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_B.dat (68372, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_B_OLD_WITH_WRONG_UNITS_DEC08_2017.dat (57186, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_NORM_flats.dat (167175, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_NORM_spots.dat (167175, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_c2_flats.dat (173862, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_c2_spots.dat (19318, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_diff_fluxes_center_pixel.dat (75567, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_median_deficit_flux_spots_center_pixel.dat (103, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_median_flux_flats_center_pixel.dat (125, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_median_flux_flats_pixel_1.dat (125, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_median_flux_flats_pixel_2.dat (125, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_median_flux_flats_pixel_3.dat (125, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_median_flux_flats_pixel_4.dat (125, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_median_flux_flats_pixel_5.dat (125, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_median_flux_flats_pixel_6.dat (125, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_median_flux_flats_pixel_7.dat (125, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_median_flux_flats_pixel_8.dat (125, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_median_flux_flats_pixel_9.dat (125, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_median_flux_spots_center_pixel.dat (125, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_metric.dat (977, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_metric_surrounding.dat (153, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_ratio_fluxes_center_pixel.dat (75567, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_relative_size.dat (120, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_residual_center_pixel_flat.dat (94802, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_residual_pixel_1_flat.dat (97107, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_residual_pixel_2_flat.dat (97095, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_residual_pixel_3_flat.dat (97105, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_residual_pixel_4_flat.dat (97107, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_residual_pixel_5_flat.dat (97097, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_residual_pixel_6_flat.dat (97132, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_residual_pixel_7_flat.dat (97121, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_residual_pixel_8_flat.dat (97114, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL\jay_residual_pixel_9_flat.dat (97109, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL_NOV17_BACKUP (0, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL_NOV17_BACKUP\B2.cat (19150, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL_NOV17_BACKUP\B_oct13.dat (27893, 2023-05-12)
ASCII_FILES_TO_PLOT\CENTER_PPL_NOV17_BACKUP\B_oct16.dat (189291, 2023-05-12)
... ...

# bf_hxrg_ppl Code to measure the BF effect in HXRG detectors using data from JPL's Precision Projector Laboratory at Caltech. This code was used to produce the results in Plazas et al. 2018 (arXiv:1712.06***2) ## Main code: bf_ppl.py ### Usage of the code: % python bf\_ppl.py The code is run in `lucius`, and it reads parameters a configuration filed named `config_bf_ppl.ini`. The code analyzes PPL data (`SOLO` or `FILMTRIP` mode) and simulated data (darks, flats, and spots) created by using a combination of the codes `sim.py` and `hxrg_simulator.py` (see below). ### Files that should be in the same directory where "bf\_ppl.py" is: - `badger.py`: Slightly modified version. In particular, the lines `fileStr['FRAMTIME'][i] = thisHdr['FRAMTIME']` and `fileStr['LODFILE'][i] = thisHdr['LODFILE']` were added in the function ` organize_ramps`. - `sextractor_engine.py` - `pixel_rejector.py` - `moments.py` - `sigma_clip.py` : From Dr. E. Sheldon - `config_bf_ppl.ini` ### Parameters in `config_bf_ppl.ini` for `bf_ppl.py`. The parameters for running the code should be specified in a configuration file named `config_bf_ppl.ini`. Their names and meanings are as follows: - `OutDirRoot`: Output directory path. - `OutDirName`: Name of output directory. Will be created if it does not exists already. All the output files (see below) will be placed in this directory, whose path is given by `OutDirRoot`. - `OutPDFName`: Name of the output PDF file where diagnostic and preliminary plots will be produced. - `SigmaCut`: Cut for sigma-clipping when averaging ramps over spots. - `Gain`: Mean gain of the detector, in electrons per ADU. - `YSize`: Size of the (squared) detector, in pixels (e.g., 2048) - `NChan`: Number of channels in the detector. - `NRef`: Number of reference pixels to use (e.g., 3). - `CorrectNL`: Should the code correct for detector nonlinearity? Boolean (`True` or `False`). - `PolyOrder`: Order of the polynomial to correct for nonlinearity. Should be `2` or `3`. - `CorrectIPC`: Should the code correct for IPC? The constant kernel is denoted by `K` in the code. Boolean (`True` or `False`). - `SubtractDark`: Should the code subtract dark images? Boolean (`True` or `False`) - `Simulation`: Should the code use simulated ramps (`True`) or PPL data (`False`)? - `ExamineRamps`: Should the code look for outlier ramps (`True`) or PPL data (`False`). The results will be plotted in the first three pages of `OutPDFName`. - `DiscardRampsSpots`: List of spot ramps number, separated by a white space, that should be discarded. E.g.: `4 20 22`. Set to `-1` if no ramps should be discarded. - `DiscardRampsDarks`: List of dark ramps number, separated by a white space, that should be discarded. E.g.: `1 3 6`. Set to `-1` if no ramps should be discarded. - `DiscardRampsFlats`: List of flats ramps number, separated by a white space, that should be discarded. E.g.: `-1`. Set to `-1` if no ramps should be discarded. - `StartFrameFlats`: Number of starting frame for ramps of flats. If you don't wish to discard any frame, set it to `0`. If you want to discard the first frame, set it to `1`. - `StartFrameSpots`: Number of starting frame for ramps of spots and darks. If you don't wish to discard any frame, set it to `0`. If you want to discard the first frame, set it to `1`. - `XBorderCut`: Do not use spots whose `x` coordinate is within this number of pixels rom the detector border. E.g., (`50`) - `YBorderCut`: Do not use spots whose `y` coordinate is within this number of pixels rom the detector border. E.g., (`50`) - `EndSpotVector`: When doing the big loop over all spots in the stacked image, this value is the final number of spots that should be considered. E.g., `-1` represents that the last entry of the spots vector is `-1`, i.e., the whole vector. - `BadPixelMask`: Bad pixel mask by Dr. Eric Huff. E.g., `/projector/aplazas/master-euclid-mask.fits`. Will be use by `SExtractor` (in the parameter fie) when finding the sources. It will also be used to discard the postage stamps of the spots that have at least 1 pixel whose value is different from `0` (good) in the mask. - `CentroidType`: Either `center` or `corner`. Chose sources close to the center of the pixel or to the corner. - `CentroidThreshold`: Distance, in pixels, from center or corner of a pixel. E.g., `0.1`. - `RegionCorner`: If `CentroidType` is set to `corner`, this value should be set to `1`, `2`, `3`, or `4`, according to the quadrant of the `3x3` postage stamp that you wish to stack. - `SextractorPath`: Path to `Sextractor` executable. E.g., `/usr/local/optical/sextractor/bin/sex`. - `ListDarksPPL`: List of PPL files with darks. E.g., `/projector/aplazas/data/WFIRST/2017-03-02/raw/andres-000[0-9]*.fits`. - `ListSpotsPPL`: List of PPL files with spots. E.g., `/projector/aplazas/data/WFIRST/2017-03-02/raw/andres-02[3-9][0-9]*.fits` - `ListFlatsPPL`: List of PPL files with flats. E.g., `/projector/aplazas/data/WFIRST/2017-03-02/raw/andres-00[3-9][0-9]*.fits`. If you want to mitigate or get rid of the `burn-in` effect, discard the first few files at this point. - `ListDarksSimulation`: List of simulated files with darks. E.g., `/projector/aplazas/TESTJULY21_90_V9_OFFSET00_LOW_NOISE_NO_NL/*BACKGROUND*_00[1-2]*.fits`. - `ListSpotsSimulation`: List of simulated files with spots. E.g., `/projector/aplazas/TESTJULY21_90_V9_OFFSET00_LOW_NOISE_NO_NL/*OBJECT*.fits` - `ListFlatsSimulation`: List of simulated files with flats. E.g.,`/projector/aplazas/TESTJULY21_90_V9_OFFSET00_LOW_NOISE_NO_NL/*FLAT*.fits` ### Output: The following ASCII files will be created by the code and placed in the directory `OutDirRoot`+`OutDirName`. Use the code `plot_fn.py` to read and plot them. The figures for the paper come mainly from the PDF file produced after running that code (`plot_fn.py`). - `jay_relative_size.dat`: Data for Fig. 8 of the paper. - 3 columns, one row per ramp frame. - | mean size (returned by `moments.py`) | error on mean | signal in central pixel of average between consecutive frames | - `jay_metric.dat`: Data for Fig. 3 of the paper. - (1 + 3\*Nframes) columns - |pixel number| Nframes columns: `f_N` for each frame | Nframes columns: error on `f_N` per frame | Nframes columns: mean signal per frame | - `jay_metric_surrounding.dat`: Data for Fig. 4 of the paper. - Row 1: sum of neighbors; Row 2: central pixel. - |row ID | Nframes columns: `f_N` per frame | Nframes columns: error on `f_N` per frame | - `jay_median_flux_flats_pixel_[1-9].dat` and `jay_residual_pixel\_[1-9]\_flat.dat`: Data for Fig. 2 of the paper. - One file per pixel, in each case. - Each of the `jay_median_flux_flats_pixel_[1-9].dat` files has one column with the mean signal in all frames of the ramp. - Each of the `jay_residual_pixel\_[1-9]\_flat.dat` files has Nframes columns and Nspots rows. The green spots in Fig. 2 correspond to the data in this file, and the red curve, the median per frame (column). - `jay_B.dat`: Data for Fig. 9 of the paper. - 8 columns: |`fc` | `b` | `c1` | `c2` | `m` | `m_err` | `m/fc` | `new_B`; Nspots rows. - The column used to create the histogram in Fig. 9 is `new_B`, which is the same as Eqn. 11 of the paper. - `b = 2*(c1)*(c2)/fc`, an alternative to `new_B`. The two give consisten mean values. - `jay_metric_no_norm.dat`: Data for Fig. 6 of the paper. Eqn. 4 of the paper with the normalization set to 1. - `selected_positions_centroid.dat`: `x` and `y` positions in teh detector of the selected spots after centroid condition has been satisfied. - `OutPDFName`: PDF file with some preliminary and diagnostic plots. The plots for the paper are produced from the flies described above and by running `plot_fn.py`. - Note: To produce Fig. 7, run the code 4 times as described in part 9 below, setting the parameters `CentroidType` to `corner` and `CornerRegion` to `1`, `2`, `3`, or `4`, respectively. Don't forget to change the `OutDirName` every time. ### Parts of the code (from top to bottom): #### 1. Plotting options #### 2. Function definitions #### 3. Parameters: Read in from `config_bf_ppl.py`. Note: For the list above, double check with the PPL log book in Dropbox. The first ten ramps of the 100 flats and spots ramps are not used, for example (to mitigate burn-in effect). Also, I split the list of files into two lists due to the way I list them through the use of regular expressions, but this is not necessary. Make sure that the spots and darks have the same exposure time (same number of samples per ramp). #### 4. Load Data Here the code uses `badger.getRampsFromFiles` so make sure you can import `badger` or that you at least have the required files in your directory. Time along the code is in miliseconds, because it is read from the files produced by the PPL, which record time in tehse units. Thus, sometimes we need to multiply by 1000 (c.f., `plot_fn.py`) to report time in seconds (as is done in the paper). #### 5. Stack data If the number of files is less than 40, take the median. If it is larger, split the list in 3, take the median of each part, and then the mean of the last 3 medians. This is to avoid running out of memory. #### 6. Switch the sign of ADU (ADU-> 2^16 -1 -ADU), subtract mean of reference pixels, convert ADU to electrons #### 7. Correct for IPC with kernel K `GLOBAL_SPOTS`, `GLOBAL_FLATS`, and darks are vectors that contain the frames for the median ramps. #### 8. Run SEXtractor on last frame of median ramp GLOBAL_SPOTS if not running simulations Convert back to ADU the last frame, use `daofind_sex_detection.config` as configuration file for `SExtractor`. The output catalog will be placed in: - `out=out_dir + '/' + prefix + '_sextractor_out_last_sample_ramp_100.param'` If you are using simulations, the code does not run `SExtractor` and uses a catalog of positions created by the user when making the simulations. #### 9. Centroid calculation from last frame of spots ramp, in electrons. Subtract the bias (`B_spots`), and then, within the loop, calculate the unweighted centroid after subtracting the local background. . If `CentroidType` is `corner`, select only those sources in a given Cartesian quadrant (Region 1 to 4). I changed this by hand and ran the code 4 times to get the data to produce Fig. 7 of the paper. #### 10. Big loop over sources to correct for NL (from flat fields), and calculate f_N Loop over sources: - Discard if it has at least one bad pixel. - For each source, loop over pixels in postage stamp - Use function `fit_pixel_ramp` to fit a quadratic or cubic function to the ramps of the spots, flats, and darks. - After fitting the ramps, calculate model residuals. Fig. 2 in paper. - Correct stamps of darks, spots, and flats for NL by using `np.root` for the quadratic and cubic cases. Subtract darks after correcting for NL. - Calculate size of corrected stamp, save in a vector. - Loop over each pixel the corrected spot stamp: - Calculate signal and time difference between consecutive frames - Turn electrons into electrons per time for each difference: `rates_vec_jay=delta_sig/delta_time`. - In the process, calculate `F_i - ` to eventually produce Figure 6. - Calculate difference in rate with respect to first frame, and then normalize to produce the `f_N` metric: `jay_metric`. The vector is `s_vec_jay/=NORM`. - For the central pixel, calculate the coefficient B : - `new_B` = (m/fc)\*(NORM/(val0\*delta\_t/1000)) - The parameters used are derived from the fit: `m, m_err=linear_fit_m (samples, s_vec_jay, err)` - Save that `new_B` in a vector; use those numbers to produce histogram of B in paper. - Note that `new_B` and `b = 2*(c1)*(c2)/fc` are consistent with each other. #### 11. After big loop, save files with output data #### 12. Calculate the mean of the size of the postage stamp in each frame; then calculate relative size to first frame Fig. 8 of paper #### 13. Plots: These wont be the final plots in the paper. Those are produced by another code (`plot_fn.py`), using the output ASCII files listed above. ## Code: plot_fn.py After running `bf_ppl.py` for different configurations (e.g., simulations, PPL data center, PPL data corner in each Cartesian quadrant), a set of ASCII files is produced in the outpur directory that was especified in the configuration file. Then `plot_fn.py` reads those files to produce most of the plots that ended up in the paper. In addition, `plot_fn.py` reads files from simulations. ### Configuration file `config_plot_fn.ini` for `plot_fn.py` - `DoSigmaClipping`: Boolean. Should the code do sigma clipping? - `SigmaClippingCut`: Integer. Number of sigmas if doing sigma-clipping. - `FramTimePPL`: Float. `FRAMTIME` keyword in the header of the PPL files. E.g., `0.837632`. - `NFramesPPL`: Integer. Number of final frames in the `f_n` plot. You can use `OutPDFName` to figure it out. It is not the initial number of frames in the raw data/simulated data, because the process we subtract consecutive frames (and sometimes we discard the first frame). - `NFramesSim`: Integer. Same as `NFramesPPL`, but for simulations. - `NFramesCornerPPL`: Integer. Same as `NFramesPPL`, but for the 4 cases where the centroid of the spots are close to the corner of a pixel. - `PPLDataDirCenter`: Location of `OutDirName` from `bf_ppl.py`, after running the code with `CentroidType` set to `center`. E.g., `/Users/amalagon/NL_plots/ASCII_FILES_TO_PLOT/H_FILTER_CENTER_PPL/MAR21_H_BAND_F11_CUBIC` - `PPLDataDirCornerR1`: Location of `OutDirName` from `bf_ppl.py`, after running the code with `CentroidType` set to `corner` and `RegionCorner` to `1`. E.g., `/Users/amalagon/NL_plots/ASCII_FILES_TO_PLOT/CORNER_PPL/SECOND_RUN/REGION1_xc_lt_0_yc_lt_0` - `PPLDataDirCornerR2`: Location of `OutDirName` from `bf_ppl.py`, after running the code with `CentroidType` set to `corner` and `RegionCorner` to `2`. E.g.,`/Users/amalagon/NL_plots/ASCII_FILES_TO_PLOT/CORNER_PPL/SECOND_RUN/REGION2_xc_gt_0_yc_lt_0` - `PPLDataDirCornerR3`: Location of `OutDirName` from `bf_ppl.py`, after running the code with `CentroidType` set to `corner` and `RegionCorner` to `3`. E.g.,`/Users/amalagon/NL_plots/ASCII_FILES_TO_PLOT/CORNER_PPL/SECOND_RUN/REGION3_xc_lt_0_yc_gt_0` - `PPLDataDirCornerR4`: Location of `OutDirName` from `bf_ppl.py`, after running the code with `CentroidType` set to `corner` and `RegionCorner` to `4`. E.g.,`/Users/amalagon/NL_plots/ASCII_FILES_TO_PLOT/CORNER_PPL/SECOND_RUN/REGION3_xc_lt_0_yc_gt_0` The following parameters specify the location of the directories with files from simulations, used to produced Fig. 5 and the green histogram in Fig. 9. - `SimDirCenterNLCorrected`: `/Users/amalagon/NL_plots/ASCII_FILES_TO_PLOT/CENTER_SIM_NL_CORRECTED` - `SimDirCenterNLNotCorrected`: `/Users/amalagon/NL_plots/ASCII_FILES_TO_PLOT/CENTER_SIM_NL_NOT_CORRECTED` - `SimDirCenterNothing`: `/Users/amalagon/NL_plots/ASCII_FILES_TO_PLOT/SIMS_NOTHING` - `SimDirCenterBHistogram`: `/Users/amalagon/NL_plots/ASCII_FILES_TO_PLOT/CENTER_SIM_BF_90RAMPS_V7` - Note: If you use different simulations, you need to especify the fluxes per pixel for a simulated spot. This is given by the following dictionary in the code: `simulations_flux={'1':[402.957, 677.249, 955.99, 1240.43], '2': [1825.97 ,3078.59 ,4363.04, 5672.92], '3':[402.915, 677.282, 957.607, 1241.39], \ '4': [1827.36, 3079.58, 4361.75, 5672.8], '5':[28285.2, 46977, 65505.4, 83830] , '6': [1832.58 ,3088.91, 4374.23 ,5687.13], \ '7': [402.496, 67***44, 954.201, 1238.62], '8': [1826.36, 3079.46, 4365.73, 5677.67], '9': [403.374 ,677.778, 956.593, 1240.58]}` ## Code: sim.py Uses GalSim to produce a simulated 2k by 2k scene with a grid of point sources. The number of spots depends on the size of their individual postage stamps; this can be chaged at the beginning of the code. As input, the code reads the PPL PSF model file provided by Chaz (`chazPSF_lamda1_cd3_f11_pix1_noboxcar.fits`). The FITS image will be saved in a directory called `output`. You can change this in the variable `file_name`. To change the placement of the sources, modify the variable offset as needed (e.g., offset=(ud(), ud()) for random offsets, or offset=(0.0, 0.0) for sources perfectly located at the center of the pixel). The simulated scene will be used by the code `hxrg_simulator.py` to produce simulated ramps. ## Code: hxrg_simulator.py Originally written by Dr. Chaz Shapiro. This version has small modifications to add BF (from the Power Law model in GalSim) and IPC. Uses as input the image created with `sim.py`.

近期下载者：

相关文件：

评论：[我要评论] [举报此文件]

收藏者：