Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
fluxcal_notes [2012/05/16 17:38] – [Telecon, May 16] amberb | fluxcal_notes [2019/09/06 22:52] (current) – removed casey | ||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== CARMA FluxCal Notes ====== | ||
- | ===== To Do ===== | ||
- | * Figure out algorithm for error estimation (currently 15%, or RMS of values) | ||
- | * check that planet brightness temp is different for different windows | ||
- | |||
- | ===== Telecon, May 16 ===== | ||
- | * online: Amber, Statia, Chat, Woojin, Manuel, Doug, John, Andreea, Tom | ||
- | * fluxcal on science: | ||
- | * Doug gave us a username at Illinois and a directory to work in | ||
- | * where should we put our scripts and other files? / | ||
- | * where do the miriad files live? (so i can grab one for testing) | ||
- | * would you like to copy files to our directory as they come in? script currently just makes a copy of the full path file to local directory, so this is not totally necessary. | ||
- | * what is the best way to automate this? | ||
- | * cron job, python script just always running in a screen, sleep for 30 min | ||
- | * Woojin: | ||
- | * more datasets - 3mm regime data looks good, large scatter in 1mm regime | ||
- | * weather in 1mm data not terrible...why so much scatter? Woojin is doing a phase-only selfcal using the planet... | ||
- | * Mars model: cross points cannot be explained by Mars diurnal variation - what is going on? lots of scatter in daytime pointing - could this be the issue? | ||
- | * Tom: | ||
- | * fluxcal_sci2 now gets abstest sources when they' | ||
- | * eventually want to get just one script for all of this | ||
- | * Peter: ? | ||
- | * email obs and Nikolaus about how to run abstest | ||
- | |||
- | ===== Telecon, May 9 ===== | ||
- | * online: Woojin, Manuel, Amber, Statia, Doug, John, Andreea, Peter | ||
- | * Woojin - new plots of brightness temp versus freq | ||
- | * want to sample higher frequencies in the 3mm band - change track for today to 105-110 GHz | ||
- | * Nikolaus try to get abstest going in sci2 - will contact Tom offline | ||
- | * actually - possible to make 1 fluxcal script work for sci1 and sci2? Nikolaus will take a look | ||
- | * Peter - hasn't gotten to the online system model stuff | ||
- | * Doug - has the go ahead from Leslie, tracking down the IT guy | ||
- | |||
- | ===== Telecon, May 2 ===== | ||
- | * online: Woojin, Manuel, Amber, Statia, Chat, Doug, Tom, Andreea | ||
- | * Woojin: plots from abstest last week | ||
- | * how can we get brightness temperature from planet flux? | ||
- | * Statia | ||
- | * Woojin | ||
- | * Andreea doing same as Woojin but using JPL models | ||
- | * MARS pipeline model is identical to marstb != SZA model w/ diurnal variation | ||
- | * TODO get good MARS model in online system (but Erik's szacalcmars only good at 3mm and 1cm) | ||
- | * Doug will let us know about getting set up at Illinois this week - maybe we can bother Lisa | ||
- | |||
- | ===== Telecon, Apr 25 ===== | ||
- | * Woojin - abstest: still needs 1mm | ||
- | * U+N still give consistent values for other cals, but M gives slightly smaller value by 5% (but much closer than two weeks ago - 15%) | ||
- | * reduction script applies different brightness temperatures in different windows | ||
- | * Jupiter model - constant brightness temp (also too big to be used) - actually may be okay for sci2...? Might be worth taking a look at... | ||
- | * Andreea - also working on abstest data from March | ||
- | * smaflux - links don't work, Tom contacted guy (Eric Weistein) but hasn't gotten answers...odds of making sense of smaflux data are low | ||
- | * Peter - nothing yet | ||
- | ===== Telecon, Apr 18 ===== | ||
- | * Woojin: abstest tracks from last week (3mm - 93.5 GHz) - | ||
- | * Mars and Jup gave same values for 3C84, but 20% lower than values from U+N | ||
- | * we should get brightness temp of U+N based on Mars calib | ||
- | * Peter - same as last week, check against what Woojin has (he will email around) | ||
- | * us: | ||
- | * add a check of flux versus last entry and flag to check it if | ||
- | ===== Telecon, Apr 11 ===== | ||
- | * waiting to hear from Doug about getting set up at Illinois (emailed Apr 5) | ||
- | * DRIP comp stalled b/c webpage hasn't been updated since Mar 16 - emailed Doug Apr 5 | ||
- | * sent notes on webpage to Ted Yu (got short response) | ||
- | * TO DO: | ||
- | * Peter: sort out what planet models the online system is using: let us know the models used, the code that does this, any tables referenced, and where the code and tables live on the high-site. | ||
- | * Andreea: re-work primay fluxcal document, send a summary of relevant information out to everyone for discussion at the next telecon | ||
- | * Nikolaus/ | ||
- | * Woojin: analyze fluxcal abstest data when they are taken | ||
- | * Amber, | ||
- | |||
- | ===== Notes on Webpage (Telecon, Apr 4) ===== | ||
- | * http:// | ||
- | * source selection: | ||
- | * ra dec input - text input should be the default | ||
- | * searching by cal name | ||
- | * extended sources? script writer knows! Depends on array. | ||
- | * show all 3 wavelength band plots at once? | ||
- | * SolSys date? what is this? | ||
- | * error bars - how to handle points that don't have an error? very large error bar? a different color point? | ||
- | * identify change over when we switch to new error reporting | ||
- | * should print FluxSource.cat entries below the plot | ||
- | * plot fluxes versus freq as well? calc rough spectral index? | ||
- | * changing plot limits: | ||
- | * okay to just enter min, not max | ||
- | * flux val reporting: | ||
- | * report avg and scatter for fluxes within plotting window instead of current value | ||
- | |||
- | |||
- | ===== Telecon Jan 31 ===== | ||
- | * set up telecon w/ DRIP people (Doug Freidel) to compare flux results there!!! | ||
- | * SMA, PdBI values for polarization comparisons? | ||
- | * 3C286 paper on archiv | ||
- | |||
- | ===== Day of Action: occupy the lounge to fight the 20% - Dec 16, 2011 ===== | ||
- | Looking over compare3 results (Nov 25 - Dec 9): | ||
- | * empty datasets: | ||
- | * Why so few? Grading! quality script says a B grade but our script does not do a PACS reduction, so the straight grade was a D and failed the first check. (For 1mm and 3mm tracks this will be most relevant) | ||
- | * mostly seems to be bootflux crashing (due to resolution issues) -> change new script to catch this bootflux failure and stop the reduction? | ||
- | * just old datasets: | ||
- | * sci2 tracks (PACS) weren' | ||
- | * cx332 track has an F grade so didn't run through new | ||
- | * datasets with fractional diff > 0.5 seem to mostly be due to bad data in pacs datasets because of bad baselines. We still don't know why the old script was able to handle this though... | ||
- | * analysis of the 5-40% dataset - what have we learned? | ||
- | * at 1cm, using all widebands is going to be an issue w/spectral index of the source - discuss this on the telecon perhaps. | ||
- | * we need to update amp flagging in the new script - amp(500) insufficient - need to track down which bands it is happening in and flag those bands in all ants | ||
- | * DATASET DETAILS: | ||
- | * 1224+213 | ||
- | * 1224+213 | ||
- | * 0108+015 | ||
- | * 0108+015 | ||
- | * NOTE: the MFCAL failed when solving for the passband, though plotting the passband in calibrators.mir with smagpplt looked fine afterward... | ||
- | * Before flagging: 3C84 2.500 / 2.280 (old/ | ||
- | * After flagging: 3C84 5.977 / 5.816 (within 3%) | ||
- | * Before flagging: 0108+015 0.500 / 0.440 | ||
- | * After flagging: 0108+015 1.167 / 1.105 (within 6%) | ||
- | * These numbers are much closer to the c0852.1D_230NGC062.3 numbers | ||
- | * MORAL OF THE STORY: Flagging C7 was hugely important... | ||
- | * 3C84 cx327.1SH_31DQFlex.15 -- no phase coherence on C21 for 1st 1.5 hours (inc. planet observation). Flagging C21 brought flux of 3C84 from old script within 2% of new script value. Note: For whatever reason, flagging C21 did not help the script' | ||
- | * 3C84 c0782.7D_230L1551I.1 The amplitude vs. time plots vary for several antennas. The output of bootflux varies from 6.3-8.1. Not sure how to fix this without an amplitude selfcal. Tried flagging the worst offender (C12) but this did not improve matters. What to do? | ||
- | ===== Telecon Dec 2, 2011 ===== | ||
- | * Mark H on the line to discuss models | ||
- | |||
- | ===== Telecon Nov 18, 2011 ===== | ||
- | |||
- | * Planet models - | ||
- | * Orton models may not be an option b/c may not be done soon and they want recognition in some way? | ||
- | |||
- | ===== Working-Time October 20, 2011 ===== | ||
- | Sorting through the results from the new/old fluxcal comparison: | ||
- | * 248 datasets total | ||
- | * empty: 14 - IGNORE | ||
- | * just old: 24 | ||
- | * WHY did new fluxcal ignore?? | ||
- | * new cuts are A or B and len >= 2 | ||
- | * just new: 117 | ||
- | * are they good? | ||
- | * why didn't old do it? (cal list and prim flux cal) | ||
- | * checked cal list and prim flux cal - still 65 that have prim cal and acceptable secondary cal...? | ||
- | * both: 93 | ||
- | * start by looking at frac diff > 0.5 datasets - which is right? | ||
- | * explosions may be due to passband failing to converge | ||
- | |||
- | **Action Items:** | ||
- | * calibrators list - should we only do certain cals? | ||
- | * email John for a summary of how his program works | ||
- | * add something to check for convergence of passband | ||
- | * 1 bad baseline can kill it - maybe some flagging to check for crazy amps at the start? | ||
- | * capitalization issue - **DONE** (Amber fixed comparison script) | ||
- | * log file issue - **DONE** (Amber fixed fluxes_temp.py and run_gainCheck.csh to write two different log files for sci1 and sci2...hopefully it's working...gainCheck hasn't run yet..) | ||
- | ===== Telecon October 7, 2011 ===== | ||
- | |||
- | * Error discussion: | ||
- | * Big problem: some tracks have high variation in calculated fluxes from measurement to measurement (bad weather?), but others have small variations but a smooth change from high to low (changing tau and/or RMS?) | ||
- | * We need to somehow reflect the weather during the track when we report errors on flux measurements...if there are 10 measurements of a calibrator' | ||
- | * How do we robustly "pump up" the statistical errors to reflect the weather? | ||
- | * What if there are phase variations on timescales smaller than the integration time, even if there' | ||
- | * What about when tracks have a smooth change? | ||
- | * Is T_sys being taken out???? | ||
- | * If this really isn't being done, should we put a calculator on the CARMA calibration website that corrects for elevation? | ||
- | * Maybe we need to make the weather-grade cut even higher, because presumably tracks that have a smooth change throughout show smooth changes in tau and/or RMS, which should get worse weather grades. | ||
- | * This may be an unsolvable problem...we should probably just cut at a high grade, and then report tau and RMS in two columns next to the flux values and be done with it. | ||
- | * We should report the weather grade (which takes array config. into account), and how the grade is calculated | ||
- | * We should report the number of times fluxes were calculated | ||
- | * For the passband calibrators, | ||
- | * We need to put systematic (model) errors on a fluxcal website, but report our statistical (or modified statistical) errors for each flux measurement. | ||
- | * Could val Vleck correction be a problem? | ||
- | |||
- | ===== Telecon August 31, 2011 ===== | ||
- | |||
- | * Getting John's script to work for us: | ||
- | * error handling - what is actually a good error estimate? | ||
- | * should this be run on bad datasets? | ||
- | * need to check out what script is doing before going live with it | ||
- | * automatic flagging reasonable? (flags on selfcal gains) | ||
- | * things to add: | ||
- | * length of track | ||
- | * weather grade | ||
- | * algorithm on bootflux output | ||
- | |||
- | |||
- | ===== Telecon June 24, 2011 ===== | ||
- | |||
- | * Woojin: | ||
- | * weekly list - 5 calibrators - purpose = secondary flux calibrators (passband) | ||
- | * monthly = full list = 209 or 290? calibrators = all miriad calibrators | ||
- | * mainly taken during array config change | ||
- | * uses Miriad bootflux task - planetary models in source code directory: planet.for | ||
- | * Planetary models (Andreea) | ||
- | * compare WMAP planetary models with some of the models: | ||
- | * Mars looks good | ||
- | * Neptune, Uranus and Jupiter seem off | ||
- | * Statia' | ||
- | * Our stuff | ||
- | * more thought needs to go into errors | ||
- | * transfer to using John's scrips - uses all bands and also does sci2 properly | ||
- | * uses all the continuum bands (USB and LSB) | ||
- | * Woojin raises issue of averaging all continuum bands | ||
- | * freq reported is LO1 - not good for sci2 especially | ||
- | * Tom Plagge | ||
- | * szaflux altered slightly version of smaflux | ||
- | * has it's own planetary models - Tom added updated version of Mars model | ||
- | * probably not necessary to switch to this from bootflux for sci2...just make a few tweaks to bootflux code? | ||
- | * sci2 calibration stuff: | ||
- | * corrected aperature efficiency for 3.5m's in miriad - fixed now | ||
- | * MWC349 calibration off of Mars model seems to give consistent results (previous work using Neptune and Uranus yielded inconsistent results) | ||
- | * Planetary models - compare between different institutions | ||
- | * Uranus model from Gerwell that SZA has isn't good anymore? | ||
- | * SMAflux uses Weinstein' | ||
- | * Tom in contact with Weinstein - should get his code? | ||
- | * PLAN GOING FORWARD: | ||
- | * combine fluxcal on science with John's -> talk with John offline | ||
- | * more simple website for PI's to check flux history - better interface to FluxSource.cat | ||
- | * Xplore shows flux history?? | ||
- | * calflux also does this? | ||
- | * which/how many bands to use for fluxcal reduction -> need to learn about planetary models first | ||
- | * document the origin of the planetary models | ||
- | * see how Statia' | ||
- | * what models should we be using?? - how do we get these into the Miriad header -> RTS group | ||
- | * Andreea and Woojin will look at where all the planetary models come from | ||
- | * Consistency between different bands on MWC349 and other calibrators | ||
- | |||
- | ===== Meeting June 8, 2011 ===== | ||
- | |||
- | * look at the not-passed tracks compared to John's | ||
- | * comparing John's values to historical not conclusive - mostly 1cm. 3C454.3 is half of past average value at 1cm. 3mm data point reasonable. | ||
- | * email John with our findings and suggestions | ||
- | * better calculation of errors - 15%?? | ||
- | * rejection of scripts based on weather and track length | ||
- | * get the 1cm calibration working properly | ||
- | * don't assume gains of 1 for tracks going into FluxSource.cat | ||
- | |||
- | ===== Meeting May 23, 2011 ===== | ||
- | |||
- | * check our results against Tom P's 1cm results? | ||
- | * update list of calibrators - 1cm cals from Tom P? | ||
- | * statia will look at neptune models from Apr 27 email | ||
- | * amber will keep track of good and bad tracks this week to compare with John's calibrator table | ||
- | * then send different accept/ | ||
- | * perhaps accept/ | ||
- | * if this is the case, produce a short plotting thing to determine whether to accept - then use what script?? | ||
- | * look at John's script! | ||
- | |||
- | ===== Telecon April 19, 2011 ===== | ||
- | |||
- | * Talk to tom plagge about what 1cm sources should look like | ||
- | * Are the planetary models any good at 1cm? | ||
- | |||
- | |||
- | ===== Meeting April 6, 2011 ===== | ||
- | |||
- | Agenda: | ||
- | * What we're doing | ||
- | * weekly tasks | ||
- | * status of script | ||
- | * look for purpose (in select=) to grab the BP versus phase cals | ||
- | * How is it used? | ||
- | * (effort = usefulness? | ||
- | * FluxSource.cat used by selfcal | ||
- | * picks freq in band but does some time averaging? | ||
- | * if you put a local version of FluxSource.cat, | ||
- | * | ||
- | * How can we make it more efficient? | ||
- | * more useful | ||
- | * something like SMA website? (show IDL program) | ||
- | * script writer uses FluxSource.cat? | ||
- | * less time consuming | ||
- | * Support | ||
- | * for code? | ||
- | * Other issues: | ||
- | * who else is adding entries to FluxSource_newadd.cat? | ||
- | * what is happening to the Flux runs at CARMA? Why? | ||
- | * errors good? | ||
- | * could this be done in quality? what about John's script? | ||
- | * perhaps when quality runs, if track is good, set it aside for fluxcal. Use grep to get grade and time? | ||
- | * where do the coordinates come from? is this the list of ' | ||
- | * THE PLAN: | ||
- | * send less files to fluxcal' | ||
- | * or...just making quality run this? Look at script (Katey) | ||
- | * how is selfcal using this? is it smart? | ||
- | * getting plots online? - purge obs webpage? put this on external site page (use at your own risk) website run by Nikolaus and Tom | ||
- | * Talk to Woojin about what he does | ||
- | * Meredith looking into planet/moon cal |